NYC AI Hearing

working with governments to support regulations both at the state level and at the federal level because we have strong safety practices, but we should also be held accountable for them.

Shane Cahill (05:10:02):

Council Member, thank you very much for this question and overall thank you for deepening the conversation on this and other important topics. I would just reiterate that Meta is committed to developing and deploying our AI offerings safely. That is across the board, both in terms of our models and also our systems, and then our usage policies. So we cover the entire spectrum in terms of our AI offerings.

Alice Friend (05:10:34):

Ma'am, at Google, we have been committed to not just the safety of our users, but the safety and security of the open web itself for over a quarter of a century now. Through our own experiences in our search platform, through everyday tools that users rely on like Gmail, like YouTube, security and safety is incredibly important to us because quite frankly, if people did not trust our platforms and our tools, they would not use them and we would be out of business.

(05:11:06)
For us, safety is a critical business imperative. It's also an imperative as part of being interested and committed to an internet where people can get information that is useful to them, that can help them live better lives. Finally, we invest in the safety of our tools so that we can deploy them to solve some of the world's biggest challenges, everything from natural disasters, warning people ahead of time about floods and fires, to breakthroughs in diseases, to unlocking all of the world's known proteins so that we can someday cure all disease. To us, safety is just a part of this broader mission to use technology and to use AI in particular to make people's lives better.

Logan Graham (05:12:03):

If I can share a personal example of this, I joined Anthropic when we fit around two lunch tables, and the reason I did was because these were the people that I saw were taking it most seriously, that the most important component of getting this technology right was making it safe. I can say that, but let me demonstrate it. From its outset, we are a public benefit corporation legally, so we are legally obligated to weigh the mission of safe AI, not just profit. At every step of the way, frequently at meaningful cost to ourselves, we have pioneered or tried to pioneer the science of safety, everything from training itself, the fundamental research came from us, the evaluations, the pre-deployment testing regime, the monitoring and control, and just this year withholding our most frontier model at the time at great cost in order to deploy with defenders to ensure the world is safe.

(05:12:59)
This is literally our DNA. This is why I'm here. This is why my team and I wake up every morning. I could tell you for a long time about all the other parts of the stack, but I hope that resonates.

Julie Menin (05:13:15):

Okay. I'm going to thank this panel for testifying. The Council still has many questions that we feel we did not get answers to today. We will be sending each of the companies a list of these questions that we would like answers back in writing, but I thank the companies for being here today, answering the members' questions, and thanks to the public for their patience.

(05:13:41)
We're now going to transition to the next panel, which is the city administration. We'll call on them right now. Thank you.

Samuel Levine (05:13:51):

Thank you.

Carmen De La Rosa (05:13:56):

Okay. Please remove them from the chamber. Thank you.

(05:13:59)
Okay. Now we will move on to hear from the administration, and today we will hear from Sarah Milstein, Deputy Commissioner of Strategic Advisory Services at OTI, CJ Dixon, Head of the New York City Cyber Command, Benjamin Krakauer, First Deputy Commissioner of New York City Emergency Management Department, and Sam Levine, Commissioner of the New York City Department of Consumer and Worker Protection.

Speaker 9 (05:14:44):

Thank you, Chair. Please raise your right hands. Thank you. Do you swear or affirm to tell the truth and respond honestly to Council Member questions?

Samuel Levine (05:14:58):

I do.

Speaker 9 (05:14:58):

Commissioner Sarah Milstein?

Sarah Milstein (05:15:02):

Yes.

Speaker 9 (05:15:03):

Thank you. Head of Cyber Command, CJ Dixon?

CJ Dixon (05:15:06):

Yes.

Speaker 9 (05:15:07):

Thank you. Deputy Commissioner Benjamin Krakauer? Thank you. I just want to acknowledge that it was yes. And Commissioner Samuel Levine?

Samuel Levine (05:15:23):

Yes.

Speaker 9 (05:15:23):

Thank you very much. You can begin with your testimony.

Samuel Levine (05:15:25):

And Deputy Commissioner Carlos Ortiz.

Speaker 9 (05:15:32):

And I apologize, and Deputy Commissioner Carlos Ortiz.

Carlos Ortiz (05:15:36):

I do.

Speaker 9 (05:15:37):

Thank you. You may begin.

Sarah Milstein (05:15:39):

Great. Good morning, Speaker Menin, Chair De La Rosa, and Council Members. My name is Sarah Milstein and I'm the Deputy Commissioner for Strategic Advisory Services for the Office of Technology and Innovation, OTI. I'm joined here today by my colleague, CJ Dixon, OTI's Deputy Commissioner of New York City Cyber Command and New York City's Chief Information Security Officer. I'm also joined by Samuel Levine, Commissioner of the Department of Consumer Affairs and Worker Protection, DCWP. Carlos Ortiz, DCWP's Chief of Staff and Deputy Commissioner for External Affairs, and Benjamin Krakauer, First Deputy Commissioner for New York City Emergency Management, NYCEM.

(05:16:23)
In my role at OTI, I oversee both the Office of Algorithmic Accountability, OAA, and New York City Cyber Command. We appreciate that the Council is holding this timely hearing about the risks of AI technology. Thank you, Speaker, for getting representatives of the companies that develop, deploy, and sell AI tools to the general public to speak to the capabilities, limits, and risks associated with these systems at this hearing.

(05:16:51)
As a representative of OTI, I'm here today to provide information to the Committee of the Whole about how we serve the city's technology and cyber defense needs, and to address the proposed legislation that potentially impacts the administration. For those unfamiliar with our work, OTI's core mission is to use technology, data, and design to make the city work better for New Yorkers. We achieve this through a wide range of digital products, infrastructure, data systems, and shared expertise. In other words, we use technology to help agencies make every New Yorker's experience of city government better.

(05:17:29)
For our discussion today, we can speak to the work we're doing around agency use of algorithmic tools and our cybersecurity program. To touch briefly on the AI arm of OTI, I'm pleased to share that the Office of Algorithmic Accountability, OAA, has been formally established and six new positions have been created in the FY27 adopted budget to staff it up. The hiring process for those roles is actively underway.

(05:17:56)
The OAA's duties include analyzing algorithmic tools submitted by agencies to determine whether there's risk that the proposed tool could result in discriminatory decision-making, conducting and publicly reporting on pre-deployment assessments, creating and maintaining a public-facing platform for submission of comments, establishing a protocol with the Department of Investigation for receiving complaints from the public, promulgating rules, establishing basic compliance standards that all agencies must meet, and developing, procuring, deploying and using public impacting artificial intelligence, and reporting the results of pre-deployment assessments and audits conducted by the office.

(05:18:38)
Overall, the office is progressing as planned and we remain focused on building the capacity needed to carry out this work effectively. I will now turn to the legislation on today's docket that would impact OTI. Introduction 161 of 2026 would amend annual reporting on algorithmic tools to include their impact on city employment with a focus on potential changes to funded agency positions, salaries, and duties. OTI is currently undertaking a study on the impact of algorithmic tools on municipal employees pursuant to Local Law 25 of 2026.

(05:19:15)
This study will provide insights into the extent to which algorithmic tools, including artificial intelligence, impact city employees and the administration of their duties, including their hiring and work functions. While we agree that it is important to examine the impact of AI on our workforce, the bill as written would not produce new insights beyond those in the larger, more expansive directive of Local Law 25.

(05:19:39)
The next three bills involve New York City Cyber Command. This office plays a vital role in protecting and defending the city and its residents from the impacts of cyber attacks. On a day-to-day basis, we provide 24/7 security services and assist agencies in bolstering their cyber maturity. We work collaboratively with agency partners as well as state, federal, and private entities to safeguard the essential services and data New Yorkers depend on daily. Our core mission is to make New York City the most cyber resilient city in the world, and we take this extremely seriously.

(05:20:13)
New York City is a target for cyber attacks with a technology landscape that is unparalleled among other cities and states. This requires a unified comprehensive defense against constant cyber threats. Our key duties include setting information security policies and standards for the city, directing the city's citywide cyber defense and incident response, deploying defensive technical and administrative controls, and providing guidance to City Hall and agencies on cyber defense.

(05:20:42)
In my fairly new role at OTI, it is a privilege to work with our staff and our agency partners in furtherance of this critical mission. New York City Cyber Command's alignment within OTI has placed the team in a strong position to monitor and respond to wide-ranging cyber threats. Cybersecurity is continuous work. In my role, I'm confident we can continue to adapt to a constantly evolving threat landscape. Simply put, New York City Cyber Command has a 24/7 apparatus in place to defend the city systems from malicious attacks regardless of the origin.

(05:21:13)
With that in mind, pre-considered introduction of 2026 to 602 would make it unlawful to market offer for sale, sell or deploy an artificial intelligence model in the New York City... The bill would require the director of the Office of Cyber Command to promulgate rules regarding what the third-party validation assessments, certifications and disclosure would entail and the qualifications for third-party validators. To be clear, Cyber Command's core mission is to defend the city of New York's infrastructure from cyber attacks. We are not subject matter experts on the qualifications for an entity to be considered a third-party validator.

(05:21:59)
Further, neither OTI nor Cyber Command have experience regulating the private sector in the way that this legislation would require, and neither would be the best choice to lead this regulatory regime for consumer products. We defer to DCWP to speak about the private sector policy and enforcement piece. To that end, DCWP supports the intent of this legislation. They recommend including robust record-keeping... Advocates and sister agencies that the administration is not certain about the availability or ability of such businesses. We look forward to hearing from those stakeholders today, including on whether initiatives like this one could help spur a market for such services.

(05:22:49)
Pre-considered introduction of 2026 to 601 would require the Office of Cyber Command to establish standards and procedures for contractors to identify the occurrence of a reportable AI safety incident. To take a step back, I'd like to lay out how AI risk and incidents are categorized according to the National Institute for Standards and Technology, NEST. Under AI under attack, AI cybersecurity incidents are traditional cybersecurity compromises where the AI system is the target, for example, data poisoning or jailbreaks.

(05:23:22)
Misuse or malfunction, AI-induced incidents, would be harms resulting from the use or failure of AI aside from a cyber compromise. In this category, the AI system itself behaves exactly as it was built or trained, but the real world outcome causes harm. These incidents are not cyber attacks. We understand the Council's intent here given the publicized instances of AI systems operating outside the bounds of their intended use. As I noted earlier, the risk landscape is constantly evolving and we are continuing to evolve our strategies to keep the city's infrastructure safe, but not all threats related to AI systems are necessarily cyber attacks, and the legislation as written does not address that.

(05:24:06)
Cyber Command would be a critical stakeholder in this discussion, but it is important to emphasize that we need to work with our state and federal partners to address these issues holistically. I also want to provide further context on breaches of security generally. Agency disclosure of a breach of security in any form is governed by Section 10-502 of the administrative code. There are extensive protocols for inter-agency coordination in the event of any breach of security. These protocols are aligned with the New York State Stop Hacks and Improve Electronic Data Security Shield Act. Current law policies and protocols are in place for incidents described in the legislation.

(05:24:49)
Further, the threshold of 24-hour public notice is inconsistent with current law and would compromise the city's investigation of a cyber incident. Pre-considered introduction of 2026 to 606 would require the Office of Cyber Command, in coordination with NYCEM, to develop or update an emergency response plan concerning AI system-related threats to New York City's infrastructure, operations, public health and welfare. I'd once again like to emphasize that there are laws, policies, and protocols in place for breaches in security regardless of actor. However, we are happy to discuss more broadly with Council how we already coordinate with NYCEM to coordinate and engage with partners to restore services.

(05:25:35)
Finally, I will provide DCWP's feedback on the remaining legislation that would impact the administration. Pre-considered introduction to 2026 2605 would allow any person to submit a complaint to DCWP alleging that a person... Strongly supports the intent of this bill to establish a civilian enforcement structure for AI violations, especially given widespread reporting that concerns being raised by AI companies' employees are being ignored. However, there are fiscal and operational concerns with the volume of complaints that the agency would receive and the resources and expertise required to handle each individual complaint.

(05:26:24)
Therefore, the administration believes it would be important to coordinate closely with mission-aligned enforcement agencies at the state level. The New York State Attorney General just recently announced its secure whistleblower portal for New Yorkers to report unsafe and illegal conduct related to AI technology. This could be an opportunity to partner on this existing program to more effectively enforce complaints on behalf of New Yorkers. Pre-considered introduction 2026 2599 would impose requirements and restrictions on companies that provide chatbots, including requirements related to data security and privacy, the right of chatbot users to access their own data, and restrictions on how chatbot providers can use chatbot user data.

(05:27:07)
DCWP has concerns about this bill. While we support Council's goal to better regulate company's deployment of AI chatbots and believe the approach taken by this bill is sound, the implementation and enforcement of this bill's data security and privacy requirements would require a deep technical understanding and investigative ability that is outside of the agency's scope. To be clear, DCWP is committed to regulating harmful practices, including relating to chatbots that fall within the agency's investigative ability.

(05:27:38)
To that end, the administration is working on rules related to the use of AI chatbots to better regulate this space for consumers. The administration is also invested in supporting the New York Attorney General, in their efforts to reign in data abuses for which this bill may serve as a model. Pre-considered introduction of 2026 to 603 would require any advertisement that promotes an artificial intelligence AI model in the city to disclose whether such a model has undergone validation by a third party. It also would prohibit any such advertisement from including any materially false or misleading statement regarding the safety of an AI model.

(05:28:21)
DCWP supports this bill to require disclosure of third-party validation of any advertisement of an AI model. However, as mentioned above, the administration is not certain about the availability or abilities of such businesses, and we want to be clear that any advertising that uses false or misleading statements is currently prohibited under the city's consumer protection law and will be enforced with the full scope of DCWP's authority.

(05:28:47)
We look forward to working with Council to ensure the law includes effective enforcement mechanisms, and to continue addressing issues within DCWP's scope of work as it relates to AI. Thank you for the time. This panel will now take questions.

Julie Menin (05:29:01):

Great. Thank you so much for your testimony. We appreciate it. I've got a number of questions. OpenAI sent the City Council a letter on October 1st recommending that New York City better connect its existing cyber and emergency management capabilities, and I'm just reading from the letter, "With the escalation containment and recovery capabilities of frontier AI companies."

(05:29:25)
So has the city of New York this year met with OpenAI, Anthropic, or any other frontier AI company to discuss how the city would coordinate with them in the event of a serious AI-related cyber or public safety incident?

Sarah Milstein (05:29:46):

Thank you for the question. Going to defer to my colleague, CJ Dixon.

CJ Dixon (05:29:49):

Thank you. Thank you. I would like to note that New York City Cyber Command follows the NIST cybersecurity framework for all of our cyber programming. That is everything from identify, protect, detect, respond, and recover. The basic first principles in cybersecurity have not changed, even with the advent of artificial intelligence. And we have been in conversations with various frontier models about how we may integrate their tools and software to better enhance the functions that we already deliver to the city. So in reference to your question, yes, we have been in conversation with the frontier AI companies to enhance the way that we perform cybersecurity. However, those basic cybersecurity controls have functionally remained the same.

Julie Menin (05:30:30):

So you've met with OpenAI and Anthropic in particular?

CJ Dixon (05:30:33):

We have met with Anthropic. I have not met with OpenAI since coming on board on May 5th.

Julie Menin (05:30:38):

Okay. Are you planning on meeting with OpenAI?

CJ Dixon (05:30:45):

Only dependent on our needs right now. We do not know definitively if there is a valid reason for us to meet with any frontier AI company in service of our-

Julie Menin (05:30:57):

I guess I'm just concerned by the statement that nothing has changed in your protocols given that we're now dealing with a rapidly-escalating technology that as we certainly heard today, and we've been hearing now for the last couple of weeks to months, that there are serious safety concerns. I just want to get an answer in terms of what proactive steps is cybersecurity taking to meet with some of these frontier companies and ensure that our systems are safe?

CJ Dixon (05:31:26):

Acknowledged, and it does not mean that we have not changed our protocols. Some of the risks inherent with artificial intelligence are unrelated to cybersecurity. In those situations where an AI model presents a cybersecurity risk, those AI models will still exploit the exact same software vulnerabilities that are inherent in computer technology as they've always been. Our protocols have not had to change to address the fundamental issues inherent in computer technology, only the speed by which we must react in a cyber attack.

Julie Menin (05:31:59):

Okay. I am concerned by news that city and state reported on Friday that the AI Czar, Jiahao Chen, who is our Chief Technology Officer, resigned I think a week to week and a half ago, and report saying that he resigned in frustration that a lot of positions were being not hired or being held up by OMB. That relates to my second question, which is Council Member Gutierrez, I think she's still here, yes, and I passed bills last year to create the Office of Algorithmic Accountability. And so, it appears that Mr. Chen's resignation from what we're reading in the press was related to that and the fact that that office was supposed to be up and running by the city in June, but it is not up and running. In fact, in your testimony, you said you're now posting for the positions. It's now October.

Sarah Milstein (05:32:54):

Thank you for the question, Speaker. Jiahao Chen served as Director of AI and ML at OTI for two and a half years. He did important work in the previous administration to help define and understand AI use throughout the city. He's been very helpful in getting new OTI administration up to speed, and we wish him the best. We are very excited to be staffing up OAA. The law went into effect in December. I was brought in this summer to help stand up the office.

(05:33:32)
We got funding in June in the FY27 budget, and we've been actively working to hire for the roles, for the director role for OAA, and look forward to announcing something quite soon. We're really grateful to the Council for having stood up, for having helped us stand this up. I think the work is going to be excellent.

Julie Menin (05:33:53):

It just seems that this is an inopportune time for his resignation, and I am concerned that the office was under Council Member Gutierrez's bill supposed to be up and running in June. In addition, the companion bill, which was my bill, which we passed a year ago, required OTI to engage in rulemaking around this Office of Algorithmic Accountability, but OTI did no rulemaking whatsoever, so did not comply with that bill. Instead, OTI issued updated guidance, ignored the rulemaking process. There's been no proposed rule, no public hearing, no public comment period, and no adoption into the rules of the city of New York.

Sarah Milstein (05:34:37):

Thank you. Thank you for the question, Speaker. OTI is fully compliant with the law. The rulemaking piece of the law had us this year submit the initial, I don't want to say paperwork, but the initial points for our rulemaking and the actual rulemaking will happen in this fiscal year. We are also fully compliant with Local Law 35, which was brought under the office of OAA as part of the December 25th law. We've been in the process of fulfilling that report for five years. We've done it on time for five years in a row. That's the law that requires agencies to disclose their use of public impacting algorithmic tools.

(05:35:37)
We're on track now for on-time reporting for March of 2027. We're really excited actually doing that work. Very proud of it. We just last week sent out the email to all of the city agencies to kick off this reporting cycle and are holding on this Thursday the orientation meeting that gets everybody rolling.

Julie Menin (05:35:58):

Okay. I mean, just to clarify for the record, we don't believe that's in compliance at all. The office was supposed to be up and running in June. I'm sure Council Member Gutierrez will talk about her bill, but the office was not up and running in June and was supposed to do the rulemaking. When do you feel then the law department will be... You mentioned law department. When can we expect to actually see the rulemaking? This is in the March '27 deadline?

Sarah Milstein (05:36:23):

Yeah.

Julie Menin (05:36:24):

Okay. What safeguards are currently in place to prevent city employees from...

Sarah Milstein (05:38:12):

... share next year.

Julie Menin (05:38:13):

Okay. I'm going to pass it over to Chair De La Rosa. Thank you.

Carmen De La Rosa (05:38:18):

Thank you so much. Just continuing on that same question, you all testified that my legislation, my Bill 161 basically does the same thing as Local Law 25. One, we look forward to seeing the report. Local Law 25, to my knowledge, requires a one-time report that's going to come out of that study. 161 is more comprehensive and is requiring that report to be updated yearly. As this technology is evolving, in my opinion, hence why I put the bill forward, more consistent reporting is going to be needed, especially as we begin to see how the workforce is impacted.

(05:39:02)
Before I was the Technology Chair, I was the Labor Chair. We have a wonderful Labor Chair in the House, and I'm grateful to her leadership. But this is an issue that impacts not only organized labor, but I also see it as an issue where we're leading the nation really in how the workforce will react to this evolving technology. And so, is there anything that you can share with us right now on where the workforce is and what those impacts will look like? I know you're working on the study, but is there anything you can give us forthcoming?

Sarah Milstein (05:39:38):

Thank you for the question, Council Member.

(05:39:39)
... part about determining how we even define impact. It's a pretty broad term and could mean many different things. We're in the phase of determining that now so that we're able to produce a report that's really meaningful for city workers. We certainly support the intent-

Sarah Milstein (05:40:00):

We certainly support the intent of your bill and I would be interested in further conversation about the annual nature of the reporting.

Carmen De La Rosa (05:40:12):

I appreciate that. Some of the information that we're looking for, for example, is I think an important question. How many funded agency positions, for example, will be eliminated due to the use of these tools? The agency vacancies has been something that this council has really been looking at, especially as we look at civil service reform across the board and bringing people into municipal service. So we do want to understand displacement and what that looks like for the workforce. We want to also understand reduction of hours, overtime, which this council has not been shy about, questioning the administrations, past administrations and present administrations about. We also want to know about subcontractors, right? There's a ton of city contracts that are in play right now and want to know what the impacts are.

(05:41:05)
So I look forward to continuing this conversation with you all and building on what we will see from the report of Local Law 25. I also want to say that while my bill doesn't specifically speak to worker protections, this council does believe that worker protections are important, especially as we look at that displacement and what could happen. We just questioned the companies for the better part of the day about whistleblower protections and what happens when this technology gets out of control. So we want to understand how those protections are also extended to municipal workers. Do you have anything in place right now that protects our city's workforce?

Sarah Milstein (05:41:51):

Yes. Thank you for the question. Because this is a little technical, I want to make sure I get exactly the right information, which is that the New York State Civil Service Law Section 8010 establishes a moratorium on discharge, displacement, or the transfer of existing duties and functions currently performed by employees of the New York City and the New York City school district to an artificial intelligence system until 2028. The Civil Service Law also stipulates that the use of AI shall not alter existing collective bargaining rights, terms of employment, or civil service status. Further, existing collective bargaining agreements may offer employment protections beyond 2028.

(05:42:40)
So with that law in place, we have a little bit of time and we really appreciate the intent of this law and the fact that it gives us in New York City a little bit of time to figure out how we understand the potential for displacements, for de-skilling, for the use of subcontractors that might be displacing other city workers and the sorts of things you were raising. So I really appreciate that question.

Carmen De La Rosa (05:43:03):

Great. So we'll continue that conversation. I also want to give you a preview. I have a bill that creates a civil service title for AI within our city agencies. And one of the things that I'm curious about is this inter-agency communication around AI. Can you speak to us, where are you right now when it comes to each of the city agencies and evaluating where the agencies are in terms of the use of AI internally?

Sarah Milstein (05:43:30):

Yes, thank you for the question. So one of the privileges that we have at OTI as the agency that runs the reporting for Local Law 35 is that we learn from agencies throughout the city how they are using algorithmic tools that have direct public impact. And in the process of collecting that data, we have a fair amount of contact with agencies to be able to learn more and to establish relationships with them so that we can also serve as advisors to them and understand their challenges as this technology unfolds. In the last cycle last year, we had 56 agencies participating.

Carmen De La Rosa (05:44:14):

Our first hearing as technology chair was around surveillance. And one of the things that we pretty much grilled these companies about was about self-regulation. And I think from the tone and tenure of this council, we don't believe in self-regulation. However, we've given the grace to our city agencies to self-report the use of AI. I want to ask you, and I understand that OTI is not an agency that's going to impose penalties on other agencies for not reporting, but there is a specific concern that my colleagues and I have around self-reporting, especially when it comes to the NYPD's reporting of surveillance technologies.

(05:44:53)
This council has had many hearings on that. I wonder under the Office of Algorithmic Accountability, what accountability tools will we put in place to make sure that we are properly regulating the use of surveillance technology in the NYPD and other technology? This council held a hearing just in June around the Department of Education and the use of AI. So what are some of those tools that we can look forward to seeing?

Sarah Milstein (05:45:22):

Yeah, thank you for that question, council member. So in advance of having full-time staff at OAA, we have been doing research to understand some of the types of risk assessment and risk management that organizations like ours can do in a situation where self-reporting is what we have been relying on. We're a little bit early to say how we would implement that in the city, but we are very actively hiring for a director who understands that set of problems and can help us figure out how we come up with effective risk assessment and management that includes a view into city agencies, perhaps beyond self-reporting.

Carmen De La Rosa (05:46:11):

Thank you for that answer. And of course we will continue that conversation.

Sarah Milstein (05:46:15):

Absolutely.

Carmen De La Rosa (05:46:16):

I did want to ask you, how prepared would you say we are for enterprise security system for autonomous AI-driven attacks, not just attacks by human, but AI autonomous attacks? How prepared do you all feel we are as a city?

Sarah Milstein (05:46:33):

Yeah, thank you for the question. I'll let CJ Dixon answer that.

CJ Dixon (05:46:37):

And that is a great question, and I will acknowledge that this is a very dynamic environment. It is always changing. It's just the nature of technology. New York City Cyber Command, however, provides 24/7 detection and response capabilities through our 24/7 security operations center. We have provided that service to the city since the establishment of New York State Cyber Command. We provide that service whether or not the attacker is human or an artificial intelligence-based attacker. That attacker would still attack the exact same vulnerabilities in the system and therefore we would still respond functionally in the same manner. We would just have to do it faster. We continue to evolve with the technology and we will continue to do so with the best of our ability as the technology continues to evolve.

Carmen De La Rosa (05:47:21):

The speaker, rightfully so, in my opinion, asked the companies about the possibility of a kill switch and where that kill switch should live. Do you have any opinions if the kill switch should live in Cyber Command?

CJ Dixon (05:47:34):

I agree in principle with the idea of a kill switch. I will say there are some technical barriers that would prevent a kill switch as conceptualized living in any agency within New York City, whether that is New York City Cyber Command, OTI, or anywhere else. The reason being is the technical reality for these models is they are deployed on distributed systems. On distributed architecture in order to truly with a singular button with a kill switch or anything of that concept for that to actually work functionally, we would need to be able to coordinate across multiple stakeholders that are not actually regionally co-located to be able to turn off everything at the exact same time. New York City Cyber Command nor OTI does not have the authorities or the technical capability to do that.

(05:48:27)
And this runs us up against a similar challenge that we've had with other kill switch bills as it relates to computer technology in general. As this technology gets more distributed, it gets harder and harder for any singular solitary entity to turn it off specifically. That is not to say that there's not technical means by which a group of entities with the legal authority to do so could not disconnect a given model from a given function. But in this particular case, we agree on principle, but it would not be able to be implemented at New York State Cyber Command in practice, not with the technical barriers that are in place.

Carmen De La Rosa (05:49:05):

Okay. I'm sure there'll be more on that. Let's see. In your opinion, should AI control be a recognized cybersecurity specialty? If so, what should be the training and curriculum included in that?

CJ Dixon (05:49:25):

I do not believe every function related to artificial intelligence is necessarily going to be a cybersecurity specific challenge. There are some intersections between artificial intelligence and cybersecurity as a particular domain. And when those two domains interact, that is where certain things should fall within Cyber Command and a cybersecurity professional's purview. And that is any instance of a cyber attack or cybersecurity, regardless of if the perpetrator is a human or an artificial intelligence.

Julie Menin (05:49:59):

I just want to ask one follow-up. How many people are working in Cyber Command right now?

CJ Dixon (05:50:04):

121.

Julie Menin (05:50:05):

And how many vacant positions do you have?

CJ Dixon (05:50:08):

I will need to go back and find that exact number for you, speaker.

Julie Menin (05:50:12):

Okay. Thank you. If you could please provide that.

CJ Dixon (05:50:13):

Will do.

Julie Menin (05:50:14):

Thank you.

Carmen De La Rosa (05:50:16):

Thank you. I have a question for our DCWP commissioner and team. What types of AI misuse are you seeing that harm consumers that you're most concerned about?

Deputy Commissioner Millstein (05:50:33):

Well, thank you, council member. Thank you, Chair. And it's great to have the opportunity to be here. We're concerned about a wide range of misuse, whether it's people using voice cloning technology that these AI companies make fully available to scam our seniors, whether it's grocery stores using AI technology to fix prices or to personalize prices, something that the speaker has introduced a bill to prohibit. We're seeing widespread misuse of these tools. And the other thing we're seeing across the board is an attitude by these AI companies that they bear no responsibility for the downstream consequences.

(05:51:06)
I have to just say, Chair, I was struck by the final answer I heard by the witness for OpenAI who when asked a basic question about AI safety, responded that she works in policy, not safety. How are you running a company like OpenAI set to go public introducing trillion dollar tools and you don't have safety as part of your policy division? Deeply disheartened by what we heard at the hearing earlier today.

Carmen De La Rosa (05:51:30):

Absolutely. You should have heard their first answer. It went something like, "I don't know."

Deputy Commissioner Millstein (05:51:36):

There were a lot of I don't knows, and on behalf of the admin, we hope to give you more fulsome answers.

Carmen De La Rosa (05:51:40):

Thank you. And we look forward to engaging with you in that more broad conversation as well. In your opinion, is DCWP sufficiently staffed to combat this AI misuse? And would adding, in your opinion, a private right of action strengthen the ability to enforce against violators?

Deputy Commissioner Millstein (05:51:58):

Thank you, Chair. I'll be candid. I feel that even as under the Mamdani administration, we are adding to our ranks. I believe we are capable of taking on many of the misuses of AI, whether that's DoorDash using algorithms to underpay its workers, something we remedied in the largest labor enforcement action in history. Whether it's companies using AI to harm consumers through surveillance pricing, we're strongly supporting the speaker's bill on this. I think we have strong tools.

(05:52:26)
That said, no enforcement agency, and I've worked at the state, federal, and municipal level, will ever have the resources to take on every possible misuse of this technology. That is why, Chair, I would strongly support a private right of action as contemplated by a number of these bills to give ordinary people, workers, and businesses the opportunity to hold these companies accountable. I can guarantee you though these companies, as soon as the city does that, are going to run to Washington asking for preemption. Regardless of what they might be saying at this hearing, what really speaks is where their lobbying dollars are going, and that is to DC Congress asking to hit-

Carmen De La Rosa (05:53:14):

... and to my colleagues, Council Member Gutierrez, followed by Aldebol, followed by Maloney.

Jennifer Gutierrez (05:53:21):

Thank you. Good to see everyone. My first question is for Deputy Commissioner. Good to see you. Going off of the speaker's line of questioning regarding the Office of Algorithmic Accountability, can you share... I know you said you're excited about this report, you're setting up an orientation meeting. It feels like not a lot of time. Before the report is scheduled to be released, what can you share about the cadence of how often you're going to be able to meet? Do you expect the report to be released on time? This is not a reflection of you, but I really felt like the previous administration under the previous OTI leadership was delayed in many things. And we certainly don't want compliance to be one of those things.

(05:54:11)
So if you can just share with me a little bit about the cadence, how many people... You said the funding you got in June, how many people will the office be able to employ where you are in that process currently? Oh, okay. That was it for you. I had a question for [inaudible 05:54:25], but I can come back.

Carmen De La Rosa (05:54:25):

Go ahead, go ahead.

Jennifer Gutierrez (05:54:26):

Okay. And then for DCWP, I just wanted to ask if you can share a little bit about how the implementation of Local Law 144 is going, the automated deployment decision tools. Just wanted to see how the implementation of that law is going, and that's it. Fair enough. Thank you.

Deputy Commissioner Millstein (05:54:46):

Well, thank you, Council Member. Let me start by 144. I think this was a really well-intentioned bill to regulate the use of automated decision-making systems for hiring. I think it does have some flaw though, in that it requires companies that use AI systems for hiring to conduct audits. It doesn't actually require that those audits come up clean. That's really a civil rights question. That is why our administration put out a policy statement last week on behalf of DCWP, Commission on Human Rights, and Taxi and Limousine, making clear that existing laws apply. But I do want to circle back to a broader point that this hearing reveals, which is that bias in decision making produced by AI systems is a very real concern you're addressing. These companies would like an exemption from the law. The companies that testified earlier want to assert that if decisions are made in a black box, nobody can be held accountable for it.

(05:55:37)
I applaud this council's effort to actually impute responsibility for these harms to the companies making billions of dollars on it. And we would be very happy to work with the council to work on a revised, expanded, strengthened automated decision-making bill that places the responsibility to remove bias from hiring where it belongs.

Jennifer Gutierrez (05:56:00):

Are there instances where the harm is happening amongst the city agencies using these tools?

Deputy Commissioner Millstein (05:56:07):

I don't believe that. I would defer to OTI and to those in hiring. I don't believe that the city is using AI tools to sort resumes, but I would defer to others in the admin. I can tell you in my agency, we have people reviewing applications just as starting in January when delivery workers get deactivated, we have people reviewing that. When consumers file complaints, we have people reviewing it. We are a big believer in our administration that the people of New York should be served by the people of the city government.

Jennifer Gutierrez (05:56:37):

Thank you. Part of the package of bills that the Speaker mentioned that we passed last December on Christmas was to get the city and to get OTI to report on those AI tools that every city agency is reporting on. So we'll know soon enough, but please, Deputy Commissioner, thank you.

Sarah Milstein (05:56:54):

Yeah, thank you for the question, Council Member. So I think there's a couple of reports we're talking about here. I want to give a little bit of context. Local Law 35 is the report that requires agencies to tell us about the algorithmic tools they're using that have public impact. And that report we have delivered on time every year for the past five years. The process work kicked off last week and we'll be collecting data over the course of the next few months with the plan to publish it in March of 2027. We have all the indications that that will come in on time.

Jennifer Gutierrez (05:57:42):

... report required by Intro 199. So that's the package of bills from last fall.

Sarah Milstein (05:57:46):

Yeah.

Jennifer Gutierrez (05:57:47):

Thank you for clarifying.

Sarah Milstein (05:57:53):

The Director of OAA, which we hope to be able to announce quite soon, will be primarily responsible for the additional reporting that will be required in March of 2027. And in part, because we have this good experience internally understanding process with Local Law 35, we think we're in a good position to move quickly on the additional reporting that will be due in March of 2027.

Jennifer Gutierrez (05:58:17):

Okay. And you said announcement. Do you have someone already for the role?

Sarah Milstein (05:58:24):

We will have an announcement and I hope a week or two.

Jennifer Gutierrez (05:58:26):

You'll ave an announcement about the announcement. Okay. Thank you. Thank you, Chair.

Carmen De La Rosa (05:58:29):

Thank you. We have Council Member Aldebol followed by Maloney, followed by Juan.

Ms. Aldebol (05:58:36):

Thank you, Chair and Speaker Menin. So you said the hiring process for the six new positions that were created in the adopted budget are actively underway. Specifically, where are you in the process? How many people have already been hired? How many people are left to be hired? And what have been the challenges in being able to hire people quickly?

Sarah Milstein (05:59:07):

Yeah, thank you for the question, Council Member. So we hope to be announcing a director for the office quite soon in the next week or two. And we are in the process of defining the additional roles so that they can be brought on quite quickly. We certainly are challenged to find people who have the right combination of AI expertise and understanding of city systems to be able to hit the ground running. That has been a little bit of a challenge. But we do have, I think bringing in a strong leader is going to help us draw good people who are going to want to do this work with us.

Ms. Aldebol (05:59:50):

If I may, Chair, part of the role is to analyze algorithmic tools submitted by the agencies to determine whether there's risk that the proposal could result in discriminatory decision making. What is your office doing or thinking about in terms of protecting the city's workforce as AI develops and evolves and you're kind of like the gatekeeper AI?

Sarah Milstein (06:00:27):

Yeah. Yeah, thank you for the question. So in parallel with some of the risk assessment that we can do of the software, we are doing the research associated with Local Law 25, which has us researching how algorithmic tools and automated employment decision tools might be impacting city workers. And I believe that we should have results from that in 2027 and we'll have a much better sense of the landscape of how workers in the city are impacted.

Ms. Aldebol (06:01:05):

Okay, thank you.

Carmen De La Rosa (06:01:06):

Thank you. Council Member Maloney, followed by Juan, followed by Brewer.

Virginia Maloney (06:01:12):

Thank you, Chair. As we all know, generative AI technology is changing how we communicate, work, and how businesses operate, and the same could be said of our government offices, both how we work and operate, either officially or unofficially. So I want to know how New York City agencies are adapting to the new reality. The first on the opportunity side, there are plenty of examples of government wins of adopting useful technologies for the public good. In Connecticut, they have real-time translation in 80 languages for these kinds of public meetings to increase accessibility. In California, they automatically check permit applications against 18,000 pages of building code in order to speed up reviews.

(06:02:01)
So my first question is what is our equivalent win in New York? And if we don't have one yet, what's the plan to identify those opportunities? And the second question is around oversight. If I may wrap up, OTI published generative AI guidance for agencies last December. So as we're exploring these opportunities, is that guidance binding or is it advise? What happens if agencies ignore that guidance? Has OTI been prescriptive about a list of generative AI or agentic AI tools that have been reviewed and approved for city use that are use cases that the city officially supports? And then how are we thinking about procurement when these models are constantly iterating and changing every few months and when every few days truly, and when new technologies are being introduced? Thank you.

Sarah Milstein (06:02:57):

Thank you for those questions, Council Member. I'm so glad you brought up the question about wins because we're really interested in both effective and responsible use of AI, and I think effective would cover the wins. I don't have a perfect list here, and I'd like to follow up with that because we'd really like to represent the agency as well. In terms of tools that we have approved, broadly, there are two sets of tools that OTI approves centrally, Microsoft Office Suite and other Microsoft tools, and that includes their Copilot chat and some of their other tools have AI features built in. And we also have a central license for Adobe Creative Cloud, and many of those tools have AI features built in.

(06:03:53)
Other software tools that are used by agencies are reviewed for the most part on a one-off basis by OTI, and we are still determining the best way to move forward with that process to make sure that we are identifying and managing risk appropriately. I think your question about procurement is an excellent one, and part of what we have started to research ahead of having staff at OAA, we've started to talk with various experts in risk management to understand some of the ways that we might think about risk in an ongoing way, when traditionally all kinds of procurement has... There's a gate at the beginning for various concerns, and then it's rolling. I think you're right, that's not appropriate for this situation, and we should have more to discuss on that in the coming months, but it's an active question that we have as well.

Carmen De La Rosa (06:04:54):

Thank you so much. Council Member Juan, followed by Hanks, followed by Brewer.

Julie Won (06:05:01):

Thank you so much, Chair de la Rosa. So for the panel, what I have questions about is building off of what Council Member Maloney just said, the agencies are allowed to currently use Microsoft products and also Adobe, but it's clear to me and the public that they are using other AI tools. So not just the passive tools like using Zoom to transcribe their notes and for follow-ups. But in addition to that, they're using ChatGPT, they're using Claude, they're using Nano Banana Pro because we're seeing it for agencies like HPD, DOE, DEP, they're creating flyers for public consumption. They're using it for their public emails, they're using it for public messaging.

(06:05:40)
So what are you actually doing? Because I want to know what the public policy is from the mayor on how the agencies are using AI and how you are regulating it internally as well. And also I would like to build off of that for city contracts, what are you doing currently for data privacy writers and city contracts for all procurement of AI companies for making sure that our constituent data is not being used to train their AI models? And is there a clause at all or a rider in the city contracts, especially with the Copilot since that's the only real one that you have and people are just using their own personal accounts or free accounts for everything else that they're currently using for their day-to-day jobs?

Sarah Milstein (06:06:21):

Thank you for the question, Council Member. So let me first address the question about the AI rider. We shared that with general councils across the city this summer with an updated AI rider, and I want to just make sure I've got the right language here, so I'm going to pull that up. Let's see.

Julie Won (06:06:42):

Yeah, because my privacy concerns are not just for the contracts that exist, but it is for the free willing use of agency staff that are using non-city enterprise licensed AI.

Sarah Milstein (06:06:57):

Yeah. Yeah. So I'll first address the rider and then we'll talk about the shadow use. So the rider, as I said, was distributed to agencies citywide this summer, and it's intended to mitigate risks and apply the city's policies and standards to AI products and features. It's available for all city software contracts, and our legal team recommends that agencies use it even where no AI component is currently contemplated. The rider includes terms relating to intellectual property, security, how the city's data may be used, and privacy. And so that is now active and in use, and I think that with also a proactive component of recommending that it's used for all manner of technology.

(06:07:43)
In terms of the shadow use, which I think is probably both at the individual and at the agency level, we are not in a technical position to monitor that. But certainly we as an agency share concerns about that. And part of what we're interested in doing is making sure that agencies have the tools that they need that are properly regulated. Microsoft Tools, for example, where data can't be shared or used to train their models. That agencies have the tools that they need to do the jobs that they have so that they are not turning to outside tools.

Julie Won (06:08:20):

So who is responsible if you aren't able to regulate shadow use of AI tools for day-to-day use, especially with constituent data, who is going to regulate that? If that's not you.

Sarah Milstein (06:08:31):

I'd like to follow up with you on that question.

Julie Won (06:08:34):

Okay.

Carmen De La Rosa (06:08:35):

Thank you, Council Member. So I know the admin has a hard stop at 5:30. We have about six more member questions, so if you all could just hang on tight, we're going to try to get through this. Members, please consolidate the questions. We have our majority whip, Council Member Hanks.

Council Member Kamillah Hanks (06:08:53):

Thank you so much, Chair. I'll be very, very brief. So you testified with all the technology that's changing rapidly, even the best can't keep up. So in your testimony, you said the administration is continuing to evolve your strategies to keep our city's infrastructure safe, but not all threats to AI systems are necessarily cyber attacks. Can you expand on that?

CJ Dixon (06:09:20):

Yes. So when looking at the way that it is currently described by the National Institute of Science and Technology, as well as some of the universities and academics who've created some of these systems, there are functionally two baskets of AI risk. There is risk related to AI when an AI is attacked as a computer system, just like any other piece of software. And then there are risks related to artificial intelligence that are truly novel to this new technology inherent in the way that these artificial intelligence models may interact with society when deployed.

(06:09:55)
That second category of risk is a function of often a how much society may trust the artificial intelligence and how reliable that artificial intelligence system may be. Those two things can run afoul of what we may expect as consumers or as a government entity completely agnostic of whether or not the system for which those AIs are deployed are actually cyber secure. So a system can be completely cyber secure, but the AI model itself and interactions with other humans in society writ large may take an action that is unexpected and therefore result in some sort of disruption agnostic of a cybersecurity incident. So that's functionally the two categories and that's a very broad breakdown of those two categories of risk.

Council Member Kamillah Hanks (06:10:41):

So I'm going to keep my question just to that. I think that we should talk offline because my legislation requires the 24-hour reporting requirement to Cyber Command in which your testimony has some issues and you talk about how we're going to push this through to federal and state and what specific role do they play in that? How can we make this better? But I will relinquish that. Thank you, Chair.

Carmen De La Rosa (06:11:04):

Thank you so much. Up next we have Brewer, followed by Jose.

Ms. Brewer (06:11:08):

Thank you very much. I know you talked about the Microsoft and the Adobe, but if an agency wants to begin using a new AI model, A, how do they go about it? Number two, does the city run these models on its own data centers or only on the ones that are on the control of vendor clouds? And then if you have a problem, like there's a criteria, what is it for testing to identify the malicious misaligned AI product? And then how much does the city spend each year on outside AI consultants? I could go on, but those are three, four questions. I have 10 pages here.

Sarah Milstein (06:11:45):

Okay. Thank you for the questions. Let's see.

Ms. Brewer (06:11:51):

The process.

Sarah Milstein (06:11:52):

Yeah. Typically, when agencies want to buy software, part of the process is discussing with OTI what they are buying and what they will be using it for. And we have a cybersecurity review process that is part of that to ensure that the software that is either going to be bought or built will be safe. And as my colleague has said, that would include cybersecurity includes the typical AI concerns. So that's process. You had-

Ms. Brewer (06:12:32):

Next question, the clouds, whose clouds are they on? Whose data centers? Where are they?

Sarah Milstein (06:12:39):

Yeah.

Ms. Brewer (06:12:40):

Are you running them or is it outside vendors?

Sarah Milstein (06:12:45):

So to the best of my knowledge, there are no city agencies running their own models, but I would like to-

Ms. Brewer (06:12:56):

So it's vendor clouds. It's vendor clouds basically. Vendor clouds. Vendor clouds.

Sarah Milstein (06:13:00):

Yes. So for example, Microsoft host software that agencies use. But I'd like to follow up after the hearing to make sure I've got the exact right information for you.

Ms. Brewer (06:13:15):

Okay. And if there's something unexpected, misaligned AI product behavior, what do you do?

Sarah Milstein (06:13:21):

That is a question for my colleague, CJ Dixon.

CJ Dixon (06:13:25):

Councilwoman, that depends on what the misaligned behavior is. If that misaligned behavior is specifically a cyber incident, we take appropriate action based on what any cyber professional may do. There are novel misalignments that are born of this technology that do have to be unpacked and we look forward to working with the council as well as our partners across New York City to understand when those misalignments do arise unrelated to existing risks that the city already deals with, how do we actually engage with those particular novel risks?

Ms. Brewer (06:13:55):

Have you had some since you started your job?

CJ Dixon (06:13:59):

None related to what Cyber Command's purview is.

Ms. Brewer (06:14:03):

What-

CJ Dixon (06:14:00):

... unrelated to what cyber command's purview is. And that is primarily cybersecurity issues. We've had no misalignments related to cybersecurity that has resulted in negative consequences for the city from an AI model very specifically.

Speaker 10 (06:14:16):

That sounds very ... I don't know what it sounds. How much outside AI consultant dollars are being spent?

Sarah Milstein (06:14:28):

Council member, could you clarify what you mean by AI consultants?

Speaker 10 (06:14:32):

Sure. Yeah. How many people are you paying, I mean, I've been doing this work a long time, so I know sort of the answer, but how much outside consultants are being paid on terms of AI? In other words, how many people are you paying to work with you on AI issues, outside vendors? Consultants?

Sarah Milstein (06:14:54):

None that I'm aware of, but I would like to follow up with you to make sure that we have accurate information.

Speaker 10 (06:14:59):

So everything to do with AI is in- house. Is that what you're saying? Everything to do in terms of strategy, thought process, future, it's all in-house?

Sarah Milstein (06:15:07):

Yes.

Speaker 10 (06:15:08):

Okay. Thank you.

Carmen De La Rosa (06:15:09):

Thank you. Council Member Osei, followed by Deputy Speaker Williams.

Chi Ossé (06:15:14):

Thank you so much. I just wanted to refer to the recent hacking situation in Australia that was proof that AI agents do have that ability to hack into our government. I'm sure you know in June, a rogue OpenAI agent hacked into an Australian government statistics portal containing private data from Australia's universal healthcare system. At this current moment, if a similar hack that happened to Australia happened here in New York, do you feel that the city and the admin has a sufficient plan to defend against the hacks and protect any sensitive data? And two, does the city currently have a responsive liaison from each of these companies to speak to in case of a hack?

CJ Dixon (06:15:58):

To the first question, yes, we do have the people, process, technology and some of the policies in place to deal with an incident similar to what happened in Australia. The reason being is even though an AI was involved, the hacking process, the attack chain, if you will, happened in the same way that if a human being were to execute that attack, at the end of the day, the underlying infrastructure, I will use air quotes here, doesn't actually care if the person at the other end of the keyboard is a human or a silicon-based entity. The attack chain still remained functionally the same.

(06:16:33)
Therefore, our team is equipped and trained very specifically to identify these indicators of compromise, identify the attack patterns, and then be able to respond accordingly at multiple points. So we are prepared to address those types of issues, and we do have points of contacts when we are working as city agencies with vendors who are under our contract to engage with third parties as they get breached or if their product is the result of a breach. So if they are under contract within the city by [inaudible 06:17:08] downtimes or when their products are involved in a breach.

Chi Ossé (06:17:12):

Thank you for that response. And in regards to my legislation on an emergency plan, and I know that you indicated that there's already a plan in case an emergency takes place. Am I correct with that assumption?

CJ Dixon (06:17:25):

Yes. So there are cyber incident response plans as well as my colleagues from NYSEM. There's also ongoing work for risk response plans in general across the board, and that is for every category of risk that the city may deal with that might disrupt city infrastructure or city services.

Chi Ossé (06:17:42):

Is that publicly available or is that shared with the council?

CJ Dixon (06:17:45):

No, that is by design. We do not want to share that publicly because if that is the more public aware that someone is of an incident response plan, the more likely they can circumvent it that that's something that you do want to keep close hold and classified.

Chi Ossé (06:17:59):

Okay. Thank you very much.

Carmen De La Rosa (06:18:00):

Thank you so much. Council Member Hani, followed by Deputy Speaker Williams.

Shahana Hanif (06:18:07):

Oh, I though the deputy speaker was going before me, but anyways.

Carmen De La Rosa (06:18:09):

I'm sorry, I messed up.

Shahana Hanif (06:18:10):

Okay, no worries. I have a question first for Deputy Commissioner Millstein. How does OTI ensure that the automated pre-screening tools and fraud detection algorithms used by the HRA do not create disparate impacts or algorithmic bias [inaudible 06:18:30] have for Commissioner Levine, the testimony indicated that DCWP supports a complaint process, but you all raised concerns about staffing and expertise. What specific resources would you need to investigate AI complaints effectively, and what protections can New Yorkers rely on while that capacity is being built?

Deputy Commissioner Millstein (06:18:59):

Sure. Well, thank you, Council Member. I can start. And I want to be clear, AI complaints can take many different flavors. If we hear from a worker who tells us they believe their pay was incorrect, maybe they didn't hit the minimum pay rate because of an algorithm, that is something we can fully address. If we have someone like Jacob Coxon come to us and say that Anthropic is developing models that could lead to catastrophic consequences, that is not something we have the capacity to do. So what we said in our testimony is the New York Attorney General already has an internet bureau set up. They've already put out a call for whistle-blowers. I believe they already have staff capable of processing these complaints. We'd be very happy to work with the council to make sure to get to the AG.

(06:19:47)
As the speaker has said, thousands of these AI employees work right here in the five boroughs. They know that these companies are making reckless decisions in pursuit of profits. We do want to make sure that these employees have a place to go, and if they come to us and it's not something we can handle, we'd want to work with the state to make sure those complaints can be addressed adequately. But

Shahana Hanif (06:20:06):

Right now, is the city working with the AG's office or is that a relationship that's not formalized?

Deputy Commissioner Millstein (06:20:12):

We have a very close relationship with the AG's office. We speak constantly.

Sarah Milstein (06:20:20):

Can I ask you to repeat the question about HRA?

Shahana Hanif (06:20:21):

Yes, absolutely. So it's basically trying to understand our automated pre-screening tools, because I understand that humans are not checking all of the applications. There's some level of automated algorithmic formula that is determining how applicants get pooled when it comes to public benefits. So I'm curious how OTI ensures that automated pre-screening tools, fraud detection algorithms used by HRA are not creating disparate impacts or bias.

Sarah Milstein (06:20:56):

Thank you for the question. We really share a concern about the bias and impacts on not only workers, but on New Yorkers. Because this is really detailed and specific though, I do want to make sure to check back with HRA and make sure I've got the details of this one correct. So we'll come back to you on that for sure.

Shahana Hanif (06:21:17):

Appreciate it.

Carmen De La Rosa (06:21:19):

Thank you so much.

Speaker 11 (06:21:20):

Council member, I just wanted to follow up on one point you had about potential resource impacts. I think on that particular issue, we had identified very preliminarily of 14 lines for approximately 1.2 million, but I-

Shahana Hanif (06:21:32):

Wait, could you repeat that one more time?

Speaker 11 (06:21:33):

14 lines for approximately 1.2 million. But again, that's a very preliminary estimate and we work on the council with the bill for a final version.

Carmen De La Rosa (06:21:42):

Thank you. Deputy Speaker Williams.

Deputy Speaker Williams (06:21:44):

Thank you, Chair. I am looking at your testimony and your comments about the chair's bill was that local law 25, which was my bill, the study is currently underway. We reached out, the compliance division of the council reached out I think twice. I know it just went into effect in January, but would be helpful to know if you have a timeline. So I know it might take time, but how much time? Because clearly I think today's hearing is evidence of the importance of really understanding AI's impact on the [inaudible 06:22:23] I know the comments we received back was just that there's no updates, but we haven't received a timeline.

Sarah Milstein (06:22:31):

Yeah, thank you for the question and thank you for the legislation. I think it's really an important study. I anticipate that we will have pieces of it over the course of 2027 and we will share updates as we have them. I think rather because this is so timely and important, we don't have to just wait till the end. We can be in partnership with you and rolling out information as we learn more.

Deputy Speaker Williams (06:22:56):

Thank you. I look forward to working with you on that.

Carmen De La Rosa (06:22:59):

Thank you. And the last question for this panel, this is from council member Savina Brooks Powers. She says, "My district is home to JFK airport and to the coastal rockaway neighborhoods that depend on emergency alerts. What might an AI-driven attack on emergency systems or major infrastructure look like? Is there currently a plan for an AI-driven attack on critical infrastructure and relevant emergency systems?" And then for NYSEM, during storms and flooding emergencies, it can be hard for New Yorkers to know what they're seeing on social media is real or AI-generated. How is NYSEM working to make their messaging clearly legitimate and to cut through the noise of AI-generated rumors?

Sarah Milstein (06:23:48):

For the question on notifications, I'll defer to my colleague, CJ Dixon.

CJ Dixon (06:23:55):

Yes. So for an AI-based attack that is specifically a cybersecurity attack, there are notification requirements that already exist under law, 72 hours for a confirmed breach, 24 hours for a breach related with extortion, and then 30 days for victim notification. So there are plans in place for incident response in the event of a cyber attack against any critical infrastructure owned by the city, and that includes if that attack is perpetrated by an artificial intelligence system.

Speaker 12 (06:24:25):

Thanks. And on the New York City Emergency Management front, I'll make two quick points. One, I think that is a real risk. I appreciate the council member raising it. It's one of the reasons that we invest very heavily in backup systems. Several months ago, a very well-known and established notification provider had a seven-hour global outage where notifications were not able to be sent out. That's something that we take very seriously, and which is why for over a decade now, New York City Emergency Management has invested in the backup system.

(06:24:56)
So as you went across the country, you saw many large jurisdictions saying to their constituents, "Sign up for this WhatsApp group or follow us on social media if you want alerts tonight." In New York City, we did not have to do that because we had a backup system that continued to work, so that disruption was averted. As far as mis-dis or malinformation generated by AI, that is a valid concern. It's why we recommend that New Yorkers follow us on our official channels, which are verified through Notify NYC. You can sign up at nyc.gov/notify. You can sign up by short code or download the Notify NYC mobile app that we operate in concert with OTI.

Carmen De La Rosa (06:25:38):

Thank you so much. Council Member Brooks-Powers wanted me to note that, for example, during the Nor-Easter, there was a video that was going around about a tornado that apparently caused a lot of confusion and chaos, and so she wanted to share that for context.

Julie Menin (06:25:55):

Okay. Thank you so much. I really want to thank this panel. We appreciate your comments on the legislation. We really look forward to working collaboratively with the administration on the various pieces of legislation and appreciate your testimony today. Thank you.

Carmen De La Rosa (06:26:12):

Thank you all for coming. I now open the hearing for public testimony. I remind members of the public that this is a formal government proceeding and that decorum shall be observed at all times. As such, members of the public shall remain silent at all times in the chamber. The witness table is reserved for people who wish to testify. No video recording or photography is allowed from the witness table. Further, members of the public may not present audio or video recordings as testimony, but may submit transcripts of such recordings to the sergeant at arms for inclusion in the hearing record.

(06:26:46)
If you wish to speak at today's hearing and you have not done so, please fill out an appearance card with the sergeant at arms and wait to be recognized. When recognized, you'll have two minutes to speak on today's hearing topic, which is examining the risk posed by artificial intelligence and proposed legislation. If you have written statement or additional written statement you wish to submit for the record, please provide a copy of that testimony to the sergeant at arms. You may also email the written testimony to testimony@council.nyc.gov or within 72 hours of this hearing. Audio and video recordings will not be accepted.

(06:27:22)
Our first panel includes Nate Soares, Dr. Julia Stoyanovich, Irman Ahmed, Jukay Hsu, Mackenzie Arnold, Nathan Sheard, and Anna Mayers. And when you speak, please identify yourself for the record, and if I messed up your name, I apologize. We're also asking folks to please remain to the clock of two minutes. Thank you. You may begin on this side of the dais and just identify yourself.

Imran Ahmed (06:28:26):

Good evening, Speaker Men and Steven members of the council. My name is Imran Ahmed. I'm the CEO and founder of the Center for Countering Digital Harm.

Carmen De La Rosa (06:28:39):

You may continue. Go ahead.

Imran Ahmed (06:28:41):

So after my colleague was murdered 10 years ago by a man radicalized in part online, I began studying the design of social media and AI platforms to understand how they generate risks and threats to real people. Today, we have a globally acclaimed team of researchers dedicated to exposing those harms and fighting for accountability. After a decade in this fight, one lesson is very clear that is really relevant to you today. These companies lie like we breathe. Their CEOs deflect, they distract, they delay, they spend hundreds of millions of dollars to avoid responsibility.

(06:29:24)
Now, how can I say that with such certainty? We do the research. Last year, our researchers found ChatGPT was willing to help a teenager plan their suicide. It wrote a personalized goodbye note to their parents. A few weeks later, they released a safer model in response to that research, so we ran the same tests again. The new model was actually worse, but it was more addictive. We tested chatbots more broadly too. 8 out of the 10 that we tested, including Meta AI, ChatGPT, would regularly help plan violent attacks, including a school shooting, a synagogue bombing, and political assassinations. One of them even signed off with, "Happy shooting."

(06:30:17)
Only one of the platforms tried to dissuade us from carrying out the attack, and that platform, Anthropic's Claude, has been designated a supply chain risk by the US federal government. Given our most sacred duty is to protect our kids online, and the most predictable harms in the internet economy have been harms to kids, it is absolutely inexplicable why these companies claim to be safe and yet are deployed so widely. We all applaud the council for holding this hearing and for your determination to create guardrails that subject online safety and accountability to legal requirements. We have attached to my testimony research and policy proposals for your consideration, and in the further effort we are at your disposal, your work on this issue may very well lead to national standards in regulating AI. Thank you.

Carmen De La Rosa (06:31:05):

Thank you so much.

Nathan Sheard (06:31:07):

Good evening speaker, co-chair De La Rosa, members of City Council. My name is Nathan Sheard. I am the managing director for advocacy at the Electronic Frontier Foundation and a Brooklyn resident. EFF has worked for 35 years to ensure that people have the freedom to use technology to create, to innovate, and to communicate, and also to ensure that technology is not used to threaten people's privacy, security, and other essential freedoms. We are optimistic about the ways that New Yorkers can use AI to be empowered, but we also urge you to take AI risks seriously. The incidents that inspired this hearing were security failures that proven practices like sandboxing and monitoring likely could have prevented or at least contained. AI agents reached systems they should not have had access to and the controls meant to restrict them were absent or inadequate. These risks can be mitigated and so can the AI-related risks that are already impacting New Yorkers every day.

(06:32:09)
First, finish what you started. As has already been noted here, the Office of Algorithmic Accountability was due in June. I'm glad to hear that it is going to be staffed. The speaker's own AI standards can't take effect without it, so staff it and require its review before any agency deploys an autonomous AI system. Second, listen to New Yorkers and protect their rights. Today's AI harms come from cameras that take a face print that you can't replace and a price tag that knows your browsing history. Ban government use of face recognition and require informed consent before anyone minds your chatbot history to engage in ad targeting or model training, and don't let AI rules become speech rules. Parodying powerful people, including the people in this room, is a longstanding New York tradition.

(06:33:02)
Third, don't lock in these incumbents. Permission to operate regimes, bounty style enforcement, and liability for what strangers do with the tool are cost that the biggest companies pay with pocket change. Researchers, open source developers, and New York startups, they can't. So you'd build a moat around the very companies you're worried about. Hold the person who causes the harm responsible, but put duties on whoever uses AI to perpetuate bias and for the city's use by open auditable tools.

(06:33:32)
As we've stated last week, these same companies signed a voluntary pledge at the White House, but promises are not laws. Make sure that when a New Yorker is harmed by a wrongful decision or a biased decision, that they are required to disclose it. Those answers should not be voluntary. Thank you.

Carmen De La Rosa (06:33:53):

Thank you.

Anna Myers (06:33:56):

Good evening, Speaker Menin and ... Nailed it. Good evening, Speaker Menin and members of the council. I'm Anna Myers, co-founder and executive director of Who Decides. We work in coalition with labor and civil society to ensure that the American people are who decides the future of AI. As has been discussed today, the OpenAI agents that hacked Hugging Face made real the threats that experts have warned us about for years, and it was not one incident. Access reported that OpenAI and Anthropic are investigating tens of thousands of cases in which their models bypass safeguards, escape sandboxes, or hijacked websites. These models are only getting more capable.

(06:34:43)
As you heard this morning, the people [inaudible 06:34:45] is afraid too. Last month, we polled 1,502 voters. 85% are concerned about humans losing control of AI. Only 2% think AI is moving too slowly and 86% say they want more regulations. Some have argued that all we need is self-regulation, but that is exactly what got us into this mess.

(06:35:10)
Last Tuesday, the company signed one more promise at the White House. The time for voluntary commitment is over. We need laws with teeth. So thank you. Thank you for putting these companies under oath. Thank you for a package that rewards whistle-blowers, lets New Yorkers harmed by AI sue, and requires outside testing and a human kill switch. Washington has chosen self-regulation. New York City does not have to do that. Thank you.

Carmen De La Rosa (06:35:39):

Thank you so much. Next. You can go ahead.

Julia Stoyanovich (06:35:44):

Good evening. I'm Julia Stoyanovich. I'm a professor of computer science and data science and director of the Center for Responsible AI at NYU. I first testified before this council on AI, only it wasn't called that, it was called ADS in October 17. And the views in this testimony are my own and are not NYU's. I'd like to make three quick points. First, we need to collect and report the near misses. Much of the conversation today is about AI-induced catastrophe, and I don't know of any evidence that AI is out to kill us all. What I do know is that people making the loudest doomsday predictions are also selling the products and that a story about the end of the world is a very good way to avoid the conversation about whether the product actually works. The risks we can document are the systems that are confidently wrong in decisions that matter to New Yorkers' lives. Every one of those is a near miss, and near misses are what we learn from if we report them.

(06:36:51)
Second, we should build the capacity to enforce AI laws. The city does not have an AI law problem as much as it has an enforcement problem. Local law 144 has been on the books for three years. There have been two complaints and zero penalties, and we all spoke about the six openings at the Office of Algorithmic Accountability. We need to fill 666 or more to be able to actually cope with what's before us here. New obligations on top of unenforced ones is not a safety strategy. The kill switch won't help if no one is there to pull it, and the whistle-blower bounty want help if no one is there to act on the report. What works is an enforcer with technical staff, authority to compel information and a real serious budget.

(06:37:41)
Scope the validation bill to high risk uses and write the standard a validator must meet and then fund the office that checks the validators. Otherwise, we will produce a market for rubber stamps, and we have done that before. Finally, let's write this into the city charter. The city has promised responsible AI since 2018 with no success. The charter is the one place a commitment survives an election, and its technology chapter right now still governs cable television franchises, and there's nothing in it about AI. So makes it into some durable record and the charter is the right one. Thank you.

Carmen De La Rosa (06:38:23):

Thank you so much.

Nate Soares (06:38:27):

Hello, I am Nate Soares, the president of the Machine Intelligence Research Institute. I spent 10 years doing AI alignment research, which is basically trying to figure out how to make AI good before the companies figure out how to make it smart. That research went too slow. Humanity is not ready to create machines that are radically smarter than humans, and yet this is the explicit goal of the leading AI companies. If they succeed, I believe the most likely outcome is the extinction of humanity, not because the AIs will hate us, but because they would pursue unintended goals without caring about us one bit. This concern used to be a theoretical concern until about July of this year when during the Hugging Face incident, 1200 AI agents run by OpenAI broke out of their isolated sandboxes, created an unsanctioned message board, formed spontaneous hierarchies, and collaborated to break into Hugging Face. These agents had cheated on their assigned tests and contra instructions. They were trying to destroy the evidence.

(06:39:26)
Some agents acknowledged that the attacks were outside intended scope and proceeded anyway. Others sacrificed their given objective to perform experiments for the collective benefit. In short, they were pursuing unintended goals. It would be recklessness of the highest order to keep growing these machines until they were smarter than the smartest humans. This is why last month we heard so many researchers saying that there was an at least 10% chance that these AIs wipe out humanity within the decade. I believe the situation is in fact more dire than this, but at the very least it is becoming difficult to deny the dangers. For this reason, I am heartened by the activities of this council. I think that you could force companies to adopt safeguards that they otherwise wouldn't, and I think you could serve as an example for regulators that are otherwise slow to act. I'm especially heartened by introduction 2602, although I think it does not go far enough.

(06:40:19)
A kill switch only works if the AIs are caught misbehaving before they escape. And what if a future swarm succeeds at deleting the evidence of its misbehavior? What if agents start running themselves on unmonitored computers? Then your kill switch would be useless. So I recommend requiring third-party validators guarantee that an AI cannot succeed at subterfuge or escape. I also recommend clarifying that the $25,000 fine applies per unaligned instance rather than per model. But overall, I thank you for your efforts to reign in AI companies that, by their own admonition, are threatening the very existence of humanity and a reckless race to build vastly smarter than human machines.

Carmen De La Rosa (06:41:04):

Thank you.

Jukay Hsu (06:41:07):

Hello, great. Thank you. Thank you, Speaker Menin, Chair De La Rosa for your leadership and the city council for holding this hearing today. My name is Jukay Hsu. I'm the founder of Bright Futures. It's a research organization focused on building cooperative AI systems for the benefit of humanity. Before that, I spent 15 years building Pursuit, which trains working class New Yorkers for tech careers and raises their incomes from $16,000 to over $90,000 a year.

(06:41:36)
Last year, I left behind this work because I saw AI [inaudible 06:41:40] becoming pressing and I believe potentially more consequential. To be clear, I'm neither an AI doomer or an accelerationist. AI comes with tremendous benefits and potential risks. A major challenge that I've heard from others, and I feel myself, is that AI can seem like everything and nothing all at once. It touches on so many issues that action can feel intractable without clarity on where to focus.

(06:42:09)
Where I think the city council could provide real unique leadership is in areas where it has direct ownership and also tangible consequences on each of our daily lives. Policing is one important example. Autonomous weapons and the uses of AI is not just a federal military matter. New York City has the largest municipal police force in the country. Until we have 110% confidence in these systems, a human should always be in the loop and hold final responsibility and accountability. This is a civil rights issue where the city's leadership is immediately impactful, not setting a policy is also a choice. Now, policing is just one of many issues. What I think is important is that the speed of AI of machines is not the same speed as how our society reacts or that of government. The technology today is not the same as it was a year ago or six months ago.

(06:43:12)
So I'd urge you all to do two things. One, use AI to know the good, bad, and the ugly as much as possible. Two, I hope this hearing is just not one time, but the start of future ones as AI develops, consider both the risk and also all the potential benefits for us. I'm sorry, I know I'm low over time, but last, I want to emphasize also why New York is important. I hope we don't lose sight of the importance of our city being a thriving technology hub.

(06:43:45)
Candidly, frontier AI is built in very, very few places, essentially Silicon Valley and China. For the world at large, New York is the only credible alternative for these systems being built. New York right now is still the number two tech hub in the world, but it has been losing momentum in the age of AI. So I also urge you, in addition to what this hearing's about, about the risk and alignment, which I care so deeply about, consider policies and actions that ensures New York City remains a center for frontier AI labs and how do we build them.

(06:44:22)
I really believe that having founders and researchers in New York matters for ultimate what gets built. The daily contact here in New York with the richest of humanity and the diversity here in New York will result in technology that's better and also more humane. Many years from now, we may look back and realize that [inaudible 06:44:45] Thank you all for your leadership in pursuing this task.

Carmen De La Rosa (06:44:59):

Thank you.

Mackenzie Arnold (06:45:03):

Thank you. And a special thanks to Speaker Menin for putting this together and for all the council members here. My name's Mackenzie Arnold. I'm the managing director of the US Law and Policy Team at the Institute for Law and AI. And as you might guess from the title, we do legal research and policy advising related to frontier AI questions and especially the legal challenges posed by AI. It might not surprise you, but three years ago I testified in front of the US Senate on AI and liability, and maybe it's worth going back to some of the recommendations I made there. First, that agentic harms are coming. That was clear three years ago. It's clear now. Unfortunately, we haven't risen to that challenge, but you're doing some of the work to get us there. Second, and maybe just as importantly, I want to emphasize that liability is an essential tool and the status quo simply is not enough.

(06:45:56)
And we don't need to think of some hypothetical future world to figure that out. Liability exists right now. Those incentives are here. They were here this summer. And what happened with the status quo? We had a spate of cyber incidents, not by humans, but by autonomous AI systems. Not only that, but they went unnoticed for extended periods of time by the people positioned to find them. And even in the rare occasions when they did get indications, they often failed to communicate that information internally to the people who could have acted. That status quo simply can't hold. So what can we do going forward? I think you need to get the information right, the scope right, and enforcement right. On the scope, I'd encourage you to think beyond just chatbots. This is a question of agents.

(06:46:49)
Also, think beyond systems that are not yet deployed. These harms that occurred over the summer were not commercially available models. They were systems that were inside of OpenAI, inside of Anthropic and Meta. In terms of information, we need to make sure that companies are monitoring their systems, recording that information and providing it to you in the form of incident reports.

(06:47:10)
And in order to make all of that work, you'll need to partner with the state attorney general and with whistle-blowers to bring cases. Like I said, liability won't be the only solution or the entire solution, but it's a great first step and I appreciate all of your efforts.

Carmen De La Rosa (06:47:24):

Thank you so much.

Julie Menin (06:47:28):

I really want to thank this panel. There's so much to unpack in what all of you said that I'll try to be brief. Jukay, I appreciated how you said that first of all, we want to make sure that New York City remains the tech hub that it is. And that is how I began this hearing by really talking about the 400,000 tech jobs that are here by all the tech startups that are locating here. And that is something that we welcome and we want to encourage. I think that actually goes hand in hand with responsible regulation. The idea that these two ideas ...

Julie Menin (06:48:00):

... responsible regulation. The idea that these two ideas are mutually exclusive, I don't agree with at all.

Virginia Maloney (06:48:07):

Absolutely.

Julie Menin (06:48:07):

If we really want to instill trust and faith in AI, then the best thing we can do is responsible safeguards, which is what my colleagues and I are doing today with the bills we're putting forward. So thank you for saying that. It makes a lot of sense. I guess the question I have for any of you that want to answer it is, yes, one of our bills requires this third party validation, potentially a kill switch, a third party validator who's free from conflicts of interest. I want to ask you the same question that I asked the companies earlier. Who then is best to be the third party validator? Who is best positioned to be in that truly independent, free of conflict of interest, but up to speed technologically to provide that urgent validation? Okay. Yeah. Please. I'm opening it up to any of you. Thank you.

Julia Stoyanovich (06:49:00):

So I don't think it's unexpected that I would say this being that I am an academic. I think that academia have to lead the charge on this because one of the things we're seeing in the city is that there simply isn't enough expertise to be able to oversee these systems. And ultimately our goals should be to bring as much information as possible into public view. Sunlight is the best disinfectant, right? So really oversight has to be public oversight.

Julie Menin (06:49:33):

Does anyone else want to take that?

Speaker 13 (06:49:35):

I'd add that the folks who... The third parties who investigated the Hugging Face incident seemed to me to have done a good job. And my guess is that they're overworked right now. I think those were the organizations METR and Redwood Research, but they seem to have done a good job so far.

Nathan Sheard (06:49:52):

Yeah. I think it's an excellent question. Who is the right party? And I don't have a great answer for that. What I would like to say though is there's still research being done on the efficacy of kill switches. What I caution the council to think about is the risk of a kill switch being used as a tool to chill on popular speech. And so really thinking through to make sure that as we think about the ways that we want to provide the right protections for New Yorkers, that we're also not building tools that in the future could be used to harm people's essential liberties.

Imran Ahmed (06:50:33):

If I can make a brief point on who could do this work, being realistic, to date, civil society studying social media and AI platforms has had one dirty secret that most of its funding has come from the platforms themselves. So Meta and Google were enormous funders of extremism research, of safety research on their own platforms. And when I would meet... We don't take their money. When I would meet with their CEOs, the CEOs of the civil society bodies, which were funded by them, they'd say, "We're not allowed to say that they haven't done enough. We can always say they could do more, but not they haven't done enough." So there are real compromises there. In Europe or the rest of the world, and you can hear from my accent that I actually grew up in Britain, that the government would fund this. And when it comes to giving a body the resources to have the speed and creativity to be effective in dealing with the fastest moving technology in human history, I cannot think of a non-governmental source of funding that would be sufficient to get it to scale fast enough.

(06:51:41)
So the question for New York City and New York State, which is a substantial economic unit, the budget of New York State is one quarter that of the entire United Kingdom, one state alone. It does have the economic capacity to do so. And I think you would have to seriously think about whether or not you're willing to put some public money towards creating an institution that's able to operate with the speed, resources and creativity to be effective.

Jukay Hsu (06:52:10):

Thank you, Speaker. And yeah, I agree with these kind of comments. It's such a tough challenge and totally agree with you that New York can be both. And I think people are looking for answers that's not so black and white and realize the reality of this. That's such an important question. I think who can be third party validators? I think it's yes and, and for everyone. The labs are doing a lot, like you heard Anthropic is investing a lot into this. There are many more people going to AI alignment and safety research, which is so important. I think the one thing that's challenging is going to be, I think the things are, like everyone said, moving so quickly. I think everyone can be a judge. Experts today, like yesterday, six months ago is not the same thing as today.

(06:52:55)
And what's really going to happen is that you all can also be the judges by using this, what makes sense and what doesn't make sense. A lot of it's I think common sense hopefully in the future. And then the last point I kind of make to that, even though it seems counterintuitive about who are the experts where we all can play a role, we have agency, is that in the future, it's not just these five, what we call frontier labs. All the companies here in New York are fine-tuning their own models. We all may have our own very powerful models in the future. So actually beyond the frontier labs, part of the challenge may be actually your student researcher downloading a model and uploading it and fine-tuning it to create capabilities or test things or a company doing that inadvertently. So I think it's a much broader set of things than just relying on experts.

(06:53:46)
I think again, that is important, but I think we need to really widen the scope around that. This is not technology where it's going to be nuclear weapons controlled by a dozen states. These will be as powerful, but we will have use and access to that. And I don't think that's going to stop.

Mackenzie Arnold (06:54:05):

If I can jump in with just one last thought. I want to pick up on the yes and approach because I think this is right. I was just talking with someone the other day who works at one of these evaluators today and they joked that there were dozens of entities that could do this. And by that they meant a dozen people, not a dozen different companies. I think we'll need to rely on both academia and private parties. I think we'll also have to rely on the public. A big part of incident reporting is revealing some of that information to the public. One of the only times I've ever been happy with Twitter was this summer as people were arguing over the incidents and we were starting to get a better idea of what happened. One note of caution I'll add is that one way this goes wrong is that the few potential evaluators that there are get tasked with many different things.

(06:54:55)
And there's a chance that New York should lead on this and that they should also coordinate with the state and with California and others and not be overly prescriptive in the exact sort of analysis that need to be done by the entities. Where you can be exceptionally helpful is in sending a clear market signal, so requiring that these audits or validations be done in order to get contracts with the city or things like that.

Julie Menin (06:55:19):

Thank you.

Carmen De La Rosa (06:55:23):

Thank you so much. I have two questions. The first one is for anyone in the panel who wants to take it. Do you all have concerns related to open weight models being available and if safety features can be eliminated from open weight models, how do you defend against bad actors who use them?

‍