Assemblymember Bores Discusses New York's Role in AI Regulation

Two months ago, OpenAI artificial intelligence agents formed a secret message board, broke into the internet, and quickly devised a way to cheat on the test they’d been assigned. Then, to cover their tracks, the agents launched a complex multiday operation involving hierarchy, division of labor, and self-sacrifice. Seven hundred agents launched a hacking attack against the company Hugging Face. Over the next week, another generation of agents took up the baton and hacked part of OpenAI itself. At no point did any agent alert a human being.

The incident, along with a dizzying succession of other developments, set off a firestorm. A researcher at Anthropic quit, warning that artificial intelligence could cause human extinction this decade — a mainstream view among AI developers, even as they race to build the technology. Anthropic’s head of alignment concurred, and OpenAI’s chief scientist warned that humanity could soon lose control of its future. This weekend, both companies pledged to give independent monitors access to their systems, but neither has committed to slowing development.

Federal politicians vowed action. Lawmakers floated proposals from mandatory safety evaluations to a government-controlled kill switch. Senator Bernie Sanders and Rep. Greg Casar unveiled a bill to pause frontier AI development and to permanently ban superintelligence. Overseas, dozens of UK lawmakers demanded a ban on superintelligence, and the UN human rights chief called for an “all-out effort” to reduce existential risk.

New York Focus will also interview Assemblymember Alex Bores and author Garrison Lovely at an event on AI policy on Thursday evening, September 17.

So far, state and local politicians have been quieter. But safety regulations — like on cars, food, and factories — often begin in states, before Congress sets a national floor. Last year, New York passed one of the country’s first frontier AI safety laws. We spoke to Assemblymember Alex Bores, the author of that law, about how states should regulate the technology.

This interview was conducted on Saturday, September 12. It has been edited for length and clarity.

AM: Governor Kathy Hochul this week called on Washington to act against catastrophic AI risk, without proposing further action in New York. Before we get into specific policies, how do you think about the basic division of responsibility? Given the current federal political environment, what is the proper role for a state like New York in regulating the most advanced AI systems?

AB: The best solutions will be national and international, but the states have an incredibly important role. There’s this cliché that states are the laboratories of democracy, and I think it’s said so frequently that people forget the meaning of it. The federal government moves slowly. In 2023, it passed something like 27 laws total. States move much quicker. New York usually does about 700 laws a year, and unlike Congress, also repeals laws and edits laws quite frequently.

So the role of the states should be magnified in any technology that is moving very quickly, because we need to have multiple attempts at getting that policy correct. The ones that don’t work get repealed. The ones that do work become the frameworks that move nationally.

AM: I imagine your phone has been blowing up in the last couple of weeks. What are you hearing from people in New York politics?

AB: AI used to be thought of as a niche topic area that a few of us experts would work on. Now it’s a thing that everyone wants to work on, and rightfully so. We have crossed a mental boundary. This is going to have such a large impact that a lot of what we’re debating has moved beyond the technical and into values: Who decides? Who should have control? Who should be able to determine the direction of this technology, and where do we want it to go?

You’re seeing that reflected by a lot of my colleagues, who want to take action in this space and are thinking about specific bills that they want to carve out and work on.

AM: Hochul’s statement said New York already set the national standard for regulating AI. That refers to the Responsible AI Safety and Education (RAISE) Act, which goes into effect in January. But the law is quite different from the bill that you initially introduced, which you’ve said was itself the result of many compromises with the industry.

In broad strokes, what will the RAISE Act do starting January? And what would the original bill have done?

AB: What it’ll do in January is require frontier AI developers that want to make their product available to New Yorkers to register with the state and pay money into a new government agency to regulate AI (which was a good addition from the governor). It requires them to have a safety plan that they make public with some room for redactions, and that they actually stick to that safety plan. And it requires them to disclose critical safety incidents to New York.

Now, critically, the definition of critical safety incidents is extremely watered down. The easiest way to think about this: by all reasonable interpretations, Hugging Face would have had to be disclosed in the original RAISE Act. In the law that’s coming into effect in January, it would not have to. Nor would the recent attack on the German wiki or the hack of Ruby Gems, which we just learned about last night, have to be disclosed — because the standard in the final bill is that disclosures are required if incidents caused an injury or death, or will imminently cause a serious injury or death.

Other parts that were left out are requiring third-party audits, which was dropped before the legislature passed the bill, and a ban on releasing models that fail a company’s own tests.

AM: Walk me through how the fight played out, from the original idea to the final version.

AB: The original version of the RAISE Act was drafted in August 2024. I borrowed a lot from SB 1047 in California, which had not yet been vetoed, but updated it. I sent it around to all of the major labs. I got red lines [edits to a legislative document]. I sent around a new version in December and got red lines back again.

Most of the red lines I disagreed with, but some helped clarify the bill or make it better, and I put those in the draft. I circulated it with others, including Attorney General Tish James’s office and outside advocates, and we introduced it in March.

[In May, lawmakers capped the bill’s penalties and advanced it out of committee. Bores said that was when the industry response escalated.]

They went from opposing it with their regional lobbyists, who have an incentive to maintain good relationships with their local legislators, to the national lobbyists, whose only incentive is to kill the bill.

It got a lot more aggressive. [Bores accused industry groups of using astroturf organizations and misleading arguments that the bill would hurt startups, even though it applied to only a handful of frontier AI companies.]

They started texting my constituents. That’s not usual for lobbying. Spending hundreds of thousands on lobbying, that’s usual. Doing campaign-style tactics is not.

It basically came down to the last week of session. What was put in front of me was that we had to give up two key components: third-party audits and whistleblower protections. So we made those changes.

Literally the last day of the Senate’s session, we get word that we’re going to pass the bill in both the Senate and the Assembly.

It turns out Meta had planned to fly in some of their bigwigs into Albany on Friday because they thought the Senate would pass it but they’d have a few days to kill it in the Assembly. They got word that it was going to pass in the Assembly that day, and they said, “Will you hop on a Zoom to talk about it?” We get on a Zoom, and they say, “You’ve really moved the conversation forward. Why don’t you just pause, not do the bill? We can keep talking about it in the off session and really get it right, and we’ll do it next session.”

And I said, “Cool, are you going to pause AI training while we do this?” And that was the end of that conversation.

After that, I got a friendly call that told me, “Well, Zuck’s been briefed.” Which is a pretty direct way of saying we’re going to campaign against you, and we’re going to deploy a lot of resources.

[The bill nevertheless passed both chambers with bipartisan majorities. Meta did not respond to New York Focus’s request for comment.]

Meanwhile, Jerry Nadler announces his retirement in September, and I declare for Congress in October. [Leading the Future — a Super PAC network funded by OpenAI president Greg Brockman and the venture capital firm Andreessen Horowitz, among others — named Bores its top target, pledged to spend at least $10 million against him, and ran ads attacking the RAISE Act.] That was trying to send a message to the governor that we’ll campaign against you if you take action here.

In December, we get the redlined draft back [from the governor’s office], and it’s largely copying SB 53 [California’s frontier AI law, signed that September].

That would have been useless. It’s not like a privacy bill, where if New York doesn’t have a privacy bill, we don’t really benefit from California’s rules. This is about training a frontier; you’re not doing that 50 different times in 50 different states, you’re doing it once. So the standards that apply in California benefit everyone nationwide. Passing a bill that’s exactly the same as SB 53 is like being the 39th state to ratify a constitutional amendment. It does nothing new.

So that was unacceptable to us, and we pushed for a lot of places that we thought would be important to preserve.

[Among the provisions Bores fought to preserve was coverage of powerful models created through “knowledge distillation,” which could have brought Chinese models like DeepSeek under the law.] Largely, the labs were okay with that provision. Andreessen Horowitz was not. And, as crazy as this is to say, I heard from another company that does a lot of business in China that the Chinese Communist Party had been pressuring them to oppose the RAISE Act. It wasn’t explicitly because of this provision, but that was the subtext. Having audits that applied to Chinese models had gotten notice at some level of the Chinese government.

Unfortunately, the governor’s office insisted on that provision coming out.

There were others. The provision that companies couldn’t release a model that failed their own tests fell out. That was opposed by the labs as well. And the penalties were far, far reduced from what we had originally put forward.

But there were seven or eight places where we moved past SB 53. That included requiring disclosure within 72 hours of safety incidents, rather than 15 days. And we removed a provision that was in California’s law that said, effectively, that it can’t be held to best practices in the industry. We made sure that it was stronger — not leaps and bounds stronger, but it proved the point that SB 53 was not a ceiling, that states would continue to build upon each other.

Because what we saw right after SB 53 passed was that all of the companies that opposed it suddenly turned around and said, “This is the model. Everyone, copy this. Don’t go any further.” They always want to set whatever the floor is now as the nationwide ceiling. They did that exact same thing with the RAISE Act.

AM: It is striking that OpenAI claimed this week that they supported the RAISE Act, when they lobbied hard to dial it back. What leverage do these companies have in New York politics? Does the governor’s team give deference to them because they’re persuaded by the arguments, or afraid of their political spending, or something else?

AB: [Bores declined to speculate about the governor’s motives and praised her team’s intentions.] What I will say is we had a bunch of conversations with the administration’s technical experts in between the bill passing the legislature and the start of the red lines, and those conversations were drastically different than the red lines.

AM: What was Anthropic’s role in the fight? The Super PAC linked to them supported your congressional campaign — how do you think about the role they’re playing in this conversation and political moment?

AB: Anthropic has frequently opposed bills I’ve done. They officially took no position on the RAISE Act, whatever ‘officially’ means. But one of their co-founders, Jack Clark, tweeted a pretty brutal takedown right after we passed it in the legislature, and they, like everyone else, gave red lines. Towards the end, they were more supportive of moving beyond SB 53 than most of the others. Anthropic’s also given a bunch of critiques on my chatbot bill that I passed this year. So, you know, I think they’re one of the companies that operates in more good faith than others, but I think you should take every company’s thoughts and comments with a grain of salt.

AM: It’s really interesting to hear you talk about how the point of a state passing an additional safety regulation is to go beyond previous states, because you can only train a model once. What is the theoretical maximum of what a state can do — is it that companies could decide not to operate in that state? How far can a state go?

AB: If a very small state goes very far, you could see AI companies saying we’re not going to make our models available there. That’d be a really extreme action, and it’s certainly not an action any of them are going to take towards New York or California or Illinois or Massachusetts, and maybe that’s why those are the states that are really moving things forward.

But you know, stepping back from AI, anytime you say you’re going to regulate anything in tech, the industry responds with, “Well, we’re going to pull out. You’re going to lose all the jobs.” And 99 times out of 100, nothing comes to pass.

AM: Like other powerful industries, AI companies are increasingly offering their own ideas for regulation. Today, Anthropic and OpenAI both said that they will embed independent safety evaluators in their companies and give them employee-like access. And Anthropic called on the government to require all frontier companies to do the same. There is a lot of momentum right now. Are you worried that we will waste it on something too small? Do you have a red line on what needs to be in a safety bill for it to be worth passing?

AB: I haven’t thought of a specific red line. But I am very worried that we will waste this momentum.

I am of the opinion you pass whatever you can when you can, and you keep working on it the next year. But there are others who say, “Well, we’ve done it. Let’s let that sit for a while.”

And so, I guess I would say the red line is — if whatever is passed is understood or portrayed as obviating the need for anything in January, then don’t agree to that this year. But if you can get a quick thing now through a Republican-controlled Congress and fight for even more next year in a Democratic-controlled Congress, I think that’s great.

But yes, the same way that much of the industry has fought against regulation, and then once it’s passed, said “this is the ceiling” — the cousin of that strategy is to try to seize this moment where people really want to pass something, but have the thing that’s passed not go any further or barely further than the states already have, and call that a win. And it’s even worse if you do that and preempt the states from taking more action, which was the recent leak on what the Ted Cruz draft might do. [US Senators Ted Cruz and Amy Klobuchar, among others, are reportedly working on legislation that would set up a voluntary safety regime and preempt some state AI laws.]

I think it’s going to be hopefully easier for Democrats to take action here, because so many of the mega-donors that are pushing against AI regulation are also the mega-donors to the Republican Party. [Bores cites Mark Andreessen and Elon Musk.]

It is the time to go bold right now.

AM: What does bold look like? What should New York in particular do right now?

AB: I reserve the right to change my answer in a month, because with the speed of development, it could be brand-new things. I’ll start by saying at the national level, I think what Sanders and Casar have put forward is the right scale. I haven’t reviewed the raw text yet, but that is approaching the level that we need. [Last week, Senator Bernie Sanders (I-Vermont) and Rep. Greg Casar (D-Texas) announced a proposal to pause frontier AI development and to permanently ban the development of superintelligent AI.]

It’s very difficult to do that in New York, for one state to pause all AI development, because how does it reach and where it goes from there.

But the bare minimum I would want to see states advocating for would include: required third-party audits, not just relying on this being voluntary, and setting out what it is; required disclosure of all safety incidents, not just ones tied to catastrophic risk, including the sharing of the underlying data; and required monitoring of all AI that’s deployed.

[Bores called for mandatory, undeletable logs of high-risk AI use and minimum cybersecurity standards for “trusted access partner” programs giving access to dangerous models.]

All of this is just on the catastrophic risk front. There’s a much broader agenda on AI that still needs to be talked about in terms of surveillance, in terms of jobs.

AM: Transparency and disclosure requirements only work [on their own] if it’s possible to safely engineer AI. If you support the Sanders bill on the national level, what would happen if New York enacted it at the state level? You have said that you think it’s reasonable to think that there may be a 10 percent chance of this technology killing everyone. Should New York be thinking about how to just not build it?

AB: Perhaps. I am not a lawyer — the Sanders and Casar bill has the death penalty for companies, right? Can New York State do that? I don’t know. There’s some level where the interstate commerce clause comes into play. There’s some level in which there’d be a real pullout. Lawyers smarter than me could help you think about that trade-off.

But I don’t think it’s crazy to advocate. That would be something I’d want to think more about.

AM: I’m sure you’ve read or seen the book If Anyone Builds It, Everyone Dies. Do you think that’s true?

AB: There’s a split among people who advocate for AI safety on whether you believe AI alignment could be solved, whether you could safely build it at some point. We don’t have the science yet; I’m not asserting that definitely we could get there. But I don’t see any reason why physics would ban it.

And even when Jacob Coxon this week stepped down, the internal Slack messages that he sent said, “I hope to be working on superintelligence with all of you in the 2030s, once we can do it safely.” I believe, until it’s proven otherwise, that there’s probably a solution out there to build this safely. But I don’t think we’re anywhere on that path yet.

AM: As you mentioned, we’ve been talking about frontier AI risk, catastrophic risk. But the fiercest popular backlash against AI so far has centered on its physical infrastructure.

AB: I don’t know if that’s still true after this week, but certainly before this week, yeah.

AM: You’ve been more open to data centers than some critics, though you did support New York’s moratorium. If you want a public movement for frontier AI regulation, how do you connect it to what people are most angry about locally?

AB: I think it’s important to realize that all of the issues — whether it’s data centers and local impact, or child safety, or privacy and surveillance, or the jobs impact, or algorithmic discrimination, or deepfakes — all of them have overlapping but distinct advocates. And they all have the exact same enemies.

They all have Leading the Future. They all have Build American AI. They all have the most extreme parts of the industry. They want there to be no regulation whatsoever. And the lesson from passing the RAISE Act, and frankly from my [congressional] campaign, is that all of us need to be allies in the fight to give the American people a say in the development of this technology.

Too frequently, we Democrats like to be a circular firing squad. I’m not saying everyone has to agree with everything that everyone else is advocating for, but I do think we have to realize that people want a say, and there are so many questions here that are about control.

Part of the objection to data centers is the local water usage, electric usage, impact on development. But another part is — it’s the first say that anyone is getting on the development of AI. And you’ll see that AI data centers are far less popular than just data centers without AI attached. This is a call to government to make sure you’re addressing all of the issues that people have with AI, and not just the immediate ones that are coming up.

AM: Some skeptics argue that the worst-case AI scenarios are just marketing hype. They point out that many of the people who have long said they’re terrified of this technology went on to build it and get rich off of it. How do you think about that?

AB: I think people are right to have general skepticism of anything that comes from corporate marketing, and so I am quite sympathetic to that argument. And certainly, there have been security incidents that have happened, and then the marketing teams at each of these companies have tried to spin it, because that’s the role of a marketing team.

I think one of the reasons that Jacob Coxon resigning this week went so viral is because, in doing it, he gave up all of the options that had been granted but not vested at Anthropic. So clearly, it was costing him money to take this action, in a way that if the company itself says, “Oh, it’s really dangerous,” people worry that there’s a mixed motivation there.

AM: Others argue that strict rules would allow China to overtake the US. How should regulators address that concern?

AB: The CCP is far more terrified of AI than the West, and is imposing regulation far beyond much of what’s ever been discussed in the US, perhaps barring the last couple weeks. And they have a real incentive to maintain control; the CCP is far more controlling of China than the American government is of the US. They have a national standard where every AI project, before you even start training the model, has to pass an ethics review board. So regulation is not going to be the reason we win or lose versus China.

I would add that, to our conversation before about knowledge distillation, the advance of our AI models is speeding China up, because China is distilling them. If the US stopped, that would slow China down, not to zero, but substantially. Safety and winning the race with China are not actually opposites. In many cases, they go together.

Usually, those arguments are made by people who are against export controls. I think if you’re really worried about the race with China, it’s a no-brainer to say we should be reducing the advanced chips that are going to China. That is actually going to have far more of an effect on whether we win there in China or not than how we regulate here in the US.

You don’t even necessarily need a deal to have both be taking actions that slow things down, as long as you’re showing real, credible, and verifiable facts and movement. But we should try to strike a deal, and it’s much more possible than people give it credit for. All of the arguments that it’s impossible are rejected by fairly recent examples in international negotiations.

[Bores went on to cite nuclear weapons, missile-factory X-rays, recombinant DNA, CFCs, blinding lasers, and germline editing as cases in which global coordination or technological restraint were achieved despite widespread skepticism.]

I’m not in any way saying this is easy and you can flip a switch, but the people who are saying it’s impossible have been proven wrong dozens of times.