‘A catastrophic risk to humanity’: New York is pushing back against AI

“Artificial intelligence is evolving faster than any technology in human history,” reads a bill recently passed by lawmakers in New York, called the RAISE Act. The rapidly developing technology, it says, carries some “very severe risks” (we know...) that existing laws have failed to keep up with. The RAISE Act aims to change this with a series of legal measures aimed at frontier AI models made by top companies like OpenAI, Google, Meta, and Anthropic. However, as always, the plan has attracted some harsh criticism as well.

First, though, what does the bill actually propose? Well, it revolves around three main goals: establishing strict safety plans to prevent severe risks from AI, enforcing the publication of these safety plans by top AI companies, and making them disclose major security incidents “so that no one has to make the same mistake twice”. If it passes, these requirements will apply to companies that train their models using $100 million or more in computing power, and make their tech available to New York residents.

You might also be wondering what lawmakers mean by “severe risks”. Luckily, they spell it out pretty clearly: a severe risk is one that could cost over $1 billion in damages, or cause hundreds of deaths or injuries. “For these kinds of risks,” reads the RAISE Act, “this bill is the bare minimum that New Yorkers expect.” As spelled out in the bill, concrete examples of the risks could include devastating cyberattacks, bioweapons designed with the help of AI, and – in the worst-case scenario – full-scale extinction by AI.

Given the potential dangers, it seems pretty sane to put these kind of (limited) safeguards in place. Still, it’s not surprising that some in Silicon Valley are pushing back. “The NY RAISE Act is yet another stupid, stupid state-level AI bill that will only hurt the US at a time when our adversaries are racing ahead,” writes Anjney Midha, a general partner at Andreessen Horowitz (a venture capital firm founded on a tech accelerationist philosophy) on X. The company has also been pressuring lawmakers to oppose the bill, according to NY assembly member and sponsor of the RAISE Act Alex Bores.

Key figures at Anthropic, a leading AI lab with a long-stated focus on developing safe artificial intelligence, also raised some concerns about the new bill. These mostly come via a thread from co-founder Jack Clark, who states that the RAISE Act is “overly broad/unclear” and doesn’t lay out a way to correct a deficient safety plan before the consequences – AKA, penalties of up to $30 million – kick in. He also suggests that steep fines for minor and accidental rule-breaking could adversely affect smaller companies (although, if the future of the human race is at stake, this seems like a relatively small price to pay).

The bill itself is designed to address some of these issues, building on the widely criticised SB 1047, a vetoed AI law from 2024, which many claimed would stifle innovation and detract from the positive benefits of AI. The RAISE Act lists some of these benefits, including “life-changing medicines, unlocking new creative potential, and automating mundane tasks”. But, it argues, we need to weigh these opportunities against the “catastrophic risks to humanity”.

The RAISE Act was successfully passed by New York state lawmakers last week, and will now need to be signed into law by New York Governor Kathy Hochul. Otherwise, it could require some amendments or get vetoed altogether. If it does pass, though, it will represent the biggest set of transparency standards to hit US AI developers so far (i.e. good news for the long-term survival of our species). Some are concerned that Big Tech companies simply won’t let New Yorkers access their AI products any more, but that would be a huge sacrifice to make.