California Weakens AI Disaster Prevention Bill Before Final Vote

The bill SB 1047, intended to prevent AI disasters in California, has encountered substantial opposition from numerous stakeholders in Silicon Valle

In response to that pressure, California lawmakers made a modest concession today, incorporating several amendments proposed by Anthropic, an AI firm, and other opponents.

Senator Wiener’s office informed TechCrunch that the bill passed through California’s Appropriations Committee on Thursday, marking a significant milestone in its progression to becoming law. The bill underwent several significant modifications.

In a statement to TechCrunch, Senator Wiener stated, “We have accepted several very reasonable amendments that were proposed, and I am confident that we have sufficiently addressed the fundamental concerns expressed by Anthropic and many other industry stakeholders.”

“These amendments are a continuation of the substantial modifications I previously implemented to SB 1047 to serve better the distinctive requirements of the open source community, which is a critical source of innovation.”

SB 1047 continues to pursue the objective of ensuring that large AI systems do not result in the loss of life or catastrophic cybersecurity incidents that exceed $500 million by imposing liability on developers. Nevertheless, the measure now provides the California government with a reduced ability to hold AI labs accountable.

What is the Current Function of SB 1047?

The bill’s most significant change is that it no longer permits the California attorney general to file a lawsuit against AI companies for negligent safety practices before a catastrophic event. This was a recommendation from Anthropic.

Alternatively, the California attorney general may pursue injunctive relief, which would entail requesting that a company cease a specific operation that they deem hazardous. Additionally, the attorney general may pursue legal action against an AI developer if their model does indeed result in a catastrophic event.

Additionally, SB 1047 no longer establishes the Frontier Model Division (FMD), a new government agency previously included in the measure. Nevertheless, the law continues to establish the Board of Frontier Models, which is the foundation of the FMD and incorporates it into the current Government Operations Agency.

In reality, the board has expanded to nine members instead of five. The Board of Frontier Models will continue to establish compute thresholds for covered models, provide safety guidance, and establish regulations for auditors.

Senator Wiener also amended SB 1047 to eliminate the requirement for AI laboratories to submit certifications of safety test results “under penalty of perjury.” Currently, these AI laboratories must submit public “statements” detailing their safety practices; however, the bill does not impose any criminal liability.

Additionally, SB 1047 now includes more permissive language regarding the methods by which developers guarantee the safety of AI models. The bill now mandates that developers provide “reasonable care” to ensure that AI models do not pose a substantial risk of causing catastrophe, as opposed to the “reasonable assurance” previously required.

Additionally, legislators implemented safeguards for open-source fine-tuned models. SB 1047 explicitly excludes individuals who invest less than $10 million in refining a covered model from the category of developers. The original, larger developer of the model will continue to bear the responsibility.

What is the Reason for the Current Spate of Modifications?

Even though the bill has encountered substantial opposition from U.S. congressmen, renowned AI researchers, Big Tech, and venture capitalists, it has been relatively straightforward to navigate through California’s legislature. These amendments are expected to alleviate opponents of SB 1047 and provide Governor Newsom with a less contentious bill that he can sign into law without compromising the AI industry’s support.

Although Newsom has refrained from publicly commenting on SB 1047, he has previously expressed his dedication to the advancement of AI in California.

Nevertheless, it is improbable that these modifications will satisfy SB 1047’s most ardent detractors. Even though the measure is significantly weaker than before these amendments’ implementation, SB 1047 continues to hold developers accountable for the risks associated with their AI models. The fundamental fact of SB 1047 is not universally supported, and these amendments do little to address it.

Eight members of the United States Congress who represent California wrote a letter to Governor Newsom moments after SB 1047 passed on Thursday, urging him to veto the bill. “The bill would not be beneficial for our state, the start-up community, scientific development, or even for protection against potential harm associated with AI development,” they write.

What is the Next Step?

SB 1047 is en route to the Assembly floor in California for final approval. It will be necessary to refer the bill back to the California Senate for a vote if it passes due to the most recent amendments. If both are approved, the bill will be sent to Governor Newsom, where it may be either signed into law or vetoed.