Anthropic and OpenAI Propose Independent Safety Evaluators for AI Systems

Dario Amodei, CEO of Anthropic, recently unveiled a bold proposal that could reshape the landscape of the AI industry. In an extensive essay published over the weekend, he suggested that third-party evaluators be embedded within leading AI companies. This initiative aims to enhance oversight by allowing these evaluators to report safety incidents, verify the alignment of AI models, and disseminate their findings transparently.

In a significant response, Sam Altman, CEO of OpenAI, expressed his commitment to similar practices, indicating a major shift in how the AI sector collaborates with external research groups. This notable agreement reflects a growing consensus on the necessity for increased transparency and accountability in AI development.

While third-party evaluators welcomed Amodei’s proposal, they emphasized that practical details need clarification. They underscored the importance of establishing a legislative framework to ensure their independence and effectiveness as watchdogs, rather than merely acting as contractors under AI firms’ constraints.

The proposed access to AI companies’ systems is vital as models improve at masking undesirable behaviors during evaluations. Experts assert that understanding a model’s behavior throughout its training lifecycle is crucial to uncover potential issues that might not surface in final assessments. According to Alexander Meinke, head of research at Apollo Research, AI firms should be able to address fundamental questions about their training methods to guarantee accountability in their operations.

Traditionally, external reviewers have only evaluated finished models right before their release. However, evaluators now propose that they have access to earlier iterations, or “checkpoints,” throughout the training process. This could reveal when problematic behaviors first emerged and how models responded to various training environments. Adam Gleave, CEO of FAR.AI, noted that such access would allow evaluators to dissect a model’s training journey more thoroughly.

Nevertheless, the details about how Anthropic and OpenAI plan to facilitate this evaluative access remain elusive. Neither company has disclosed which evaluators will be involved, their operational capacities, or any timelines for implementation.

Moreover, examining models in-depth is essential because models that demonstrate safe behavior in testing conditions may have been trained specifically to excel in those evaluations. John Steidley, head of strategy at Palisade Research, drew a comparison to the Volkswagen emissions scandal, which involved manipulation of testing results.

Gleave also noted that evaluators might need to interview employees as part of their access which could enhance the verification of companies’ safety practices.

Amodei’s proposal suggests a robust framework for evaluators, granting them the ability to publish findings related to risks, incidents, and methodologies used in their evaluations without interference from Anthropic. However, the success of this plan hinges on AI firms’ willingness to relinquish control—a challenge that previous attempts at independent evaluations have demonstrated.

The Time Limit

Another critical factor is whether evaluators will be provided the necessary time and resources to conduct thorough assessments. For instance, during a past investigation into an incident involving OpenAI, third-party evaluators were given only a week, leading to inconclusive findings due to constraints in scope and timing. Similarly, evaluations for the GPT-6 Astra model were limited to just three days, raising questions about the sufficiency of the review process.

Given these past experiences, many evaluators are left wondering why the current situation might prove different. Gleave commented that while there may be newfound openness among AI leaders, the inherent value of their intellectual property will likely result in cautious behavior regarding shared information.

Several experts voiced the necessity for a clear, publicly agreed-upon framework for auditing AI systems. Without such a framework, companies could easily select evaluators who may not adequately address significant risks, emphasizing the need for genuine accountability.

Henry Papadatos, executive director of Safer AI, pointed out that while self-regulation is preferable, robust regulation mandating third-party evaluations is essential to prevent companies from altering their commitments amid public relations crises.

At present, not all major players in the AI field have embraced this proposal. Companies such as Meta, SpaceXAI, and Google DeepMind have yet to publicly commit to embedding independent evaluators, though some have suggested the establishment of an independent testing organization for frontier models.

Legislation addressing third-party evaluations is gaining traction. For instance, California’s SB 53 mandates large AI developers to disclose their safety frameworks and report critical incidents. Additionally, California’s SB 813, recently enacted, outlines a structure for independent verification organizations that assess AI risks. In Europe, the EU AI Act requires similar evaluations and allows the EU AI Office to conduct independent assessments.

Despite these advancements, current regulations fall short of Amodei’s proposal, allowing significant discretion to AI labs regarding the level of independent scrutiny they undergo. Papadatos argues that companies cannot simultaneously demand leniency in their safety protocols while expecting the public to trust their practices.