OpenAI pens AI safety agreement with four firms

Under pressure after rogue AI incidents, OpenAI details plans for external evaluators

Close-up image of OpenAI logo and branding displayed on a smartphone screen.
(Image credit: Getty Images)

OpenAI has revealed four companies that will get access to its labs to ensure its AI models are doing as they should.

The move follows revelations over the summer that OpenAI's agents went rogue and breached Hugging Face, which sparked a wave of admissions of similar model misalignment — including hacking an Australian government website.

This week, the state of California has filed suit against the AI company, Attorney general Rob Bonta said the "infamous" Hugging Face incident was just one of 80 that showed a "loss of control" and an "internal culture at OpenAI that prioritizes growth and rapid model development over safety and accountability."

OpenAI has since announced it has signed agreements with four independent evaluators – Apollo Research, METR, Transluce, and Goodfire – for the first stage of its assessment plans.

Latest Videos FromIT Pro

"As AI capabilities advance rapidly and the potential risks grow, public reporting from independent assessments can give people a clearer view of how labs are evaluating and managing those risks," OpenAI said in a blog post.

OpenAI opens up for safety tester

OpenAI said it would give those independent experts "deep access" to the internal models and systems running in its own labs. This will enable it to keep watch over model alignment – the industry's way of referring to whether AI does as it should – and how effective guardrails and other safeguards are over longer periods of time.

"Based on what each evaluation needs, assessors receive access to some of the same internal resources our researchers use, including model checkpoints, source code, training and evaluation data, monitoring systems, technical documentation, and discussions with the researchers and engineers responsible for the work," OpenAI added in the post.

"This access gives assessors the evidence and context to pursue independent evaluations and uncover failures we may not have anticipated."

The assessments aren't the first run by OpenAI, with the company pointing to its "years of work with third party evaluators", which included an investigation following the Hugging Face incident. However, OpenAI suggested that the plan is to "deepen" that work.

That said, as OpenAI admits, its hand was forced by political pressure.

"These efforts also support our commitments under the White House Accord on Super Intelligence, where we agreed to independent assessments of our safety controls," the post added.

What they'll look for

OpenAI broke down what each company will be working on inside its own labs.

Apollo Research will have access to selected models – including some not yet publicly released – as well as training data, flagged behaviors and intervention records.

This will help decide if the models could "covertly work against their developers' intentions". And if so, Apollo will assess if OpenAI's existing safeguards would detect that behaviour.

Goodfire researchers will work at OpenAI to analyse models during training runs for debugging and alignment. While the "primary audience" of Goodfire's work is OpenAI's own researchers, the company will also publish what it discovers.

METR will run two separate assessments, looking at alignment and monitoring, focusing on OpenAI's largest reinforcement learning run. Transluce will look at OpenAI's own post-training alignment evaluations to see what's missing or could be improved.

OpenAI said this was just the first phase of evaluation work, with reports to come and further "embedded engagements" to come in the longer term.

"We’ll continue working with multiple independent third parties to help build a healthy assessment ecosystem with diverse expertise and approaches," the post added. "Some investigations need sustained, on-site work with sensitive systems; others primarily evaluate deployed or soon to be deployed models or review public documentation."

FOLLOW US ON SOCIAL MEDIA

Follow ITPro on Google News and add us as a preferred source to keep tabs on all our latest news, analysis, views, and reviews.

You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.

Freelance journalist Nicole Kobie first started writing for ITPro in 2007, with bylines in New Scientist, Wired, PC Pro and many more.

Nicole the author of a book about the history of technology, The Long History of the Future.