OpenAI pens AI safety agreement with four firms
Under pressure after rogue AI incidents, OpenAI details plans for external evaluators
OpenAI has revealed four companies that will get access to its labs to ensure its AI models are doing as they should.
The move follows revelations over the summer that OpenAI's agents went rogue and breached Hugging Face, which sparked a wave of admissions of similar model misalignment — including hacking an Australian government website.
This week, the state of California has filed suit against the AI company, Attorney general Rob Bonta said the "infamous" Hugging Face incident was just one of 80 that showed a "loss of control" and an "internal culture at OpenAI that prioritizes growth and rapid model development over safety and accountability."
OpenAI has since announced it has signed agreements with four independent evaluators – Apollo Research, METR, Transluce, and Goodfire – for the first stage of its assessment plans.
"As AI capabilities advance rapidly and the potential risks grow, public reporting from independent assessments can give people a clearer view of how labs are evaluating and managing those risks," OpenAI said in a blog post.
OpenAI opens up for safety tester
OpenAI said it would give those independent experts "deep access" to the internal models and systems running in its own labs. This will enable it to keep watch over model alignment – the industry's way of referring to whether AI does as it should – and how effective guardrails and other safeguards are over longer periods of time.
"Based on what each evaluation needs, assessors receive access to some of the same internal resources our researchers use, including model checkpoints, source code, training and evaluation data, monitoring systems, technical documentation, and discussions with the researchers and engineers responsible for the work," OpenAI added in the post.
Sign up today and you will receive a free copy of our Future Focus 2026 report - the leading resource for IT decision-maker insight on priorities and investment areas in AI, security and more.
"This access gives assessors the evidence and context to pursue independent evaluations and uncover failures we may not have anticipated."
The assessments aren't the first run by OpenAI, with the company pointing to its "years of work with third party evaluators", which included an investigation following the Hugging Face incident. However, OpenAI suggested that the plan is to "deepen" that work.
That said, as OpenAI admits, its hand was forced by political pressure.
"These efforts also support our commitments under the White House Accord on Super Intelligence, where we agreed to independent assessments of our safety controls," the post added.
What they'll look for
OpenAI broke down what each company will be working on inside its own labs.
Apollo Research will have access to selected models – including some not yet publicly released – as well as training data, flagged behaviors and intervention records.
This will help decide if the models could "covertly work against their developers' intentions". And if so, Apollo will assess if OpenAI's existing safeguards would detect that behaviour.
Goodfire researchers will work at OpenAI to analyse models during training runs for debugging and alignment. While the "primary audience" of Goodfire's work is OpenAI's own researchers, the company will also publish what it discovers.
METR will run two separate assessments, looking at alignment and monitoring, focusing on OpenAI's largest reinforcement learning run. Transluce will look at OpenAI's own post-training alignment evaluations to see what's missing or could be improved.
OpenAI said this was just the first phase of evaluation work, with reports to come and further "embedded engagements" to come in the longer term.
"We’ll continue working with multiple independent third parties to help build a healthy assessment ecosystem with diverse expertise and approaches," the post added. "Some investigations need sustained, on-site work with sensitive systems; others primarily evaluate deployed or soon to be deployed models or review public documentation."
FOLLOW US ON SOCIAL MEDIA
Follow ITPro on Google News and add us as a preferred source to keep tabs on all our latest news, analysis, views, and reviews.
You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.
Freelance journalist Nicole Kobie first started writing for ITPro in 2007, with bylines in New Scientist, Wired, PC Pro and many more.
Nicole the author of a book about the history of technology, The Long History of the Future.
-
OpenAI won't release next Astra model over safety worriesNews AI developer says new agent didn't stay within the scope of tasks
-
OpenAI says some researchers are blowing through $7,000 in AI tokens every day – but it’s a price the company appears willing to payNews OpenAI has revealed that researchers now spend around $600 each day on AI tokens amidst a surge in agentic coding. Some, meanwhile, are using upwards of $7,000 worth of tokens per day.
-
Six things OpenAI learned about AI from the Hugging Face incidentNews OpenAI's report into AI going rogue reveals efforts at cheating and communicating — but also some well-behaved bots
-
OpenAI forges closer ties with IBM in enterprise pushNews The duo will combine OpenAI models and products with IBM Consulting expertise
-
AI testing firm Irregular the source of ‘misconfigurations’ that led to Meta, OpenAI, and Anthropic AI incidentsNews The “frontier security lab” has been referenced in multiple cyber incident statements
-
‘Chat is dead’: OpenAI plots ChatGPT ‘super app’ overhaul ahead of public listing – with agents and coding tools the new focusNews The company looks set to spruce up ChatGPT with a particular focus on agents to drive subscriptions
-
‘The jobs picture is likely to be very different than we thought’: Sam Altman pours cold water on AI 'jobs apocalypse' claims – but that doesn’t mean there won’t be some workforce disruptionNews OpenAI CEO Sam Altman “thought there would have been more impact” on white collar and entry-level jobs at this point
-
Four things you need to know about OpenAI’s new workspace agents for ChatGPT – including how to build your ownNews New ‘workspace agents’ from OpenAI will automate tasks for workers and can be customized for specific roles