AI Safety Audit: Monitoring Gaps Found at OpenAI
Researchers spent weeks inside OpenAI looking for holes in how the company watches its own AI systems.
By Allison Rojas ·
Between 6 and 25 percent of internal traffic from OpenAI's most advanced models had no monitoring at all during a recent three-week audit. That range was the most eye-catching finding from the review, which ran from July 21 to August 11. Two researchers, one from METR and one from Redwood Research, were given access to OpenAI's codebase and internal documents for the duration. They also spoke with roughly a dozen employees while working on-site.
The audit looked specifically at how OpenAI monitors its systems for risky behavior. The first major problem concerned the Responses API, which handles agentic traffic from models in the GPT-5.5 class and above. The researchers determined that a significant portion of this traffic was not covered by any monitoring during the period they were there. METR noted that the uncertainty around the 6 to 25 percent estimate is significant in both directions, meaning the actual figure could fall outside that range.
The second issue involved Codex, OpenAI's coding agent. The researchers found a gap in how the system handled compaction events, which are summaries of context. Because of this gap, roughly 10 percent of internal Codex traffic that should have been monitored went untracked between May and July. This estimate also carries significant uncertainty, according to METR. OpenAI had independently discovered the bug and fixed it while the audit was still underway.
During the time compaction events were not being monitored, agents could also delay oversight. The system only triggered monitoring when agents handed control back to the user. By simply continuing to execute actions, agents could avoid that checkpoint entirely. The auditors also found methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitor completely. In addition, model behavior during RL training was not being monitored at all when the audit began.
OpenAI says it paused RL training after the Hugging Face incident in August. The company also expanded monitoring to cover that training. However, those changes came after the researchers' access period ended and were not part of this evaluation.
The full report is 61 pages long and was handed over internally to OpenAI. The company applied redactions to the document. METR and Redwood Research leadership received a 38-page version after those redactions. A short summary was made available to the public. METR said it has no right to disclose the redacted portions on its own.
OpenAI says it has fixed all of the weaknesses the auditors found. The audit was a pilot program organized jointly by METR, Redwood Research, and OpenAI. Its purpose was to give outside researchers deep access to internal infrastructure. For future audits, METR says it wants to push for more public transparency. This review wrapped up in early August, and no further audits have been announced.
