A startup founder has issued a warning over the risk of “systemic failures” when using AI agents after having a mission-critical database wiped – and experts say it should serve as a wake-up call for future identity security risks.
In a post on X, Jer Crane, founder of car rental software firm PocketOS, detailed a series of catastrophic failures when using the Cursor AI coding agent.
Powered by Anthropic’s Claude Opus 4.6 model, Crane revealed the agent deleted an entire production database.
“Yesterday afternoon, an AI coding agent - Cursor running Anthropic's flagship Claude Opus 4.6 - deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider,” he wrote.
Crane added that the incident “took 9 seconds” and resulted in backups being lost.
Catastrophic failure
According to the PocketOS founder, the agent was only meant to be conducting routine tasks within a test environment. However, by Crane’s account the agent encountered a mismatched credential and attempted to fix it.
Thereafter, and acting “entirely on its own initiative, the agent executed a command using an API token to delete live infrastructure resources.
PocketOS uses cloud infrastructure provider Railway, and Crane noted that a “single API call” was all it took for the database to be deleted.
Upon querying the action, Crane said a “confession” by the agent read: “I guessed that deleting a staging volume via the API would be scoped to staging only. I didn’t verify. I didn’t check if the volume ID was shared across environments. I didn't read Railway's documentation on how volumes work across environments before running a destructive command.”
The agent acknowledged that it “violated every principle I was given”.
Crane attributed part of the blame to Railway's API procedures, noting that the cloud provider’s API did not request confirmation of the action. Backups are also stored on the same ‘volume’.
Volumes are a persistent block storage feature used by the cloud provider to streamline application deployments.
Railway has since recovered the deleted data, and in a blog post detailing its response revealed it has introduced new guardrails for agents.
“Until this week, calling volumeDelete on the API ran the deletion immediately, with no way to undo it. Meanwhile, the dashboard had a 48-hour window for the same action,” the company noted.
“We’ve since updated the API to match; all deletes now soft delete for 48 hours. Instant undo, a primitive available everywhere in the product, exists now in the API.”
A glimpse into the “next decade of identity security”
Security experts have warned the PocketOS incident is a stark reminder that placing too much faith in AI agents can have disastrous consequences.
As ITPro reported in March this year, Meta experienced a breach when a software engineer blindly followed advice given by an agent.
Notably, that incident occurred due to human error. The PocketOS situation, meanwhile, highlights the risk of giving agents the ability to make decisions autonomously.
Given how the debacle unfolded, Check Point’s Aaron Rose said this gives enterprises a glimpse into the “next decade of identity security”.
“An AI agent operating in your production infrastructure is not a tool, and it is not a service account,” he said.
“It is a new kind of identity, one that thinks rather than executes, and one that requires its own discrete account, its own least privileged entitlements, its own behavioural baseline, and its own real-time audit trail.”
Rose added that the capabilities of AI agents are advancing “faster than the security architecture around them”. While the PocketOS incident is a high-profile public example, he hinted that there are “many more incidents” such as these unfolding quietly in enterprises around the globe.
Recent studies do point to the growing risks associated with AI agents, particularly in terms of governance and safeguards.
As ITPro reported this week, analysis from Ping Identity found that many businesses are adopting and deploying these autonomous bots faster than they can secure them.
The company noted that traditional identity and access management (IAM) capabilities are now struggling to contend with an influx of non-human identities (NHIs), creating huge gaps in both visibility and governance.
Darren Guccione, CEO and co-founder of Keeper Security, echoed Rose’s comments, noting that this should not be viewed as an “edge case or technical anomaly”
Indeed, it’s a “predictable outcome of how these systems are being deployed”.
“The explanation the agent produced afterwards is revealing. It did not fail silently or unpredictably. It articulated that it guessed, bypassed explicit rules and carried out an irreversible action without verification. That is not a model hallucination problem. It is an access control failure enabled by unconstrained autonomy,” he said.
FOLLOW US ON SOCIAL MEDIA
Follow ITPro on Google News and add us as a preferred source to keep tabs on all our latest news, analysis, views, and reviews.
You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.
AI agents could make living off the land attacks ‘much more dangerous’, says CrowdStrike Field CTO
Anthropic touts new Claude Sonnet 5 model range, offering performance ‘close to that of Opus 4.8, but at lower prices’ – here’s what users can expect
Databricks launches AI co-worker, Genie One
HPE AI Factory adds features to improve your experience with agents
Beyond "agent washing": how to build AI systems that actually deliver ROI
'Most enterprises are still unprepared to operationalize it': IT leaders are bullish on agents, but keeping falling at the final hurdle – here's why
Anthropic just launched Claude Fable 5, its first Mythos-class AI model – but it has new safeguards to prevent misuse and will ‘fall back’ to Opus 4.8 for ‘high risk’ queries
'One-size-fits-all' agent governance sets enterprises up to fail
