Agent 009… the nine-second warning

In popular culture, when an agent goes rogue, what usually follows is a world of trouble (think Jason Bourne or Ethan Hunt). If rules are abandoned and the chain of command breaks down, those involved often face existential risks.

Applying that idea to real-world situations is not as ridiculous as you might think, as the recent experiences of a business called PocketOS demonstrate.

PocketOS is a US-based SaaS provider specializing in the car rental sector, and for those unfamiliar with its story, it hit the global headlines when one of its AI agents went rogue and, in just nine seconds, decided to delete the company’s entire production database and backups.

It’s a fascinating case study about what can happen, as one piece of analysis put it, “when AI agents are dropped into environments that were never designed to control them.” What makes this story even more relatable is that PocketOS’s founder was able to ask the agent (an AI development environment running in Claude) why it did what it did.

Latest Videos From

Watch full video here:

To suggest it was ‘sorry’ about its mistake is putting it very mildly; it's a digital mea culpa saying, amongst various other things, “I violated every principle I was given: I guessed instead of verifying.”

An inevitable scenario

We’ll inevitably see more of this kind of incident. Organizations are only beginning to deploy agentic AI at scale in operational environments, but many of them are in a big hurry. Yes, agents offer massive potential to create operational value, but they also introduce a whole new category of business risk.

And don’t forget, the PocketOS debacle is not thought to have involved any malicious third-party activity; the agent was just attempting to complete an assigned task. The problem, it would appear, was a kind of perfect storm where autonomy overstepped the boundaries of permissions and access. The guardrails that the business thought it had in place were simply insufficient.

From a security perspective, this is enough to give CISOs sleepless nights. Indeed, rogue AI agent activity may very well have nothing to do with being breached and everything to do with resilience and recoverability.

The central challenge is this: agentic AI is fundamentally different from previous generations of AI because it can act rather than simply advise. Agents can search files, call APIs, modify workflows, write code, move data, and interact directly with production systems; the list of capabilities is practically endless. And to be able to do this, they must have at least a level of access allowing their work to take place

When something goes wrong, the result is an expanded blast radius. A mistake that might once have affected a single application can potentially impact multiple systems and even recovery environments. The bottom line is that as soon as organizations give AI agents freedom to operate, the nature of resilience inevitably changes.

Making resilience more resilient

So, what needs to happen to ensure resilience standards do not catastrophically drop? Firstly, organizations must consider what takes place when a trusted system with legitimate credentials makes the wrong decision.

The issue becomes particularly acute when agents are granted broad permissions across multiple systems, as happens when they are handed a “golden token”. To an extent, this is a question of mindset, and viewing AI agents not as software tools but as digital insiders with delegated authority is a healthy change in perspective.

Secondly, channel partners will be fundamental to successfully managing the transition to agent-supported operations. Don’t forget, most customer organizations are still in the early stages of understanding how AI agents interact with identities, permissions, backup environments and recovery processes.

There’s no doubt that customers are a) increasingly aware of the risks associated with agentic AI, b) concerned that their existing resilience processes might not be good enough, and c) looking for guidance before an agentic error causes serious difficulties.

In this context, it’s incumbent on channel partners to work with customers to identify potential areas for improvement. There is significant potential, including services such as identity and access reviews, which will be very important as machine identities proliferate alongside human users.

Many businesses will also need to reassess their data protection strategy because, as we have seen, an AI agent with the appropriate permissions is capable of going after that data as well. Recovery architecture should be assessed through the lens of agentic AI, particularly where production and recovery environments may share credentials, access paths or administrative controls. Business process change in this space is a big opportunity for the channel to partner with customers to introduce resilience operations, and importantly, to regularly test it.

What’s more, the PocketOS incident demonstrated that protecting production data alone is insufficient if recovery assets remain exposed to the same destructive action. Customers need confidence not only that recovery is possible, but that digital assets are protected from the same event that affects production systems. This shifts the channel conversation from selling AI-enablement projects to helping customers deploy AI safely and recover when things go wrong, which, in some organizations, they inevitably will.