How the rise of the AI ‘agent boss’ is reshaping accountability in IT

The use of AI tools and agents is concentrating accountability into the hands of fewer employees. Paired with unreliable outputs from hallucinating AI tools, the question of who is responsible for AI outputs – and how much oversight agents need – is front of mind for IT leaders.

In a 2025 blog post, Microsoft’s CMO of AI at Work, Jared Spataro, argued that IT managers will soon be expected to “think like the CEO of an agent-powered startup.”

“As agents increasingly join the workforce, we’ll see the rise of the agent boss: someone who builds, delegates to and manages agents to amplify their impact and take control of their career in the age of AI,” Spataro wrote. “From the boardroom to the frontline, every worker will need to think like the CEO of an agent-powered startup. In fact, leaders expect their teams will be training (41%) and managing (36%) agents within five years.”

However, as responsibility for managing AI moves closer to employees, research from Asana has found that six in 10 ‘knowledge workers’ who manage AI agents said their jobs were being made more difficult by the generation of "confidently wrong outputs”. This figure reveals a critical trust barrier, which has also heightened concerns surrounding accountability.

Of the 2,025 knowledge workers that Asana surveyed across the US and the UK in 2025, a third said they had no idea who to contact when an AI-related issue arose; a figure supported by the Wharton 2025 AI Adoption Report, which found that a third of enterprises have no individual role that’s accountable for AI (such as a chief AI officer).

“If managers are now the “CEO” of an agent-powered startup, their core skill must be how to achieve delegation with verification,” explains Alexander Feick, vice president at eSentire Labs. “You’re not learning to trust the agent; you’re learning efficient ways to do the work and to prove its outputs were good and well aligned.”

Managing AI agents in the real world

During 2025 Feick and the team at eSentire worked with CSIRO, Australia’s national science agency, to analyse data in a ten-month trial exploring how large language models (LLMs) such as GPT-4 could support cybersecurity analysts to detect and stop threats, while reducing fatigue.

“In our environment, the new skills we needed to develop were mostly practical and operational as opposed to mystical prompt tricks,” says Feick. “Breaking work into verifiable microtasks (interpret this command, summarize these logs, draft this customer note), asking for evidence and alternatives instead of conclusions, spotting when a model is confident but wrong, and treating model outputs as rough drafts that must be back-stopped with telemetry and procedure before they are trusted.”

The anonymized data was collected at eSentire’s Security Operations Centres (SOCs), situated in Ireland and Canada. During the trial, 45 cybersecurity analysts asked GPT-4 more than 3,000 questions, mostly for routine, low-risk tasks such as interpreting technical data, editing text and analysing malware code.

Analysts overwhelmingly used the system as an “evidence interpreter”, not a decision engine. And according to the research only 4% of queries asked for explicit recommendations, signalling a preference to keep judgment and escalation decisions with humans. However, the trial still raised issues around trust and accountability.

“Usage was concentrated among a subset of analysts, and the study notes cases where low engagement followed isolated errors or refusals,” Feick tells ITPro. “From an accountability standpoint, that tells you two things: one, the tool needs to be reliable for the tasks people actually use it for, and two you need clear expectations that ‘AI is a draft,’ plus fast verification loops so analysts don’t either over-trust it or abandon it.”

If an AI agent is going to produce work, Feick believes that companies need to embed traceability and governance by default, answering questions such as: who/what initiated it, what evidence it relied on, what tools it touched, what it changed, and what checks it passed.

“If you can’t reconstruct the chain, you can’t defend the workflow, and you can’t improve it,” Feick says. “Accountability stays with the organisation and the human decision-makers, not the model. That’s why we design around traceability and gates: you want the system to make it easy to reconstruct what evidence was used, what changed, who approved it, and what controls were applied—so a miss becomes diagnosable, not mysterious.”

Start building trust into AI systems

As organizations develop and refine verification workloads, they must do more to upskill analysts.

“That’s the mindset shift,” Feick says. “We should judge systems on measured performance in the real workflow (including adversarial and edge cases), monitored over time, with clear criteria for when we slow down, route to humans, or roll back,” explains Feick.

He adds that companies managing AI tools and agents need to stop asking if models are good enough to trust and instead focus on ensuring outcomes are agreeable even in the event of AI errors.

“This trial forced me to stop treating trust as something you feel about a model, and start treating it as something you build into a system,” says Feick. “Agents are where the trust conversation gets real, because the failure mode isn’t ‘the model said something weird’. The failure mode is: a system took an action, moved state, changed a ticket, escalated (or didn’t), contacted a customer, triggered automation – whatever it is – and now you’re accountable for that outcome.”

As enterprises deploy more autonomous AI tools and agents, accountability is becoming more concentrated, more operational, and harder to avoid. The idea of an ‘agent boss’ is an empowering one, but it requires the right systems to be put in place for verification, traceability, and rollback.

The lesson from eSentire’s work with CSIRO is not that AI agents should be trusted less, but that trust itself must be redesigned. In practice, that means treating AI outputs as drafts, building fast verification loops, and ensuring every automated action can be reconstructed and defended after the fact.