Why AI pilots fail when the technology works

Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027. MIT’s findings are blunter still, with roughly 95% of generative AI pilots delivering no measurable impact on the P&L. It would be easy to treat those numbers as proof that AI has been oversold, but I think they point to a much older problem.

Businesses are good at proving that technology works in a demo. They are much less good at making it work on the kind of ordinary Tuesday when the data is messy, the hand-offs are awkward, and nobody is completely sure who owns the outcome.

We see versions of this all the time. A team vibe-codes a convincing pilot in a fortnight, gets a good reaction in the demo, and then spends the next six months working through permissions, security, edge cases, integrations, testing, and the less glamorous question of who has to support it when it breaks. After a few months, it becomes pretty clear that the model was never really the problem.

A quick clarification before going further. When I talk about AI here, I mean AI agents, systems that use a large language model at their core but can also act on their own, calling tools, querying knowledge bases, and searching the web as a minimum, to complete a task.

Too much of the AI conversation has been stuck on output, from tasks automated and hours saved to code generated. Those measures are useful up to a point, but they do not prove that anything has actually improved.

Faros AI found that in organizations using AI heavily, more code was being thrown away or rewritten soon after it was produced, which is uncomfortable if the business case assumes faster output means faster progress. Software still has to be reviewed, tested, integrated, secured and maintained. Generating code quickly is useful, but the value only really shows up when people can still understand it, trust it and keep it running months later.

The same problem shows up in the metrics businesses report. A dashboard can show pilots launched, users onboarded, tokens consumed and outputs produced, while still avoiding the question that matters. Has the business actually improved? Are processes faster, is risk lower, are customers waiting less, or has the organization just created more activity to measure?

The drop-off between proof of concept and production is usually less dramatic than people expect. The pilot worked because the data was limited, the integrations were controlled, and the awkward exceptions could be pushed aside.

Production brings all of that back. The data is inconsistent, the systems do not quite join up, and ownership becomes harder to pin down once the tool touches a live process. Technology programs have been running into these problems for years, but AI tends to find them faster. Choosing the model is often the easier discussion. Keeping it useful after the demo is the harder work.

Where the channel can actually add value

The channel has a useful role to play here, but only if the conversation moves beyond platform and model selection. Customers often start there because those decisions feel tangible, but they rarely decide whether an AI project survives production.

The better work is less exciting and more revealing. It means understanding if the customer is ready to run the system properly, how much trust there is in the data, who owns the outcome when something goes wrong, how quality will be measured, and how a person can challenge a decision the system has made.

Two practical things are worth pressing on early. The first is buy versus build, where the wrong move is often building something that should have been bought. Almost nobody should be training a model from scratch, and many organisations should be wary of building custom applications around an LLM when the same need can be met inside a platform they already pay for. If the need is common, buy it. Building starts to earn its keep when the workflow is genuinely differentiating, sensitive or tightly integrated.

The second is cost, because AI economics behave nothing like software. There is no fixed licence fee you can model once and forget about. Token pricing, usage and model choice move the bill around constantly, and successful adoption can make the cost rise. So instrument it from day one and measure cost per outcome, such as each invoice processed or proposal raised, rather than cost per token. It should be reported clearly as its own operating cost, not hidden inside a wider technology budget.

Buying or building also brings the same need to test and monitor the system while it runs. Conventional software is expected to give the same answer to the same input, while AI outputs vary, models drift, and a vendor can quietly change how a model behaves while the badge stays the same. Observability and guardrails need to be built in from the start, so there is a clear view of whether the decisions being made on your behalf are still good enough, and where the limits sit for actions the system should never take.

The deeper point, especially for anyone reselling someone else’s platform, is that buying may shift responsibility, but it does not remove accountability. Your SaaS vendor may deliver the feature, but you remain accountable for the outcome, on your customer’s data, under your name. The agent inside Salesforce or ServiceNow may be calling a model from Anthropic, OpenAI, or Google behind the scenes, which means many of the same questions still apply.

The durable advantage is not the agents, but the people who can build and govern them. We run our delivery teams with a senior engineer alongside juniors from our academy who, 12 weeks earlier, were working in places like Amazon warehouses and Starbucks. On one central government programme, that approach has put more than forty agents into production, with five to seven million pounds of projected first-year return. The work called for engineering instinct, appetite and enough room for people to learn by building, rather than a crack team of PhDs.

This is why stronger progress tends to come when AI is treated as an operational capability, rather than a series of experiments. Customers need a way to move from one working use case to a measurable outcome, supported by the governance, testing, observability, and ownership that keep AI useful long after the first demo has done its job.