‘A company should be able to use a model without giving up the knowledge that makes it unique’: Microsoft CEO Satya Nadella says enterprises shouldn’t be sharing so much data with AI providers

Companies using AI should be wary about losing control of their data – and the risk of AI developers making use of that knowledge.

That’s according to Microsoft CEO Satya Nadella, who in a recent blog post warned many AI users are "paying twice" thanks to what he calls the "reverse information paradox”.

This is a process through which AI providers, particularly frontier labs, get access to proprietary data about their own customers. Nadella said the trend represents a huge risk for enterprises, and called for protections akin to patents.

The critique is intriguing as Microsoft itself is an AI company, offering plenty of AI-driven products to customers and cramming AI features into its software. Beyond that, Microsoft was an early funder of OpenAI, though that relationship has weakened of late.

Nadella pointed to economist Kenneth Arrow's information paradox, in which the seller of data or knowledge risks giving it away in order to prove it's worth selling. He noted that AI has created the reverse problem, in which the buyer gives away knowledge in order to use the product purchased.

"You essentially pay for intelligence twice, once with money, and again with something even more valuable: the proprietary knowledge you must reveal to make that intelligence useful," he noted. "The better you want the model to perform, the more of that knowledge you have to feed it!"

"Over time, the information asymmetry becomes increasingly skewed. The seller learns more and more about you as you use what you purchased, while you learn very little about what the seller is learning in return."

The ‘AI exhaust’

This concern isn't just about uploading corporate data to systems, or accidentally including proprietary data in a prompt, though those are both concerns. Instead, Nadella said every interaction with AI inherently includes information that could be useful.

"Models learn from 'exhaust,' the prompts people write, the tools agents use, and especially the corrections people make when the model is wrong," he wrote.

"Every correction is distilled into institutional know-how. It’s the kind of knowledge a competitor could never buy, and the kind that leaks almost imperceptibly: trace by trace, correction by correction, eval by eval."

Ironically, Nadella said it was necessary for the "great innovation" of allowing model developers to use public data – some of them face lawsuits for hoovering up copyrighted data to train their systems.

When it comes to business data, providers should share anything learned with customers. Very few actually do this, however.

"If learning flows in only one direction, economic value converges toward the owners of the learning infrastructure rather than the creators of the knowledge itself," he said. "Therefore, it’s imperative that we distribute the learning infrastructure to every firm so that they can control their own learning loop."

Nadella's tips on how to protect your company from AI

Nadella said the existing setup requires a protection system on par with patents, but also advised companies to set a "trust boundary" by considering a few key aspects – some of which may be easier by simply switching to open source AI.

He suggested companies should control their own "evals", or evaluations of the AI system, because that reveals what looks "good" to the organization, and keep ownership of decisions, feedbacks, and other outputs for their own use.

Beyond that, companies should build capability with AI via their own learning environments where models can learn using real workflows without exposing corporate data.

Improving choice can be achieved by decoupling the orchestration layer from any single model, enabling the ability to switch to another model if one is withdrawn. This can also help organizations cut costs by choosing the right model based on their individual needs.

Lastly, he said to "compound" those four ideas to "create your own continuous learning loop".

"In other words, a company should be able to use a model without giving up the knowledge that makes it unique," he added.

FOLLOW US ON SOCIAL MEDIA

Follow ITPro on Google News and add us as a preferred source to keep tabs on all our latest news, analysis, views, and reviews.

You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.