Fears over “AI model collapse” are fueling a shift to zero trust data governance strategies

Fears over unverified AI-generated data are pushing organizations to implement a zero trust posture for stronger data governance.

Large language models (LLMs) are typically trained on web-scraped data, and as the use of generative AI grows these sources increasingly contain AI-generated content.

This is set to accelerate in the coming years, according to Gartner. Analysis from the consultancy found 84% of respondents expect their organization to increase funding for generative AI in 2026 – meaning that the volume of AI-generated data will continue to rise.

As a result, future generations of LLMs will increasingly be trained on outputs from previous models – outputs that may be inaccurate or biased. This means an ever-heightening risk of “model collapse,” where AI tools’ responses may no longer accurately reflect reality.

"Organizations can no longer implicitly trust data or assume it was human-generated,” said Wan Fui Chan, managing VP at Gartner.

“As AI-generated data becomes pervasive and indistinguishable from human-created data, a zero trust posture establishing authentication and verification measures is essential to safeguard business and financial outcomes.”

By 2028, Gartner found the proliferation of unverified AI-generated data is set to have prompted 50% of organizations to have implemented a zero trust posture for data governance.

Active metadata management practices will become a key differentiator for firms, the consultancy said, and organizations should consider taking several strategic actions.

How to approach a zero trust data governance strategy

According to Gartner, IT leaders should appoint a dedicated AI governance leader, responsible for AI governance, including zero trust policies, AI risk management, and compliance operations.

This person should work closely with data and analytics (D&A) teams to make sure that both AI-ready data and systems are capable of handling AI-generated content.

Cross-functional teams should also be established, including cybersecurity, D&A, and other relevant stakeholders.

These teams will carry out comprehensive data risk assessments to identify business risks related to AI-generated data - and decide which activities need new strategies.

Elsewhere, building on current D&A governance frameworks will be crucial to tackling the issue, according to Gartner. The creation of new or updated frameworks will focus primarily on updating security, metadata management, and ethics-related policies to address new risks arising from AI-generated data.

Regulatory compliance challenges are afoot

As AI-generated content becomes more prevalent, Chan warned that regulatory requirements for verifying ‘AI-free’ data are likely to intensify.

Crucially, these could differ significantly across geographies, posing serious compliance challenges.

“In this evolving regulatory environment, all organizations will need the ability to identify and tag AI-generated data," he warned.

"Success will depend on having the right tools and a workforce skilled in information and knowledge management, as well as metadata management solutions that are essential for data cataloguing.”

FOLLOW US ON SOCIAL MEDIA

Make sure to follow ITPro on Google News to keep tabs on all our latest news, analysis, and reviews.

You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.