‘Real-time data remains the voice that every leader must hear’: This one engineering role could be the key to building a truly data-driven enterprise

Data streaming engineers are in hot demand, according to new research, with 85% of organizations planning to expand data streaming engineering teams in the year ahead.

A survey of 200 UK CEOs, MDs, and C-level executives by Confluent found that data streaming engineers are now key to turning raw information into tangible insights that power AI.

Data streaming engineers play a vital role in managing the continuous movement and processing of data, the study noted, and are increasingly helping shape AI strategies.

“While traditional data engineers focus on storing, cleaning and preparing historical data, data streaming engineers operate in the present tense, ensuring data is continuously available, trusted and actionable the moment it’s created,” said Richard Jones, VP Northern Europe at Confluent.

More than half (56%) the firms strongly agreed that all data-driven organizations should have at least one data streaming engineer. Meanwhile, 83% said they could make more informed decisions faster with dedicated data engineers

Elsewhere, 88% believe that organizations that hire them will be better placed to adopt, use, and manage AI moving forward.

Demand grows for data streaming engineers

Confluent’s study found that demand for data streaming engineers is expected to grow in the coming years, with 94% of business leaders agreeing that every data-driven organization should employ them.

85% of respondents said they plan to accelerate hiring on this front moving forward.

“AI can only work with the data it’s given, and when that data is incomplete or out of date, the consequences can be serious," said Jones.

"That’s why it’s so encouraging to see businesses investing heavily in data. If leaders want AI to make informed decisions, it needs an accurate, real-time view of what’s really happening across the business. Without that, AI can sound knowledgeable, but it won’t be truly intelligent.”

Data-driven decision making

Demand on this front comes amidst heightened enterprise AI adoption, according to Confluent. Yet many organizations still face issues with real-time decision making when using the technology.

Most leaders (59%) believe that their colleagues’ use of AI has increased the expectation for fast or even instant decisions. Nearly two-thirds (61%) admitted they struggle to capitalize on fast-moving insights or trends, for example.

More than eight-in-ten often have to choose between making a quick decision and an informed one, and 59% of leaders frequently rely on gut feelings.

Meanwhile, six-in-ten leaders admit that data is too difficult to access at their level, with 71% revealing that it’s already out of date by the time it reaches them.

The result here is that enterprise IT leaders are making poorly informed decisions, which has a negative impact on overall productivity and profitability.

Three-quarters admit they have regretted decisions made too quickly, while 71% regret waiting too long and missing out on opportunities.

Notably, nine-in-ten respondents said they want to make more data-driven decisions at work, and the same number said they’d feel more confident if they had access to real-time data insights.

"The real foundation of confident decision-making isn’t instinct or AI alone, it’s real-time data. Accurate, trustworthy, context-rich insights that can flow across an organization in the moment leaders need them most," said Jones.

"Real-time data remains the voice that every leader must hear."

FOLLOW US ON SOCIAL MEDIA

Follow ITPro on Google News and add us as a preferred source to keep tabs on all our latest news, analysis, views, and reviews.

You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.