The role of data engineers is changing, according to new research, and they’re becoming increasingly crucial to businesses thanks to the rise of AI.
Three-quarters of business leaders increasingly believe data engineers – those who manage databases and extract information for actionable insights – are now integral to the success of their company, according to a survey from MIT Technology Review and Snowflake.
That rises to 86% for the largest companies with revenue over $10bn, the survey found.
"Senior leadership is quickly realizing that if they’re not using data in their decision-making processes, they risk lagging behind their competitors," said Chris Child, VP of product data engineering at Snowflake.
In a post detailing the survey results, Child noted he saw two trends contributing to the increasingly prominent profile of data engineers.
"The first is the sheer volume of information being generated and made available today – a function of data being easier to collect and cheaper to store than ever before," Child wrote.
"The second is a growing realization among top executives that without using data to inform decisions, their organizations are essentially flying blind. This is driving demand for AI to enable the use of data in more and more complex decisions than ever before."
No wonder then that the survey found three-quarters of companies expect their data engineers to face ever increasing workloads.
Data engineers are swamped at work
Indeed, the survey found that the amount of time spent by data engineers on AI projects has doubled from 19% of their day in 2023 to 37% this year – and those polled expect that to hit 61% of their work day within two years.
But they're not just putting work into AI, however, and data engineers are also getting a benefit. Three-quarters of respondents said AI has boosted the output of data engineering teams over the last two years, with a similar figure reporting an improvement in quality of work, too.
While half of those surveyed plan to begin deploying agentic AI in their workflows in the next year, with 20% already there, respondents reported expectations of continued productivity boosts.
But that optimism is tempered by concerns about complexity and exacerbating tech stack fragmentation, Snowflake noted. Because of that, the role of data engineers may become more about management.
"Over time, I think the role of the data engineer will be less about writing code for every single pipeline and more about managing the infrastructure that AI agents are operating in,” Child noted.
Shifting roles for data engineers
The importance of data engineers in AI projects means the C-suite has finally come to realize and acknowledge the importance of the field.
"This has really brought data engineers to the attention of senior business leaders, with data engineers increasingly serving as essential partners for shaping strategy and driving business outcomes," said Child.
Indeed, data engineers jobs are evolving – and the role's influence is expanding.
Two-thirds (66%) of respondents said data engineers are influencing decisions about what tools to invest in and from which vendor. Meanwhile, half said engineers now have an impact on overall data strategy and how AI is used in the business.
"AI is pushing leaders to reconsider data processes as a result of a digital transformation," said Child.
"As AI agents take on more work, data engineers will spend less time on extracting, transforming and loading work and more time on the data and AI strategizing that will help take the business forward."
Child advised data engineers to take advantage of the opportunities offered by the rise in AI by upskilling in the subject, ensuring they have a strong understanding of key business objectives, and building strong communication skills to be able to expand their influence further.
Make sure to follow ITPro on Google News to keep tabs on all our latest news, analysis, and reviews.
MORE FROM ITPRO
