Chief data officers (CDOs) might not be the most important executives in the C-suite right now, but they’ll rank among the most influential figures in no time.

That’s according to Deloitte's Chief Data Office (CDO) report, which found 54% of CDOs believe they have less influence on the business than their C-suite colleagues, but 44% believe that CDOs will become of equal influence by the turn of the decade.

"While over half of CDOs currently feel less influential than other C-suite leaders, there's a powerful shift on the horizon," said Andy Whitton, data, privacy and analytics partner at Deloitte.

The survey found that 57% of CDOs now report to a CIO or COO, up from 39% last year. Beyond the C-suite, nearly two-thirds of CDOs predict their teams will add employees over the next year, growing by an average 27%.

Budgets are also on the rise, with 43% saying they've seen a boost over the last year and 56% reporting an increase in data spend.

"Organizations are clearly doubling down on data, rapidly increasing their data maturity and literacy to gain crucial competitive advantage," said Whitton.

"This is positioning the CDO role as a pivotal and increasingly influential force across the C-suite, helping to drive sustained growth."

Chief data officers are riding the AI wave

The rise in influence for CDOs is perhaps no surprise given the sharp enterprise focus on AI and the key role they play in driving adoption of the technology.

Half of CDOs polled said they were accountable or responsible for AI or generative AI in their organizations, for example. Indeed, three-quarters of those surveyed revealed they have AI deployments or experiments operational right now.

Data governance and AI were by far the top priorities for CDOs, though their importance differed by sector. AI was deemed the top priority for CDOs in financial services, highlighted by two-thirds of respondents in that industry.

Meanwhile, 50% of corporate CDOs and 70% of public sector CDOs said data governance was their key focus.

Whitton said this highlights a “nuanced approach” taken by CDOs, with those operating across different sectors tailoring their priorities based on individual needs.

"For example, AI and GenAI are top of the agenda for 66% of financial services CDOs, whereas corporates are homing in on data products, and the public sector is rightly prioritizing data governance,” he said.

Those CDOs who named AI as their core priority also tended to predict their organization will realize benefits more quickly than those who aren't prioritizing the technology – though perhaps the eye on AI payoffs like productivity gains are why they're choosing to focus on the technology in the first place.

It’s not all plain sailing for CDOs, however. Long-running finance-related issues were identified as key barriers to progress.

Nearly half of the CDOs said budget and resource limitations were the biggest hurdle to AI adoption, for example, followed by skills gap and ethical and regulatory concerns.

A separate report from earlier this year found that two-thirds of CDOs believed data quality was the biggest challenge to AI adoption, hampering them from maximizing the use of the technology.

MORE FROM ITPRO

• Why diversity is key to a successful AI strategy
• Data governance in the spotlight as UK firms ramp up AI adoption
• A quarter of firms still don’t have a formal data strategy

Only a third of businesses are making meaningful progress in AI adoption, and chief data officers (CDOs) are pinning the blame on poor data quality.

More than two-thirds (68%) of CDOs said data quality was their top challenge, according to the Ataccama Data Trust Report 2025. Meanwhile, four-in-ten struggle to maintain consistent data quality, directly hindering AI outcomes.

The report noted that trust is a critical factor when leveraging data in daily operations. Without it, organizations face inefficiencies, poor decision-making, and the risk of compliance failures, ultimately limiting their ability to achieve business objectives.

However, the report found most organizations still struggle with siloed systems.

"Untrusted data erodes every decision it informs. Without real insights into data quality, businesses risk cascading failures, from unreliable AI outputs to stalled growth," said Krishna Cheriath, chief digital Officer at Thermo Fisher Scientific.

"Trust must permeate every layer — data, models, and decisions."

Bad data leads to bad insights, the report noted, affecting decision-making, slowing down operations, and wasting valuable resources. Similarly, it jeopardizes compliance, leaving organizations vulnerable to regulatory and financial risks, and diminishes ROI.

Knowledge gaps are hampering digital progress

Knowledge gaps around data trust and governance are slowing progress, the report found, with a lack of unified standards leading to inconsistency. Without guidelines for data formats, definition, and validation, CDOs find it hard to establish a centralized system of control.

A third of organizations experience processing delays because there are too many barriers that stand in the way of integration.

"Fragmented systems bleed efficiency and inflate costs," said Andrew Foster, chief data officer at M&T Bank.

The report also found legacy systems are still a major barrier to innovation, with CDOs finding that outdated systems are ill-equipped to handle increasing data volumes.

Many systems are designed to provide period data updates, for example, rather than continuous, real-time streams. It’s this area in particular that’s causing serious trouble for CDOs ramping up AI adoption, with only a third of enterprises reporting meaningful progress on this front.

The report called for new national standards for data quality in the UK, and suggested that the proposed National Data Library - a core goal within the UK’s AI Action Plan - could play a key role in bolstering national data governance benchmarks and best practices.

These standards would ensure clear compliance guidelines while supporting the UK’s pro-innovation regulatory goals.

"The report makes one thing clear: enterprise AI initiatives rely on a foundation of trusted data," said Jay Limburn, chief product officer at Ataccama.

"Without addressing systemic data quality challenges, organisations risk stalling progress. The UK’s approach to AI regulation shows how aligning data trust principles with national standards and infrastructure modernization can deliver tangible results."

Organisations make key business decisions on the basis of what they know, and what they know is derived from analysing their data. To enable the best decisions, data has to be accurate and reliable. But there are still organisations out there with data silos, which by their very nature can be anything but accurate and reliable. These silos often contain duplicate, out-of-date, or otherwise dangerous data.

Added to this, maintaining two or more data sets on the same topic uses time and technology resources an organisation could more fruitfully channel into other work. It’s time to purge the data silos and stop them from ever coming back. Doing so, however, involves hunting for silos across the breadth of an organisation and implementing measures that address the cultural deficiencies that let silos flourish in the first place, all of which is easier said than done.

What damage can data silos do?

There’s widespread agreement that data silos are a bad thing. “There can be no debate,” insists Dave Williams, consulting associate director for leading data consultancy Dufrain, Dave Williams. “Data silos are the scourge of the modern company.”

Chief among the problems they raise is the lack of centralised control, he tells IT Pro. Without strong governance to ensure the data an organisation holds is accurate and up to date, data silos are simply repositories that can lead to ill-informed decision-making.

“Silos can deeply impact the decision intelligence of any organisation,” agrees Janet Bastiman, chief data scientist at Napier and chair (data science and artificial intelligence (AI)) at the Royal Statistical Society. “Without a single holistic view, decisions may be taken without full information,” she elaborates. “More importantly, updates may not filter to all silos, meaning that duplicate data can be in different states.”

How to identify data silos

If you think your organisation has no data silos, are you absolutely certain? Have you checked everywhere? Well, you might be surprised, Williams suggests. “Data silos are typically created in business areas when a centralised data platform is either too well governed to accept disparate business data, or where an IT team is too disinterested to incorporate data together because the corporate value is considered low.”

There can be other factors at play too, such as business units thinking they have some kind of advantage if they maintain their own data. This can be born from a sort of psychology of competitiveness that has its roots well outside the technology, in the overall way business units view each other – as competitors rather than collaborators.

Whatever the reason for the existence of silos, finding them can be tricky. Just asking around isn’t always going to be enough. Some tech-based sleuthing might be able to identify large files such as spreadsheets, the owners of which can be quizzed. But if this approach is taken, Bastiman offers some sage advice. “The individual who does this must have clear authority from the executive team to have access to sensitive silos,” she says. This is especially important if there’s already a competitive, perhaps antagonistic, vibe between business units.

Indeed, former chief data officer (CDO) for Network Rail, as well as CEO and co-founder of Carruthers and Jackson, Caroline Carruthers, tells IT Pro technology isn’t the ideal tool for hunting down data silos. told IT Pro that technology isn’t the ideal tool for hunting down data silos. ”I’d argue hunting data silos doesn’t really need any sort of technological method,” she says. “Tech can do whatever you want it to if you put your mind to it, but data silos themselves are fundamentally people problems.”

Preventing data silos is better than ‘curing’ them

Once data silos have been identified, the next steps are to remove them and prevent them from appearing again. This too, for Carruthers, is a people thing, not a technology thing, and the responsibility often lies with the CDO. “A CDO’s role here is to understand why data silos exist in the first place, as they’ll have happened in the first place because some employees thought they were needed,” she explains. “Fundamentally, you can’t wipe out a silo without putting something better in its place, and you can’t put something better in its place without first understanding the need.”

Williams is also clear that workplace culture really matters, and has a key role to play in preventing data silos from reappearing once they’ve been eliminated. “Once all data silos are identified and broken down, companies must rethink their cultural and technological processes to protect themselves from data silos popping up in the future,” he tells IT Pro. He proposes the promotion of the development of “collegiate ways of working to ensure everyone is working from the same one true source of data”. Once such a regime is in place, he continues, “culture and technology must work together in lockstep to affect long-term change”.

Developing and maintaining a collegiate, collaborative approach to work, using cloud-based data systems, might take time. But there seems to be general agreement that data silos arise as a result of certain ways of working and are prevented by the adoption of others. In the end, then, data silos are a technological solution to what is very much a human situation. Change the situation – eliminate the data silos – and make better business decisions, too.

This article originally appeared in issue 34 of IT Pro 20/20, available here. To sign up to receive each new issue in your inbox, click here

Data now sits and the heart of enterprise strategy and senior stakeholders recognise its significance. The most forward-looking businesses are appointing a chief data officer (CDO) to ensure their organisations take advantage of the information they collect. The role of a CDO, however, has changed over time and will continue to evolve as data becomes more important to businesses in the coming months.

Such is pivotal the role of information in modern business that consultancy firm PwC reports that two-thirds of companies have seen a growth in references to data in their annual reports since 2017. Data frequency at these firms has increased by 78% in the last five years.

While most organisations know the power of data, not all executive teams support that appreciation with the appointment of a CDO. Recruiter Nash Squared reports that two-thirds of digital leaders see big data being as key to creating a competitive advantage during 2023, but only a fifth feel their organisations are effective at using data.

In fact, PwC found – despite growing recognition of the role played by data in an effective business strategy – less than a quarter of companies have a CDO in place at a senior executive level. Even when a data chief is in situ, these C-level appointments are concentrated in a few sectors and regions.

It’s hard to find a good CDO these days

Nash Squared CEO Bev White believes one explanation for this paucity of data leadership is that good CDOs are tough to find. “It's a talent-short market – and what we're seeing is a lot of people moving around roles because they're getting poached repeatedly,” she says.

White says the best data chiefs are likely to have a solid background in large-scale analytics but are also great communicators: “CDOs have got to be able to talk with people who are in decision-making roles and convince them that an investment is a good one because the business will then capture a slice of the market it can't currently reach.”

The requirement to demonstrate the value of emerging technologies – in areas such as analytics and machine learning – means data leaders must be proactive and persuasive. White says CDOs holding these capabilities will be very marketable through 2023 and onwards.

“They've got to be able to form a strategy and have a strategic mind,” she says. “And they need to be able to create a really strong data environment, where people understand the value of bringing data together into a place that they can use to make good, solid decisions and create new offers for their markets.”

One data chief who’s already demonstrating these strengths is David Charnley, head of transformation and strategy at NatWest Group. He says data leadership was historically focused on data governance and information management, with responsibilities that stretched from data quality and lineage to controlling metadata repositories.

While those responsibilities still hold true in some organisations, Charnley believes the CDO role is evolving – and the result is a broad spectrum of data chief roles. At NatWest, data leadership involves looking across analytics, digital decision-making processes and data science capabilities for customer-facing businesses, but also dealing with data management concerns and the technology that underpins the work that data teams do for the business.

“I think that combination of all of those disciplines together, unified with the technology, is really, really powerful,” says Charnley. “Because it then means the CDO is very much in control of the data strategy and its implementation for the bank.”

Data platforms are now seen as an investment

Research suggests the most pioneering businesses are now aware of the role a talented data chief can play. The boards and CEOs of these organisations are appointing senior executives who are accountable for data success. PwC says three out of five CDOs are now members of the C-Suite with oversight of enterprise-wide data.

That’s a trend that’s likely to continue, says Alex Read, senior manager of data platforms at EDF UK, who suggests next-generation data leaders will have to do more than simply extract value from the technologies they buy. He says successful data chiefs will focus on how their organisations can work effectively to create value.

“Historically, there was too much focus on the data platform – just enabling the business-as-usual elements, such as reporting at a point in time, and enabling regulatory requirements, which is all of the standard stuff you'd expect the business to need,” he says. “Now, data platforms are seen as an investment. And I think what you need to do as a data leader is to communicate to your key stakeholders what that investment means.”

At EDF UK, Read is creating an integrated data stack that runs on the Snowflake platform. By using tactical solutions – such as the Snowpark development framework, AWS SageMaker for machine learning, Collibra’s data catalogue, AirFlow for orchestration, and Matillion for data extraction and transformation – Read focuses on how this combination of tools can support innovative, information-rich services for EDF UK and its customers.

“I think that's a massive step up,” he says. “This work is about more than just platform consolidation and the reduction of legacy platform costs. It's about, ‘OK, great, once we've done that foundational work, what's the value-add in having this platform, such as data science and other downstream analytical processes?’ These are lenses that a data leader should be looking through as part of that journey.”

Transmitting strong data skills across the enterprise

Data chiefs can’t just focus on developing internal capabilities, though. As well as ensuring their team has the techniques to make the most of information, CDOs need to find ways to turn the exploitation of data into an enterprise-wide activity. For organisations to create useful services, it’s important that all kinds of professionals hold strong data skills.

That’s something that resonates with Marc Jennings, CIO of analytics and artificial intelligence (AI) at holiday firm TUI, who says machine learning and data science skills must be embedded across the business. While the governance of the technology platform rests under the CDO’s wing, the significance of insight means data savviness is now everyone’s responsibility. Smart CDOs will ensure this gestation of skills takes place through 2023.

“For me, that's absolutely crucial,” he says. “It's not just my space, it's not just my team’s. We want to democratise data. We want to make great decisions available everywhere. Everyone needs to have informed evidence and facts and be able to build capabilities, give insights, interpret information and make the right decisions. Responsibility for data has got to be held everywhere across the organisation.”

Welcome to issue 33 of IT Pro 20/20, distilling the most important themes of the previous month into a simple, easy-to-read package.

For our first issue of the year, we’ve taken a deep dive into the most compelling industry trends for 2023. From the generative AI powering a whole lot more than just ChatGPT, to the freshest face in the C-suite, there’s plenty for technologists to grasp ahead of what looks to be another whirlwind year for the tech sector.

We’ll also look at E-Ink and why, after 50 years, it’s only now starting to truly take off, plus why blockbuster mergers and acquisitions could soon be a distant memory.

The next IT Pro 20/20 will be available in March – previous issues can be found here. If you would like to receive each issue in your inbox as they release, you can subscribe to our mailing list here.

As a data leader, you already know that data plays a key role in your company’s digital transformation. But it can be frustrating to manage this resource across many channels within your organisation.

A recent survey of 300 data leaders (CDOs, CAOs and CDAOs) in the United States, Europe and Asia Pacific showed that companies juggle data across five different data management solutions on average, which has prompted 46% of those surveyed to invest in new technology to integrate multiple solutions over the past two years.

Download this paper now to see why it's time to prioritise more than just solution integration, and why you need to integrate business and culture objectives to benefit your organisation and its data consumers.

Provided by

Your company deals with sensitive information — all companies do. Whether it’s customers’ private information, your company’s financial records and accounts, or that new top-secret project you don’t want getting out just yet, keeping your data secure is a top priority. Unfortunately, it seems to be getting harder all the time.

We’ve seen many major data breaches at many big name companies, including T-Mobile, Microsoft, and General Electric. Even the Internal Revenue Service (IRS) fell victim to a major data breach.

It doesn’t matter if you’re in charge of a Fortune 500 company or a small business, you must protect your data. While there's no foolproof way to secure all of your sensitive data, these steps can help prevent a data breach.

Data breach defined

Although the exact definition can vary between regulators, a data breach usually occurs when any unauthorised party gains access to confidential information.

The term usually covers everything from one of your own employees logging into a file without permission, to cyber criminals hacking into your systems and taking the personal information of employees and customers.

Normally a breach also involves the failure of a security layer, which results in the accidental or purposeful destruction, alteration, theft, or disclosure of protected data. It is considered a breach if an unauthorised individual simply looks at the data. The consequences can be severe for business, regardless of its exact nature.

The consequences of a data breach

The consequences of a data breach can be as varied as the different forms of breaches themselves. It could be as small as a single employee finding out his peers’ salaries and threatening to sue for higher pay. Alternatively, it could be as severe as cyber criminals or hackers accessing your system’s files and encrypting them before demanding a ransom.

If you follow recent news, you might have seen a variety of data breaches making the headlines which often involve accessing customer data, like addresses, names, social security numbers, and even credit card numbers. These breaches can cost the companies who are affected millions of dollars in lawsuits and lost business.

After a breach has occurred, detecting, defining, and recovering from the incident can be a long and slow process for an organisation. Although the consequences of this kind of leak can be brutal for larger corporations, they can be the death knell for a small business. The best strategy is to be prepared if it occurs and prevent it from ever happening.

10 steps to keeping your data safe

While there is no surefire way to eliminate all data breaches, these nine steps will help your business prevent catastrophic leaks.

1. Hire a professional

You probably have a financial controller or accountant in charge of payroll and accounts payable. Why would you not have a specialized IT security person to safeguard the entirety of your business?

Technology changes quickly, and you need someone dedicated and accountable who can find your vulnerabilities and help you shore them up. This IT watchdog can be your best bet in defending your business from external and internal data breaches.

2. Separate your business and personal accounts

Keeping your business and personal accounts separate should be obvious, but we’re not talking just about bank accounts and credit cards. This separation also goes for all of your accounts, including email and data storage. Keep them separate and have unique passwords for everything.

The last thing you need is someone hacking into your personal email and suddenly having access to sensitive business data.

Make sure everyone in your company follows this practice too. It only takes one small mistake to expose the whole company.

3. Checkout FINRA

The Financial Industry Regulatory Authority (FINRA) is a government-authorized nonprofit organization that oversees US broker-dealers and has a pretty good handle on what it takes to have top-notch cybersecurity.

It has put together a Small Firm Cybersecurity Checklist that’s a handy tool for any business looking to up its cyber security game. It’s free to download and could help you secure some aspects of your business you hadn’t thought about before.

4. Restrict access as much as possible

As much as possible, limit each employee's access to data. That means any employee who doesn’t need access to a program or data file doesn’t get access.

By all means, provide each employee with the permissions she needs to do their job, but limit anything that isn’t necessary. The fewer people accessing data, the lower the risk of a breach—accidental or otherwise.

5. Minimize your data

Think of your business as your home, and all that data is the stuff in your home. The more things you have cluttering an area up, the harder it is to keep track of the important stuff. It’s time to clean up.

Eliminate old programs or data files that serve no purpose. Team up with your IT security officer to establish the proper procedures for identifying and eliminating unnecessary files.

Keeping your data tidy can also help you identify a breach problem sooner too.

6. Encryption

Today’s technology offers plenty of avenues for encrypting your data. Use them. Don’t just encrypt data sitting in files; use encryption for files on the move through email and other means.

7. Educate your employees

One of the most common ways data breaches occur is through an honest employee mistake. You must teach your employees how to create unbreakable passwords and how to identify potential phishing scams and other security threats.

Train, test, and educate your employees on the importance of information security. It can be a tough job to get your employees on board with cyber security training, but it’s essential.

8. Get the C-suite on board

Cyber security awareness and understanding must start at the top. If the company’s executives don’t fully understand the threat and consequences, it will create a difficult hurdle for each step of the process.

Cyber security is an investment, and like any investment a company makes, the C-suite needs to see the benefit before they’ll be willing to pay for it.

9. Don’t ignore hard copies

Data breaches don’t always happen online. Printed documents can cause equal damage in the wrong hands. Keep all sensitive files in a locked cabinet and designate a gatekeeper who only gives access to those who truly need it.

Also, invest in a quality crosscut paper shredder to properly destroy any documents you need to eliminate.

10. Have a data breach response plan

While preventing a data breach is always the number one goal, your company needs to have an action plan for handling a breach. The ability to quickly detect a breach can save millions of dollars in some situations.

Talk with a cyber security professional and establish steps to identify, contain, and then recover from a data breach.

Data protection is an ongoing process

Once you’ve completed all 10 steps, get ready to do them all over again. Technology changes quickly, and cyber criminals are continually finding new ways to break into sensitive information and profit from it.

If you want to avoid becoming a victim of the next big data breach, stay one step ahead of the game. Companies that become complacent with their cyber security end up being vulnerable to attack.

What is data and why is it so important? Data expert Caroline Carruthers will use her experience both as a global data consultant and as a former Chief Data Officer to explain why organisations need to start taking data strategies seriously. Drawing on the Carruthers and Jackson model, used by public, private and charity sector bodies, Caroline will give the audience insights into how different elements of an organisation can work together to tackle barriers and gain more from a data strategy.

Caroline Carruthers

Co-founder, Carruthers and Jackson

Caroline Carruthers is a data expert and co-founder of data consultancy firm Carruthers and Jackson. She was one of the first women to take on the role of Chief Data Officer in the UK public sector for Network Rail. Caroline has authored several best-selling books on the role of data in organisations and now consults with public, private and charity sector organisations on how to get the most out of their data.

Whether it’s the new oil, the new gold, or the new depleted uranium, most businesses agree that data is becoming more and more critical to continued success. Organisations across the world are investing in their data competency, and throughout this process, the office of the CDO has been thrust to the fore.

But what exactly does a CDO do? What impact can they have on business? And how much of a priority should it be for businesses to appoint one? To answer these questions, and more, we talked to Caroline Carruthers, author, data expert and one half of data consultancy Carruthers and Jackson. Caroline is also delivering keynote speech at our upcoming virtual conference, IT Pro Live.

Also in this episode, we discuss Amazon and IBM’s decisions to limit their sale of facial recognition technology to police forces, what IBM’s cloud outage means for smaller companies, and whether a recent healthcare data leak could put an early end to video consultations.

Footnotes

News

• IBM to kill its own facial recognition technology
• Amazon halts police use of its facial recognition tech
• IBM Cloud suffers major global outage
• Babylon health admits to leak of GP video consultations

In-depth

• What is big data?
• What is data analytics?
• How data science is transforming business
• The IT Pro Podcast: Bridging the gender data gap
• Using Big Data to transform business processes
• CDO job description: What does a CDO do?
• Need a chief data officer? Maybe you should rent instead of hire
• Panel Profile: Addison Lee CDO Graeme McDermott
• The IT Pro Podcast: How to build a data-driven business
• IT Pro Panel: Building a data-driven business

Subscribe

• Subscribe to The IT Pro Podcast on Apple Podcasts
• Subscribe to The IT Pro Podcast on Google Podcasts
• Subscribe to The IT Pro Podcast on Spotify
• Subscribe to the IT Pro newsletter
• Subscribe to IT Pro 20/20
• Register for IT Pro Live

First coined 15 years ago, DevOps refers to a collection of tools and practices in the software industry that teams adopt to make the software development cycle more efficient and shorter. It is the perfect marriage of software development (Dev) and IT operations (Ops) – and allows development and operations teams to work in unison. This methodology is based on the principles of shared ownership, rapid feedback and the continuous pursuit of workflow automation. 

The core DevOps principles, according to GitLab, are automating the software development life cycle, collaboration and communication, continuous improvement and minimization of waste, and focus on user needs with short feedback loops. "By adopting these principles, organizations can improve code quality, achieve a faster time to market, and engage in better application planning," according to the platform.

Since DevOps was conceptualized, businesses around the world have adopted this methodology in their droves as they migrate away from traditional structured waterfall development methods. The majority of organizations out there use DevOps in at least some way, depending on the source you find, with figures varying from 74% to 83% depending on the source and the research conducted. But the most important organization that matters is your own – and whether adopting DevOps could make a genuine difference to outcomes and productivity levels. 

The origins of DevOps

The traditional waterfall methods of development were falling away in the late 2000s to make way for agile methodologies, with speed over the quality of output more of a priority. Meanwhile, the rise of cloud computing saw platform as a service (PaaS) and infrastructure as a service (IaaS) products maturing into propositions that businesses could adopt. Then continuous integration (CI) began to emerge, according to Steve Mezak, the founder and CEO of Accelerance, a software development outsourcing company.

Concerns grew in the software and development industries that the siloed nature of work could not continue; the process of creating the software could no longer be segmented from deploying and maintaining the software when live. This, according to Atlassian's principal solutions engineer Ian Buchanan "was a fatal level of dysfunction in the industry." 

"They [IT operations and software development communities] railed against the traditional software development model, which called for those who write code to be organizationally and functionally apart from those who deploy and support that code," he wrote in a blog post. "Surely there’s a better way, they said."

The first conference named DevOps Days was held in Belgium in 2009, founded by project manager and agile practitioner Patrick Debois, and the first annual "State of DevOps" report was published at Puppet Labs – a pioneering company in the DevOps industry – by Alanna Brown, vice president of growth at Remote and formerly the senior director for marketing at Puppet.

How does DevOps work?

DevOps methodologies favor releasing small and iterative updates to software rather than massive updates every six to 12 months throughout the same period. This, in turn, feeds into a strategy of bug fixes and new features that allow customers to quickly respond to shifting trends or customer feedback. Should there be new market opportunities or changing demands and behaviors, adopting DevOps principles positions an organization to respond more productively and efficiently. 

Adopting DevOps significantly reduces the waiting time and ensures businesses can be more responsive. If one element needs changing following customer feedback, for example, it can be tweaked quickly and tested immediately – without needing to jump through hoops for approval. There are different rationales as to why DevOps is so popular and effective. Some say it's down to how teams are tasked to collaborate much better while others suggest it's the specific tools and workflows that streamline operations. However the best implementations tend to involve a combination of both. The flexibility of DevOps also makes it arguably better than other project management techniques for software development.

What are the benefits and drawbacks of DevOps?

There have been years of research into the benefits of DevOps for organizations, with the consensus falling on the fact that DevOps practitioners increase the frequency of releases and improve test automation practices. Research published in 2016 suggested as much. The key benefits include greater encouragement of collaboration between departments – which, in turn, improves communication and welfare – as well as continuous releases paving the way for more experimentation and rapid feedback collection.  

It's the frequency of releases that means you can deliver more features to users or customers in a timely manner, while less time is required for testing and quality assurance (QA) than in traditional siloed models. Moving at pace also encourages greater business agility, which lets you respond quickly to market shifts and be better prepared to make quick decisions about the state of the product more confidently. Iterating faster also means you can be ahead of the curve. Even failure, in this model, can be seen as a success. Moving faster means you're more likely to recover from a setback.

Challenges of adopting DevOps, meanwhile, according to the 2016 paper, include the communication structures that prohibit cross-departmental collaboration – and needing to address the culture shift in the organization. Not all industries are also suitable for DevOps, while ambiguity about what DevOps actually means may make adoption that much harder for teams going through the process. Because DevOps requires fairly massive structural changes, it can be difficult for legacy organizations lumbered with inefficiencies to achieve the transition. The concept of reorganizing at a single stroke can be a daunting prospect. The cultural shift can be difficult to manage as well. Workplace attitudes are often deeply ingrained, and it can be hard to convince previously distant teams to collaborate if they don't want to do so. DevOps can also be a tricky concept to sell to the board.

What are some common DevOps tools out there?

While there are no hard and fast rules for which software should be included in a DevOps toolchain, the same options tend to crop up in most DevOps environments. There's a huge variety out there, but these are some of the biggest names in the DevOps business. 

Docker: Docker is the technology that has been spearheading the uptake of containerization within the enterprise. The beauty of containers is that while they're similar to virtual machines, the fact that one OS image can host multiple containers makes it much lighter on resources than spinning up multiple VMs. The reason Docker, in particular, has proved so popular with the container crowd is that it's made it a whole lot easier and safer to create, manage and scale container-based applications, which can save IT departments huge amounts of time and money. Its flexible and scalable nature has also made it popular within the DevOps community.

Jenkins: Jenkins is an open-source tool used for continuous integration of code. It includes support for a huge number of plugins allowing it to work with a massive amount of systems and projects. This is one of the reasons it has become so invaluable to many DevOps toolchains, along with its ability to trigger builds based on a variety of factors. Like many of the tools used in this sector, it's heavily focused on automating as much of the software development process as possible.

Puppet: Used as a configuration management utility, Puppet is central to many DevOps teams. It's excellent for spinning up new servers and virtual machines and features its own language that is used to define the specific system resources needed. Another commonly used alternative to Puppet is Chef, and while they're widely considered to be neck-and-neck in terms of popularity and sophistication, many argue that Puppet is best suited to operational tasks where Chef favors uses that lean towards development.

Data science is one area of the digital sector that is desperately short of talent. In fact, IBM thinks data science will account for 28% of all digital jobs by 2020, but worryingly, the same report revealed that on average, each of these places remains unfilled for up to 45 days due to a lack of talent equipped with the necessary skills.

"Machine learning, big data and data science skills are the most challenging to recruit for, and can potentially create the greatest disruption if not filled," according to IBM's The Quant Crunch report.

The Royal Society found that demand for workers with specialist data skills in the UK has more than tripled over the last five years to 231%, comparable to a general increase of regular workers of 36%.

And with the European Commission forecasting that 100,000 new data-related jobs will be created in the region by 2020, the fact there aren't enough people with the right skills to fill the role is certainly worrying.

What has triggered the data scientist skills gap?

Simply put, there aren't enough data scientists to go around - the demand for data analysis has grown exponentially over the last few years, and there aren't enough people being trained to meet the demand.

This growing gap between demand and available talent has meant that almost half of all European companies are thought to be struggling to fill their data scientist positions, according to recent data published by O'Reilly Media.

The report found that European organisations are heavily investing in products that help improve the accessibility and usability of data. Specifically, 59% are building platforms that allow for data integration and extract, transform and load (ETL) processes. Similarly, 53% said they were already working on data science platforms, while 53% said they were prioritising data preparation and cleaning, something that is core to the development of AI and analytics services.

However, the lack of talent is holding back progress. As much as 47% of the region's organisations are currently struggling to fill their data science roles, with 37% also struggling with data engineering positions. That's compared to just 25% for security-based roles.

The findings are broadly in line with data published by Gartner, which found that organisations are not only unable to source talent for full-time positions, but also do not have enough data scientists consistently available throughout the business.

The amount of data businesses are generating compared to even just a few years ago is staggering, and it's growing fast, with Forbes stating that 90% of data present in the world today was produced in the last two years alone. IDC thinks a mammoth 163 zettabytes of data a year will be generated by business by 2025.

There are more and more devices coming onto the market that scrape data, including wearables, smart home/office devices and that doesn't even include the rising number of people using the internet to shop, interact with brands and more, generating business insights that are powering business decisions.

Although the amount of data is impressive and is a positive thing, what isn't so positive is that data is useless without it first being analysed and then used to inform business transformation. Without the manpower to work out what all of the information means, analytical insights which add value to the business can't be deduced. Rather, if an organisation lacks the tools to coerce information from harnessed data, they may even be better placed abolishing data initiatives and cost-saving, to a degree. Simply, an investment in data tools must be matched by an investment in personnel.

How can data scientists benefit businesses?

Data scientists transform masses of data into analytical insights which help businesses make faster, more informed decisions.

Approximately 80% of the data generated over the next five years will be unstructured, highlighting the need for analytical tools operated by skilled personnel in order to extract insights. Using such tools, data scientists drag numbers and statistics into tables to form predictive models that can simulate a variety of possibilities.

Armed with the knowledge of the potential outcomes that each course of action can result in, businesses can select whichever solution aligns best with their initiatives. Logical, best-case scenario actions can be prescribed that improve overall performance. Moreover, as businesses build a record of performance metrics, an internal database of business-insights is formed, with which businesses are able to base decisions on recurring trends.

Boiled-down, the skill-set of a data scientist consists primarily of mathematics and statistics, programming knowledge and analytical thinking. Automated programmes can replicate these skills in the modern day, however a significant advantage of the data scientist is that they can think creatively. In this emerging field, it is certain that data science can be used in presently undiscovered ways to create further insight.

The possibilities are endless. A creative data scientist may forge new methods of gathering, interpreting and analysing to produce a profitable data strategy. According to Bernard Marr, "the corporate data superstars of the future will be people who can find new data to solve business problems and come up with new and innovative methods of applying data statistics".

In this, it can be said that the impact of data science is only as good as the data scientist. And with the skills gap only widening, sourcing such talent comes at a cost.

What is industry doing to help close the skills gap?

The European Commission has made a commitment to try and fill the data scientist skills gap across the whole of Europe by running sessions exploring what businesses can do to try and make the problem less severe, addressing the demands of organisations and using the data they collect to gain key insights about their customers and how to grow their business.

Businesses are also starting to react to the country's data scientist shortage and are collaborating with other firms and educational establishments to try and close the gap before it becomes too large to manage.

For example, data management and analytics firm SAS has teamed up with HSBC and the Data Lab to introduce an MSc course in Data Science for Business. The course will run at the University of Stirling and has been designed for those looking to start their career in data analytics. It teaches students how to use advanced analytics and apply these skills to real-life scenarios.

"There is a shortage of graduates emerging with the skills to apply the technical aspects of data science and use analytics to make sound business decisions," Dr Kepa Mendibil, course leader of the MSc in Data Science at the University of Stirling's School of Management said.

"Through this course, we have focused on the practical challenges that organisations are experiencing by merging disciplines to develop a teaching programme that makes the link between business, management and data analytics."

The Office of National Statistics is also innovating to ensure the country has the resources it needs, collaborating with universities across the country to implement data science courses, with both MSc and short courses.

Although closing the data scientist skills gap is not something that can be solved immediately, it's important businesses and the government work together to ensure they are making steps towards creating a better-equipped workforce.

The Royal Society's report recommends a fundamental change in education. To provide young people with the necessary platform to pursue data-specialist roles, transferable skills such as communication and problem-solving should be embedded into the national curriculum. And to ensure the skills gap does not completely spiral out of control, the change is needed within the next ten years.

A two-pronged approach is needed - teaching young people about the merits and opportunities of data - starting as early as possible and training existing staff how to analyse data to ensure the true meaning of information companies hold isn't simply lost forever.

What does the data scientist role entail?

While data scientists are in high demand, the role can differ quite substantially from company to company, explains data scientist Jessica Kirkpatrick, at tech recruitment firm Hired.

"In simple terms, data science is taking the scientific method and applying it to a business' data set to derive insights that help the company make strategic decisions," she says. "However, that can mean a wide variety of things in terms of day-to-day duties."

While one data scientist might be tasked with taking raw data from an application or website, for instance and organising it in a database in a way that makes it easy to analyse, another might find themselves running A/B tests on a website to see which version performs better with users.

"For example at Hired, we might pilot a new interview scheduling tool with a small group of clients and see how this new feature affects their ability to hire candidates," Kirkpatrick adds.

"We then compare how efficiently this test group is able to hire candidates with the rest of the clients who do not have access to the tool. This allows us to understand if the tool helps our clients or not and if we should add it as a permanent feature to our product."

But the role is increasingly focused not just on the data, but on company strategy, Kirkpatrick explains, helping executives identify the most efficient areas of the business and which areas require extra resources, or where companies should scale down.

"We have been seeing an increase in [the number of] Chief Data Officer roles in recent years, because companies are realising that it's important to have someone with deep analytical knowledge of the business in the boardroom," Kirkpatrick says.

Chief digital officers (CDO) have a lot on their plates; not only are they responsible for the digital transformation of their organisation and future-proofing of its operations, but they also have to do so without getting in the way of the CIO and CTO.

That could explain why the role isn't that common - some organisations don't 'officially' have a CDO, it's just part and parcel of the CIO and CTO roles. However, there is a growing trend of businesses leaning towards the CDO, with many realising that this area of IT needs specific focus that cannot be bundled into one of the existing roles.

Of course, like all current digital trends, a newfound appreciation of the CDO role has been accelerated by the COVID pandemic. A recent Gartner Board of Directors survey found that 69% of boards have ramped up their digital business initiatives in response to the disruption caused by the pandemic. This includes the mass shift to remote working and the widespread adoption of digital services such as VPNs and video conferencing platforms.

What does a CDO do?

The CDO is tasked with driving their organisation's digital transformation strategy and coordinating all of its efforts to do so. This means following a company vision - one they themselves may not have created - to bring their business into the future and make sure that different departments, from IT to the finance teams, feed into its digital transformation targets.

The key responsibilities include mapping out the major priorities for digital transformation and serving as the point of contact for innovation. But the role also intersects with other areas, with the CDO expected to stay on top of whether an organisation has the digital skills at hand to realise its strategy, as well as evaluate the return on investment (ROI)

Above all, the CDO must ensure the appropriate processes, procedures and technologies are used to their full potential so 'digital' becomes ingrained into the business - rather than something consigned to the back of the mind.

While encompassing a broad brief, a CDO's duties also carry a tremendous amount of responsibility given digital transformation is becoming increasingly key to nearly all businesses' future. It's the norm for CDOs to report straight to a firm's CEO, with far more discretion to act unilaterally than other key figures across a business may have perhaps.

One downside of the role is that CDOs are often confused with chief data officers – another job position that has become increasingly popular in the last few years, as a growing number of organisations discover the significance of data. In fact, when it comes to companies which don’t employ a chief data officer, CDOs can even expect to be burdened with the additional responsibility of the other in order to succeed in their job. Gartner predicts that, by 2023, this will apply to one in two (50%) of CDOs without a chief data officer peer.

According to Mike Rollings, distinguished research vice president at Gartner, both roles are crucial “in accelerating digital business and building a data-driven organisation”.

“Some organisations may not have the luxury of having both roles, but the absence of executive focus on either of these critical endeavors can undermine the other,” he said.

While ​​chief digital officers are often regarded as impermanent roles, often tied in with the length of the process of digital transformation that they are overseeing – it’s not a coincidence that the position has been described as “transformer in chief”.

However, taking on the added role of chief data officer “can solidify the longevity of the role in the organisation”, states Gartner. This is due to the increased demand for data and analytics skills, coupled with the urgency of deploying digital transformation since the start of the pandemic.

According to Rollings, “the success of these endeavors is intertwined”.

“A digital business cannot exist without data and analytics,” he added.

What qualifications does a CDO need?

A CDO doesn't necessarily need a set roster of qualifications to carry out the role. The candidate could come from a varied background and of course, they may not only hold the role of CDO but also partake in activities linked to the company's digital strategy as part of their daily duties.

As digital transformation in becoming more commonplace, if an employee is carrying out some CDO duties as part of their wider role, there is a possibility that they may be promoted to CDO in future, as demands calls for it.

But does the job title really matter? If an employee remains in an alternative role, but still focuses on delivering value for the business through a time of change, does the label need to be so official?

Forrester Research's vice president and principal analyst Nigel Fenwick doesn't think so, as he conveyed in a blog post back in 2015, writing: "Let's not get too hung up on titles what really matters is the ability to combine a deep understanding of the customer with an understanding of how digital technology will drive new sources of customer value."

According to a report by PwC that looked into the qualities a CDO should possess in order to be successful in their role, it highlighted five personality traits that a CDO should have. They are being a progressive thinker, a creative disrupter, a customer advocate, an innovative technologist and a universalist.

Which one an organisation employs is very much dependent on the type of business and what they're trying to achieve, according to PwC.

"European companies are leading the field when it comes to hiring CDOs, with 13% having appointed someone for the role. However, as digital transformation continues to move up business agendas, it's time for those who haven't yet invested to prioritise filling the CDO position, as competition for the best talent will only increase," said Ashley Unwin, PwC's UK and EMEA consulting leader.

"Many organisations struggle to know where to start in recruiting for this newly created role. These five archetypes provide an overview of the different roles and responsibilities a CDO could hold in order to help companies work out the best fit for their specific needs and drive their organisation towards a digital future."

How much do CDOs earn?

Given how new the role itself is and the fact that many CDOs wear hats with different names, it's hard to say for sure what the earning potential is. Salary Expert reckons that average CDO salary in London is £143,446, around 24% higher than the average CDO salary in UK. An entry-level CDO with 1-3 years of experience can expect to earn around £97,000, according to the website, while a senior level CDO with 8+ years of experience will earn an average salary of £182,000.

When looking from afar, it may seem to many that a CIO got to the position they're in because they followed a structured pathway: study an IT-related degree, use the skills practically in their first job, and slowly but surely move up the ladder to become a C-level executive and part of an organisation's board.

However, in reality this is almost never the case.

For many CIOs, their first taste of IT, and their initial interest in the subject, often comes later on in a professional career. In this IT Pro Panel, we look at three IT leaders who have all reached the top of their profession in different ways.

The South East Asia exploration - Amnesty International CIO John Gillespie

Gillespie's career began in sales and marketing, before he then became a business analyst. It was only after a move to South East Asia where a role in IT really transpired as a possible career path. It was here that he became ICT advisory of The Asia Foundation in Cambodia, and then IT manager at Nam Theun 2 Power Company in Laos.

"I ended up running the IT for what was a big hydro electric construction project and I discovered that I loved that, and loved bringing the magic and power of technology to an organisation and seeing that impact on the way that people worked and the way the organisation functioned," Gillespie tells IT Pro.

He adds that this was his first real flavour of IT, setting up an organisation from scratch.

"We started with an empty server room and a big pile of cardboard boxes that would hold up the servers and all of the bits and pieces; we had a two-year period where we build up a complete organisation with all of the classic accounting systems, websites and email and that's where I got a chance to see the whole thing from end-to-end," he states.

After that, Gillespie returned to the UK, and following a few roles in consulting, architecture and in business analysis, he heard about the job at Amnesty International, which was "far too good to turn down". Gillespie has been CIO of the organisation for almost five years now.

The science-IT skills overlap - The Francis Crick Institute CIO Alison Davis

Davis knew she liked IT at a relatively early age: she had heard that to be good at IT she needed analytical and logical skills - two of the things she says were her strong points at school.

However, she enjoyed science even more and took a liking to chemistry in particular.

"The beauty of this was that if you started a degree in 1981, you could get into IT from a variety of disciplines as long as you could demonstrate you had a numerate mindset and aptitude tests would prove that," she explains.

Davis did study Computer Studies as an A/O level, which was a halfway house between GCSEs and A-Levels, but she went into chemistry for further education and then ended up at GlaxoSmithKline (GSK).

"I knew that as I was at a big company, I'd have the freedom to move and develop a career that wasn't a pure bench chemist," she says.

When GSK decided it needed a helpdesk in 1986, Davis got her big break.

"I had been a local lab expert fixing the printer and sorting things out for people and there was an advert for the helpdesk and I didn't know I wanted to leave chemistry, but I knew at some point I didn't want to continue, so I applied for it," she tells IT Pro.

These were the very early days of having a helpdesk - before the IBM PC got real traction and before ITIL was around - but it was here that Davis discovered a lot about IT.

Eventually she became CIO at the Francis Crick Institute, which she says is the ideal place for her because it enables her to stay around the science field, but work in IT.

Unearthing the inner IT in you - Wrightington Wigan Leigh NHS Foundation Trust acting associate director of IM&T, Mark Singleton

Singleton believes he didn't have the self-discipline to go to university after his A-Levels, and therefore his first job was as a temp working for the NHS. He did study IT and business at GCSE and A-Levels, though, which came in handy when he started filing things because he could create a quick-access database.

"It was doing things like filing records, so it wasn't anything extravagant, and the NHS organisation that I was at started relying on these as it improved the way of working and it was here that they recognised my IT skills," he says.

From there, Singleton produced reports and did some analysis for the NHS and this kickstarted his analytical experience in healthcare. Since then he worked for a few other NHS organisations before moving to Wrightington Wigan Leigh NHS Foundation Trust, where he worked his way up to head of BI and associate director of IM&T.

"I worked my way up through various data and analytical roles and despite not having any referents in my career I did decide to do a degree while I was working, and I'm currently on the last year of completing my masters," he tells IT Pro.

"So I absolutely believe in further education, but I believe unless you want to be a doctor or solicitor, in terms of IT, it's better to get work experience first and decide where you want to be and what area you want to be in, and then get the qualifications that wrap around that."

All three IT leaders took different routes to get where they are, proving that many skills from different departments are interchangeable with IT, and that progress and development can be made even at a late stage in IT.

If you're a senior IT decision-maker and you'd like to apply to be part of the IT Pro Panel, please email panel@itpro.co.uk.

London houses twice the total number of AI companies of Paris and Berlin combined, according to a new report launched on the eve of London Tech Week.

The UK's capital is home to 758 firms for which AI is a central part of their business models, a survey commissioned by the Mayor of London has discovered.

These companies span 30 industries, with focuses on insurance, finance and law, according to AI advice platform CognitionX, which compiled the London: The AI Growth Capital of Europe report. Investment in London's AI businesses grew by 50% last year to surpass 200 million, including a 48 million funding round for smartphone health app Babylon Health and a 27 million Series A round for authentication tool Callsign.

Meanwhile, 32% of the firms having at least one founder from a minority ethnic background, and 25% with at least one female founder.

Speaking at the launch of London Tech Week, Mayor of London Sadiq Khan said: "There are few areas of innovation that have the power to define our future economy and society more than artificial intelligence. As mayor, it is my goal to ensure both that London is at the forefront of developing and capitalising on these new technologies, and that all Londoners can benefit from the opportunities that they create.

"The research describes a city with a rich technology ecosystem, a strong pipeline of AI innovation and an academic and investment base set up for the long term."

Khan hopes to capitalise on London's AI knowledge to turn the capital into a smart city, a goal he announced last year.

Today he outlined a roadmap featuring more than 20 initiatives to achieve that aim, including proposals to bring full fibre broadband to all new homes, working with TfL and councils to roll out 4G more widely across public buildings, and adding smart sensors to public infrastructure like lamp posts, as well as supporting bids to test 5G.

The use of open data at TfL and City Hall to build apps for the public is set to continue, with Khan allocating 365,000 to improve data-sharing among public services.

"We are really looking forward to working with startups and other partners to ensure Londoners have good access to homes and can easily travel around the city," Lilli Matson, director of transport strategy at TfL, said. "This is central to quickly developing innovative solutions to deliver more homes and offer better options to encourage more journeys to be made by walking, cycling and public transport."

The mayor also plans to invest 365,000 to improve data-sharing among public services and proposes to draw up a pan-London cybersecurity strategy to safeguard people's personal information, a plan that would bring together bodies including the London Digital Security Centre and the London Cyber Innovation Centre.

London's chief digital officer, Theo Blackwell, said: "We will strike a new deal with citizens on use of their data so when it is appropriate to share data, it is done in a trusted, safe way and for public benefit."

Khan will also launch the Civic Innovation Challenge, first announced in December, to grant startups 15,000 to test ideas that are designed to tackle social challenges like inequality, climate change and the capital's ageing population.

"As one of the world's leading technology hubs, we need to be bold and think big, to experiment and try things out that have not been done elsewhere," he said. "I see London's future as a global 'test-bed city' for civic innovation, where the best ideas are developed, amplified and scaled."

Responsibility for the UK government's technology initiatives should be moved wholesale to the Department for Digital, Culture, Media and Sport, according to a major think tank.

Whitehall should shift the Government Digital Service (GDS), which currently operates under the oversight of the Cabinet Office, into DCMS, urged a report by think tank Policy Exchange.

DCMS has already stripped GDS of its data policy and governance functions, tucking both under its wing over Easter.

But having the Cabinet Office in charge of some digital initiatives and DCMS leading on others which "adds organisational complexity, diffuses accountability and slows the overall progress of transformation" the think tank contends, adding that DCMS's sport portfolio should be moved to the Department of Health.

"Bringing all of GDS together into DCMS would help create a more coherent Department for Digital and Culture, able to focus on all aspects of digital transformation across the public and private sector," its report, called The Smart State, reads.

GDS started in 2011 with a remit to digitise government's public services, improving access for millions of people. It centralised government body websites under Gov.uk and digitised services such as passport renewals, driving licence applications and registering to vote.

Led by Francis Maude, the Cabinet Office minister at the time, and GDS chief Mike Bracken, GDS also spearheaded concepts like using smaller suppliers for IT contracts and re-using digital services built in-house (government-as-a-platform) to reduce costly duplication of services.

Since their departures, however, GDS has moved into a more advisory role, with departments leading on their own digital initiatives.

Writing the foreword to yesterday's report, Maude said: "There are worrying signs that in recent years progress has slipped. Without constant pressure from the centre, the natural tendency in any large organisation is for individual departments to slip back into defensive isolation.

"Government-as-a-platform will not happen without clear direction from the top. It is time to reboot. Government 2.0 is overdue."

Under Policy Exchange's recommendations, DCMS would gain "full leadership" of projects like the AI Sector Deal, a public-private partnership to fund innovation in machine learning.

GDS would also be tasked with creating single digital government accounts for citizens over the next three years that list all the services a citizen is using, who has access to their data, and allow them to see the level of performance of their local public services.

The concept of digital IDs is already being explored by GDS with the Verify project, which it aims to bring to 25 million people by 2020, according to its Government Transformation Strategy.

But other departments, including HMRC, are developing their own identity tools in tandem.

Policy Exchange would like to hand GDS more power over departments, suggesting that it score departments' progress on delivering digital transformation objectives, and work with the government's chief data officer to open up public data through common standards.

It also recommends that GDS work with the UK data watchdog - the Information Commissioner's Office, and the new Centre for Data Ethics on creating 'privacy by default' guidelines for public bodies that handle people's personal data.

IT Pro has approached the Cabinet Office and DCMS for comment.

When ageing infrastructure was preventing Gatwick Airport's modernisation drive, it turned to HPE to help transform its entire network in just 18 months without any downtime.

The project's urgency was dictated by Gatwick's old network rapidly approaching end-of-service-life, as well as the need to upgrade cyber defences and support a denser deployment of cameras, sensors and other technologies across the airport.

An added dimension to the complexity of the rollout came from the fact that the airport couldn't afford to have its daily operations impacted by downtime, a challenge not many companies let alone airports would readily take on.

"My appetite for risk is maybe higher than [our] competitors," acknowledges Gatwick's CIO, Cathal Corcoran.

If the core functions of an airport are to ensure planes fly, passengers get where they're going, and their bags get there with them, then a modern airport also serves as a "glorified shopping mall", Corcoran contends, as well as a building site. "We're always knocking something down, we're always building something."

Gatwick required a modern foundation to support all these operations, but the trouble was that its previous network wasn't up to scratch.

"It was running well and silently but with a lot of manual intervention - it took a lot of hand-holding, it took a lot of maintenance," Corcoran tells IT Pro.

However, it couldn't support the modern technology Gatwick wanted to deploy - including 4,500 CCTV cameras and a multitude of sensors to monitor everything from bin overflows to where passengers would go while waiting for their flights.

While the airport carried out RFPs with "all of the main players", Corcoran ultimately chose HPE's Aruba network in a deal worth $15 million, as well as the giant's IT services arm, Pointnext, as its implementation partner.

"The key reason we went for it was the end-to-end offering, and by that I mean the strategy we bought was HPE kit, HPE architecture, HPE design, and it was the HPE Pointnext organisation that put it in," he says.

Moonlight maintenance

To meet the 18-month deadline, Pointnext and Gatwick's own engineers worked in four-hour windows overnight to take out between 40 and 50 switches for up to four days every week, compared to between four and six per night on a more conventional network implementation.

Making the timescale harder to stick to was the fact that Corcoran's team had to avoid bank holidays when the airport sees a spike in travel and that the four-hour window included two hours for rollbacks. The team also had to "roll with operational issues", flexing around any airfield problems the airport experienced.

The stakes were high. Any errors made in these overnight windows could impact the first wave of planes leaving Gatwick the next morning, which could create a domino effect of delayed departures almost certainly making the news headlines.

"If you get this wrong the airport is going to have a very, very bad day and that may continue into a second bad day," confirms Corcoran.

He and Gatwick's incident management team organised airport-wide "fire drills" to ensure everyone from baggage handlers to retail employees knew the manual processes in the event of a network outage, to mitigate the risk of any downtime, something he believes enabled the project to move as fast as it did.

Despite the challenges, Pointnext and Gatwick completed the deployment on time in March 2018, racking up 10,000 man hours in the process.

Marc Waters, MD of HPE UK & Ireland, tells IT Pro: "What was impressive on this was the speed. A core network refresh of a 24/7 operation delivered in 18 months... requires teamwork and courage on both sides, at all levels, from the team on the ground, the programme management team and the senior team in terms of delivering it."

Cutting down plane turnaround times

With hundreds of VPNs to support multi-tenancy, capacity for the routing of millions of IPv4 addresses, dozens of 12900E switches, and hundreds of HPE 5930 and Aruba 2930M switches, Gatwick is now realising the benefits of the Aruba network.

They range from the "less sexy" jobs like monitoring bin levels to more transformative use cases, such as adding sensors to fuel trucks, baggage transport and planes themselves to check if a plane is being fuelled on time and baggage loaded on time, removing the delay created by human input of this data.

These will help Gatwick achieve its aim of having planes ready to fly again 30 minutes after landing, Corcoran says, which is considered best-in-class.

"We're looking at how do we do that with late inbound aircraft," he says, "because aircraft can come back to you late for a multitude of reasons like being late from where they came from, bad weather, flight controller strikes."

Another goal is to analyse 'passenger flow' through the airport, and sell that data to retailers so they can better target potential customers.

Gatwick is using a third party SaaS analytics firm on top of the Aruba Wi-Fi network to look at where passengers go, down to details like "why do they turn left instead of right".

This strengthens another role the airport plays that of IT reseller. "We sell IT services off the back of this," says Corcoran. "We sell IT services to the easyJets of this world, the BAs of this world, the Jamie Olivers of this world."

"What we've been able to achieve here is pretty staggering," adds Waters. "But a lot of that comes down to Gatwick in terms of the technology department here the degree of ambition around what this airport can achieve with technology, and the degree of courage to push the boundaries and go quickly."

Picture of Gatwick Airport/Credit: HPE/J Milstein

The role of the data engineer has never been more vital.

As AI transforms industries, the importance of data engineers has only continued to rise, with businesses relying on their expertise to unlock the full potential of their data, much of which can be leveraged for new ends.

From constructing robust data pipelines to ensuring information is accessible and secure, data engineers handle the technical groundwork that enables effective decision-making.

Their work is crucial not only for day-to-day operations but also for long-term strategic growth, as companies invest in advanced data architecture to stay competitive in an evolving marketplace.

What does a data engineer do?

Data engineers are responsible for designing, building, and maintaining the frameworks that allow organizations to store, process, and analyse their data effectively, which includes constructing scalable data pipelines, implementing ETL (Extract, Transform, Load) processes, and ensuring that data is readily available for analysts, scientists, and others.

They also take on the critical task of monitoring and optimizing these systems, troubleshooting issues, and improving performance to keep up with growing data demands. In essence, they create the foundation upon which data-driven decisions are built.

A key distinction of the data engineer’s role lies in their technical focus on infrastructure and operations, which sets them apart from other data-related positions. For example, data scientists primarily concentrate on analyzing data, developing predictive models, and extracting actionable insights, whereas data engineers ensure the data they work with is clean, reliable, and accessible.

While there is some overlap between data engineers and data architects (as well as other data-oriented roles), they serve different purposes.

Data architects are typically involved in high-level planning, creating blueprints for an organisation’s data framework and ensuring it aligns with any relevant business goals. Data engineers, on the other hand, are responsible for implementing and maintaining these frameworks, handling the day-to-day operations, and adapting systems as requirements evolve, such as through leveraging AI.

According to the UK government’s Digital, Data and Technology (DDaT) Capability Framework, data engineers focus heavily on operationalizing secure and efficient data flows, further distinguishing their role within the data ecosystem.

Data engineering is popular right now

The demand for data engineers has skyrocketed as organizations increasingly prioritise data-driven strategies and AI integration, driven in part by excitement over generative AI.

In the UK, the data sector is thriving, with over the UK’s £72.3 billion ($92 billion) AI market driving demand for skilled data engineers. Businesses are investing heavily in data infrastructure to harness the power of AI, creating a strong need for professionals who can design such systems.

Despite broader economic challenges and slower hiring in some industries, the job market for data engineers remains robust – and seems likely to stay that way.

Organizations across finance, health care, retail, and tech are actively seeking skilled candidates to build the infrastructure necessary to unlock actionable insights, ensuring that data engineers are more in demand than ever.

What skills are necessary for data engineers?

Data engineering is not only in high demand but also offers competitive salaries that reflect the critical nature of the role. In the UK, the median salary for a data engineer is £70,000 per year, with top-tier professionals earning significantly more, per ITJobsWatch. Data engineers in the US can expect to earn more, with average salaries ranging from $85,000 to 135,000 per Glassdoor data.

Salaries have seen a steady upward trend in recent years, driven by the growing reliance on data infrastructure and the need for experienced engineers.

To excel as a data engineer, candidates need a robust set of technical and analytical skills.

Proficiency in programming languages such as Python, SQL, and Java is essential for creating and managing data pipelines, while expertise in ETL processes and familiarity with cloud platforms like AWS, Azure, or Google Cloud are also highly valuable. Additionally, engineers are likely to need a strong grasp of database management systems, such as PostgreSQL and MongoDB, to ensure data is stored securely and efficiently.

Soft skills also play a vital role in a data engineer’s success, as they do in many jobs. Notably, collaboration is key, as data engineers work closely with data scientists, analysts, and architects to align data systems with organizational goals. Problem-solving and critical thinking are crucial for troubleshooting and optimizing systems, while an understanding of governance and data security ensures compliance with regulations.

Of course, these are only some of the skills necessary, and each business (or organization) will want something slightly different to another.

Data engineering offers a clear path for professional growth, starting with entry-level roles such as junior data engineer or analytics engineer, which can offer hands-on experience in building and maintaining data systems, often under the guidance of senior team members.

As skills and expertise develop, professionals could advance to roles such as senior data engineer or lead data engineer, where they take on more complex projects and mentor junior colleagues. The importance of data engineers is only expected to grow as businesses continue to rely on data to make decisions and enhance their operations.

With AI’s adoption accelerating, data engineers have a pivotal role in creating the infrastructure required to support advanced analytics and machine learning models.

Helping companies more efficiently use their data is set to become a key industry in the coming years, and there will be plenty of room to grow alongside the section for anyone thinking of becoming a data engineer in 2026 and beyond.

A new research report from Couchbase has investigated enterprises' reactions to digital disruption, surveying 450 heads of digital transformation such as CIOs, CDOs and CTOs in the US, UK, France and Germany.

It shows that the vast majority of industries are facing digital disruption, but that in practice, transformation projects often aren't living up to expectations.

Here, we summarise some of the key findings in five charts.

Regardless of its scale, there is no doubt that digital technology has the potential to disrupt industries across the world. Indeed, 73% of digital leaders believe that their industry is being disrupted by digital technology, while a further 16% say that it's only a matter of time.

This disruption is reflected in spending. On average, organisations spent $5.67 million on digital innovation and transformation projects in the last 12 months, with 15% spending over $10 million.

As the scale of investment continues to increase within organisations, there will clearly be a great deal more focus on whether objectives are being met and benefits realised.

No matter its scale, digital transformation needs clear goals in order to justify the spend. The most important benefit of digital transformation projects that respondents identified was improving customer experience, with 94% claiming it as "very" or "somewhat" important.

Other significant benefits identified included improving the experience of end-users in the business, making internal processes such as HR, the supply chain and finance more efficient, and creating services and experiences that stand out from the competition.

Yet despite all the promises of digital innovation, projects are often falling short of expectations. The great danger for transformation teams is that sizeable investments create results that, while welcome, are still only incremental improvements rather than revolutionary transformations. Almost half of organisations (48%) have only been able to make small improvements in customer experience through digital projects.

At the other end of the scale, 4% claim to have revolutionised the experience to the extent that it's unique in their industry.

Against this backdrop, digital teams need to use every resource at their disposal, in particular data, to ensure that they can provide truly revolutionary improvements to the business. The use of data is critical to new digital services, but also the ability to use that data in the right way at the right time.

Yet despite its importance, 90% of digital decision-makers feel that their ambitions to use data for new digital services are curtailed by a lack of resources, the complexity of using multiple technologies or reliance on legacy database technology.

Picture: Shutterstock

Addison Lee's newly-appointed CIO wants the taxi firm to make better use of data and machine learning to delight its diverse customer base.

Ian Cohen joined Addison Lee just one month ago, replacing the company's longtime CTO, Peter Ingram, and is charged with looking after all the company's technology from frontend customer experience through to backend infrastructure.

However, his priority lies with customer engagement, telling IT Pro that the attraction of joining the firm is in building digital products flexible enough to benefit three different types of customers.

"The individual in the car is our customer but the driver is our customer as well," he said. "We focus on keeping both of those parties incredibly engaged and fulfilled and building solutions to both their benefit."

"We also have the third corner of a customer triangle, which is any [corporation] where we have a network relationship - whether that's an affiliate or a corporate client or a travel management company they're also our customer," he explained.

"The challenges are around building digital solutions to delight that very diverse audience. But that's the attraction."

Doing more with data

Describing Addison Lee as a "hugely data-driven business", Cohen will seek to make better use of data to fuel these new applications.

"The levers we need to pull around data are manifold and frankly we don't do enough in that space," he admitted, saying he will look at machine learning, robotics and artificial intelligence (AI) to help improve route planning, amid other types of applications.

"This is an organisation that has grown up from how you allocate jobs based on routes and plans that were originally in a controller's head, that has moved onto complex algorithms and software that hierarchically allocates journeys," he said.

"The datasets we'll start to ingest, the insights we'll start to assemble, go beyond what traditional allocation systems do."

Working alongside a CDO

Cohen will work alongside Addison Lee's existing CDO on the firm's use of data, and hopes he'll be able to bring a different perspective too.

"Our approach on data is hugely operational because we are an operationally-geared business, but increasingly our focus on product and customer engagement and customer delight means we'll look at a variety of new datasets and metrics to drive that forwards," Cohen explained.

Calling the various tech job titles of CTO, CIO, CDO and others "pretty irrelevant", the former CTO of FT.com is instead focused on taking the right approach to tech, starting with his first town hall for the company's developers that he held last week.

"I didn't really mention much about technology, about servers or backends or development frameworks," he said. "What we talked about was building a compelling premium customer experience, how we take the brand and the values of Addison Lee onto a global stage, how we do that at scale and at pace."

Uber's not our rival, but we'll take their developers

Working with 80 of the FTSE 100 companies, Cohen doesn't see Addison Lee as an Uber rival, and said his new firm won't be "racing to the bottom" like "everybody else" in its space.

Instead he hailed Addison Lee's "startup mentality", despite its 40-year heritage, claiming it's building a "customer focused digital mobility solution which is practically unique".

"There's absolutely no doubt that Uber changed the market and the arrival of Uber has caused a shift in focus," he added, "but the proposition that Addison Lee has is different.

"[It] isn't so much around a hailing-type operation - though we do have that - as around a trusted, engaged and service centric model that is different to Uber."

However, Cohen said any developers thinking of joining the ride-hailing giant would be welcome at Addison Lee's London office.

"You can pack your bags and go to San Francisco and hope to get a job with Uber, Lyft or whoever, or you could come and work with Addison Lee and help shape the future of mobility from inside the greatest capital city in the world," he said.

Picture of Ian Cohen, courtesy of Addison Lee

The Chief Data Officer (CDO) of independent investment research firm Morningstar, James Rhodes says that the key skill needed for a CDO to succeed is an understanding of the business they're in.

"It's great having all of this data, but what are you going to do with it? That is a business problem the technology side of using tools and infrastructure is much easier," he says.

Rhodes gives an example of a business leader coming to a CDO with a problem.

"They're not coming to you saying we really need some big data analytics'. They're going to say, I've got a problem and wouldn't it be great if we could sell more of this product".

"You then have to go away and interpret what that actually means, whether you have the data to provide an answer, and then if you do, you have to analyse that data, build the models and get back to them with a solution to their problem," he adds.

So business acumen is incredibly important in order to be the CDO but that doesn't mean a technical background isn't necessary. Rhodes spent more than a decade at IBM in various technical roles, before becoming a CTO at consultancy Rocaton Investment Advisors. As well as redesigning most of the company's systems that were used for research, and moving its infrastructure to a virtual environment, he created a tool for investors to check how their plans compared to their peers.

But Rhodes and his team realised that it was actually asset managers that were logging in and using it as a sales prospective tool. They then redesigned the tool to focus specifically on asset managers, and ran this as a standalone project outside of the main company for about two years. The tool was purchased by Morningstar in May 2016 and this is what led to Rhodes getting the call to become the investment advisory company's first ever CDO.

Rhodes says that as a data geek' he thought it was a great opportunity for him - but emphasises that unlike many companies, the role of data at Morningstar was different.

"It's not like other companies that are trying to monetise the data they have; Morningstar already does thisit's not a company trying to figure out how to do more with data, it's a data company at its core," he states.

A selection of data

The type of data that Morningstar has coming in to the organisation is wide-ranging. It includes mutual fund data, equity data, real time business data and credit card transaction data to name just a few sources.

"Real time data is obviously updated in real time but things like mutual fund prospectuses are only getting updated so often, so the challenge is how we find the balance of getting the systems, technologies and processes in place that can handle the data that is coming in," he says, adding that there is a couple of petabytes of real time data coming in alone.

The investment advisory firm has many ongoing projects, but Rhodes picked out its work in the analytics space as one area he is particularly interested in.

"With our clusters being set up and now being fed with data, and the ability to write Spark algorithms and start analysing all of this is pretty exciting to me; I'm waiting to see what we can come up with now that we have some of the core foundations in place," he states.

The company also has a number of artificial intelligence initiatives on the go.

"Part of what we do is collect vast amounts of data from various different places and use different AI technologies and techniques but layer them on top of our processes to collect and curate our content, creating value-added information out of structured and unstructured data," he says.

But the long-term aim is for AI to be used in other value-added areas not just to automate tasks, Rhodes states.

They're doing it so we should too'

Rhodes urges organisations to not dive straight into big data just because other companies are doing it.

"I'm a big believer that analysing data for the sake of it is not valuable," says Rhodes.

He believes that there has to be an objective in mind, and that this has to serve either the company's end clients the people who use Morningstar's products or data feeds in his case, or its internal clients such as those doing research within the investment advisory firm.

"So it's fundamental to me that we don't do this in a vacuum it has to be connected and evolved with the business at all times," he says.

This means Rhodes has to work closely with the various teams at Morningstar, including those working on products and in research, and the data governance, technology and data science teams.

Picture credit: Steve Hall Hedrich Blessing

Fifteen per cent of chief data officers (CDOs) will become CEO or move into other C-level positions by 2020, according to analyst firm Gartner.

In its second annual CDO survey, Gartner found that early adopters of the CDO role are pioneering a new organisational function to sit alongside IT, business operations, HR and finance.

"The office of the CDO is being established as an operational department with the appropriate staffing, budget and responsibilities," said Debra Logan, vice president and Gartner fellow.

Gartner found that 30% of the surveyed CDOs said they report directly to the CEO, demonstrating the importance placed on the role.

And the value attached to the role has led Gartner to predict that more than a sixth of successful CDOs will move into a CEO, COO, CMO or other C-level role.

Hiring a CDO

Jamie Popkin, a Gartner vice-president, said: "24% of the respondents said that the board of directors wanted the role.

"The board's involvement, together with the responses pointing to a specific crisis or problem, and the 41% of respondents who said that the CEO or CFO wanted the role, makes it safe to assume that this was a major strategic decision that affected business operating models," he added.

Popkin emphasised the importance of CDOs working closely with CIOs - although perhaps surprisingly only 62% of CDOs said they treat the CIO as an ally or partner.

More than half (54%) of the organisations that Gartner surveyed said that they had partially or fully implemented an 'office of a CDO' (OCDO), with a further 20 per cent looking at implementing an OCDO within the next year. However, a fifth of organisations said they were unlikely to implement an OCDO.

The main business objectives of the OCDO were to increase customer intimacy (62%), competitive advantage (60%), and efficiency (54%), while the main organisation-wide responsibilities are oversight of analytics initiatives (69%) and data governance (68%), followed by defining the analytics strategy (64%) and ensuring information reliability and value (64%).

Chief data officers (CDOs) will exist in 90 per cent of enterprises by the end of 2019, Gartner has predicted.

The research firm said the desire to use information in smarter ways will fuel this "sharp rise" in CDOs, who will supervise data use and inform business strategies based on it.

"Business leaders are starting to grasp the huge potential of digital business, and demanding a better return on their organisations' information assets and use of analytics," said Mario Faria, research vice president at Gartner.

"It's a logical step to create an executive position the CDO to handle the many opportunities and responsibilities that arise from industrial-scale collection and harnessing of data."

Yahoo, CitiGroup and Barclays all have CDOs now, said Gartner, while government departments have also been quick to capitalise on this new executive position, with the former chief of the Government Digital Service (GDS), Mike Bracken, adding the CDO role to his responsibilities last March before he left the organisaiton, and the NHS appointed Geraint Lewis as its CDO in 2014.

However, Gartner also predicted that only half of CDOs will be successful by the end of 2019, because of the structural challenges they will face, among other obstacles.

Most CDOs will learn on the job because of the new nature of the role, and will face the difficult task of creating an information strategy with relevant metrics that tie the activities of their team to measurable business outcomes.

Faria said: "With the explosion of datasets everywhere, an important task is determining which information can add business value, drive efficiency or improve risk management.

"The CDO's role will raise expectations of better results from an enterprise information management strategy, with stakeholders wanting a clear idea of the exact mechanics of making success a reality."

To make the role work, Gartner has outlined six recommendations for new CDOs, starting with forming an enterprise information management strategy, then building trust with the CIO, educating senior leaders about the importance of data, establishing key indicators to measure and tying these to business metrics, and lastly adopting formal information asset measures.

Gartner will run a more in-depth CDO programme at the Gartner Enterprise Information & Master Data Management Summit 2016, which takes place between 2-3 March in London.

How do you cope with change? It's a big question with no simple answer. While there is no definitive coping strategy, experts can agree with one simple premise change is the new normal and CIOs, just like any other leading executives, must get used to a constant state of flux.

That was one of the key conclusions from a recent invitation-only event organised by BCS, The Chartered Institute for IT. Evidence from the meeting suggests that successful digital leaders focus on the changing nature of work and on driving organisational change.

That concentration is a matter of urgency. If a CIO does not disrupt the IT organisation or the business, then someone else probably will. One only needs to think of the rise of the CDO and the increased influence of the CMO over technology purchasing.

Gartner predicts CMOs will spend more on technology than CIOs by 2017. The analyst firm also believes that 25 per cent of blue-chip businesses will have a CDO in situ by the end of this year.

CIOs, then, must be aware that other c-suite executives are already trampling on their traditional stamping ground. Without proving their value to the rest of the business, CIOs are in danger of being undermined and usurped.

What's more, the challenge to CIOs is not just confined to the upper echelons of power. Attendees at the BCS event recognised that driving change is tough and the challenge of preparing people for constant transformation, especially younger staff members, is only going to increase.

Conversation turned to the belief that the up-and coming generation have different expectations regarding the use of technology and the working practices of modern business. Younger people, it was suggested, see life differently; they gather experiences and they want to be entrepreneurial.

Retained employees must be careful to avoid blocking such creativity. CIOs and their senior peers should learn from the digitally engaged generation. They should use practices like reverse mentoring to explore and understand how younger people make the most of technology, rather than expecting all workers to conform to agreed standards and principle.

Great ideas can bubble up from any point in the business. C-suite executives must be open enough to recognise that the culture of an organisation, as well as customer demands and business models, will also be in a constant state of flux in the digital era.

There is, in short, no option but to embrace change. CIOs who fail to grab the opportunities created by transformation will soon become an executive spare part.