How the cloud can add value to Big Data

CGI representation of a cloud containing light, signifying data held in the cloud

Cloud computing and big data systems have without doubt been among the most disruptive technology trends of the past decade. These developments have irrevocably changed the way that technology companies operate, and how value is delivered to stakeholders.

With cloud computing, firms have been able to optimise IT operations through significant reduction of the need to invest in on-premise hardware and software, as well as provide businesses with more strategy options than ever before. Digital transformation and the shift to the cloud continues to show momentum, with the public cloud services market growing 29% from the previous year in 2021 according to IDC.

Simultaneously, big data technologies have provided enterprises with an unprecedented avenue of data value from assets. In the past, high-volume, diverse data proved troublesome for enterprises that were used to working with traditional database technology. But great technical developments allowed businesses to reduce the overhead required to transfer raw data into sorted information within a data store, as well as drastically increasing the efficiency, accessibility and speed of mass data processing.

These innovations have allowed for actionable analysis on a range of once-challenging data sources, including web logs, documents and machine sensors. Even so-called ‘dark data' has been revitalised through these technologies.

Big Data systems help organisations solve the trickiest problems, but they generally require a significant up-front and ongoing IT investment, in addition to dealing with the sheer amount of data in more ambitious projects.


2021 Gartner critical capabilities for data integration tools

How to identify the right tool in support of your data management solutions


Therefore, it makes sense that enterprises are turning to cloud providers with expertise in managing and maintaining scalable, and flexible, computing and storage infrastructure. While on-premise data systems are far from defunct, organisations are starting to effectively push the limits of big data analytics at scale by tapping into big data systems hosted on cloud infrastructure.

While many of these early cloud projects involve high volumes of structured data, there are 3 key components that are already enabling extraction of value from massive, diverse data sets on cloud infrastructure:

Cloud Analytical Databases

These cloud-based services, such as Amazon RedShift, are elastic data warehouses optimised for analytics with existing Business Intelligence tools. This type of analytical database includes management as well as solutions monitoring for the provider.

Hosted Hadoop Services

Hadoop clusters are often hosted in the cloud, which negates the need for on-premise infrastructure. Some Hadoop cloud offerings come with additional managed services, like job troubleshooting, software installation, testing and other such services.

Data Integration and Analytics

Databases built for high-performance analytics can make use of cloud techniques like compression, column-based storage and high-speed inserts of structured data, perfect for sophisticated queries and truly insightful business analytics.

Early adopters are already demonstrating how the cloud can improve upon the value proposition of big data, delivering elastic and cost-effective solutions for analysing data at unprecedented scale.

This article was first published on 12/09/17, and has since been updated

Esther Kezia Thorpe

Esther is a freelance media analyst, podcaster, and one-third of Media Voices. She has previously worked as a content marketing lead for Dennis Publishing and the Media Briefing. She writes frequently on topics such as subscriptions and tech developments for industry sites such as Digital Content Next and What’s New in Publishing. She is co-founder of the Publisher Podcast Awards and Publisher Podcast Summit; the first conference and awards dedicated to celebrating and elevating publisher podcasts.