It’s been more than a decade since the UK government abandoned plans to create a new NHS IT system, costing taxpayers billions of pounds. The scheme had been designed to deliver electronic records, digital scanning, and integrated IT systems across hospitals and community care.

For many in both the public and private sectors, that contracting saga lives long in the memory and its failure has now returned front of mind with government plans to create the new NHS Federated Data Platform (FDP).

What is the Federated Data Platform (FDP)?

The government hopes the FDP will allow NHS organizations to gather operational data from across the huge organization’s disparate systems and consolidate it in one place.

Writing on gov.uk last year, National Data Guardian Dr Nicola Byrne explained how the “value of our data will only be realized if it can be organized in such a way that the NHS is able to use it well to improve treatments, services and ultimately our health and care”.

“The programme’s work to support better health and care for all through the better use of our data is too important an ambition to fail,” she added.

However, with the platform value set at £480 million and multiple bidders expected, this new scheme doesn’t come without controversy. Fears over privacy, and concerns about data security, are all driving debate. 

Iain O’Neil, a former NHS director, and now managing partner of health and social care at TPXimpact, believes we will only “truly unlock the potential of NHS data” once the fundamentals are right; this includes data standards, interoperability, and getting data where and when it’s needed most.

He suggests funnelling all NHS patient data into one private firm could be a risk, explaining: “The FDP will work best if it supports an ecosystem of suppliers tackling the difficult challenges the health and care system faces.” 

And looking ahead five to ten years, he adds: “Clinical decisions based on data gathered from the 67 million people plugged into the NHS Service should be ubiquitous, enabling learning and better decision making.”

A revolution for managing and storing data

A major challenge for the FDP will be to gain frontline buy-in from receptionists, clinicians, nurses, and all working with data within the NHS. Unless the culture inside the organization fully adopts it, it won’t achieve its aims, experts argue.

But Alex Case, who previously worked in Downing Street overseeing cross-Whitehall Brexit delivery, believes the platform could “revolutionize the way we store, manage, and utilize healthcare data in the UK” – although he agrees this is only one part of the story. 

“There needs to be a recognition of the need to coordinate and orchestrate a 'patient journey' across the different components of an integrated care system, because that’s the point of taking a consolidated view of the data,” Case, who is now public sector industry principal at Pegasystems, explains.

Many argue implementing cutting-edge technology and infrastructure in partnership with the private sector could allow for a more coordinated healthcare system. Case, however, adds: “It’s crucial the NHS ensures data is collected and shared in a responsible and transparent manner, with robust safeguards in place to protect patient confidentiality. Additionally, there is a need for clear guidelines around data ownership and access to prevent any potential misuse or abuse.”

He argues the platform should be feature-rich, rather than simply focused on data integration, if patients are to be placed on “the right pathway to get the right treatment, at the right time”, while also accounting for NHS workloads, backlogs, and prioritization.

“The government and their technology partners have the responsibility to work together to ensure we deliver the highest standards of ethics and governance so that we can address the healthcare needs across the nation better and more reliably than before,” Case warns.

Experts agree the FDP should lead to better decision-making, given it will break down silos, including CEO of Apheris, Robin Röhm. “By providing access to distributed data sources, stored in different sources and locations, the data analytics platform can then be utilized for a multitude of operational purposes from care coordination to improving medicine supply chains.

“Data is most valuable when it is extensive and detailed. As the current handling of data within the NHS involves sensitive data being stored on many separate systems, this often leads to poor access and visibility of data by stakeholders.

“Federated data ecosystems are the most pressing response to the problem of working across organizational and geographical boundaries as personal data never leaves its secure environment.”

Avoiding the threat of vendor lock-in

The need to keep information in the FDP from being personally identifiable is a major issue for campaigners and patient-led organisations. The security risks are real; for example, according to Palo Alto Networks’ Unit 42 team, healthcare was the third most-targeted sector in the UK, with 9% of all attacks targeting healthcare organisations.

Amanda Brock, CEO at not-for-profit OpenUK, suggests some form of NHS systems standardisation could help manage every aspect of the user journey from appointment booking to service delivery, while updating patient records along the way to be more efficient. But she warns its success depends on its implementation; the NHS must own the platform and it’s built on open source software in a “collaborative and transparent way”. 

She adds public sector platforms frequently “pay lip service to the legal definition of open source using an OSI-approved licence and sharing the source”, but this is “not enough to avoid lock-in”, she states.

“The scale of this project means there are few people that have the skills to lead it successfully,” Brock further explains, “but the risk with appointing a single vendor is of course lock-in. The NHS has spent years fighting against this. 

“We need to find a balance between the current status quo of myriad disjointed platforms and apps, and one single vendor that owns everything in perpetuity and profits off that patient data at our expense.” 

Risking privacy is a price worth paying

The FDP will lead to better patient care, as it makes a person’s complete medical history far more accessible, adds Waseem Ali, formerly CDO at Lloyd’s of London and current CEO of data and analytics consultants Rockborne. 

This is because of its “treatment-first, paperwork-later” approach. Ali’s years of experience with healthcare data, however, shows the use of integrated records has always been an issue, especially amid privacy concerns.

“It’s understandable for people to be hesitant about their data being shared,” he concedes. “However, most people are already sharing more information with Amazon or Facebook, sometimes without even realizing it. 

“So, if you are happy for Amazon to know your consumer preferences and shopping habits, then why aren’t you happy to share your healthcare information with the NHS so that you can receive the optimum level of care?”

Ali talks of the potential benefits that would exist if “healthcare professionals knew everything about a patient”. “This information could be harnessed to pre-empt the types of services they may need, which would be a gold mine for preventative care,” he adds. 

“Yes, there will always be the argument about data privacy, but the number of lives that could be improved, or even saved, is hard to out-argue.”

Delta Lake is an open source framework designed by Databricks that allows businesses to manage and analyze the data they generate in real-time. 

Taking the form of a transactional storage layer, it provides a foundation for storing data and tables on Data Lakehouses – which Databricks refers to as an open data management architecture. This allows for data warehousing and machine learning operations to be applied directly onto a data lake.

It’s the default storage format for all Databricks operations and is an open protocol, with the organization completely open sourcing Delta Lake with the launch of Delta Lake 2.0 in June 2022. 

With Delta Lake 3.0, launched at Databricks Data + AI Summit 2023, Databricks has launched a new unified file format – UniForm – that solves a longstanding compatibility issue in the way businesses store metadata in their data lakes. This launches alongside a bid to remove data silos by addressing connector fragmentation, and performance improvements.

What is Delta Lake?

Introduced as an open source project in 2019, Delta Lake was created to ensure the data that powers real-time insights that organizations crave is genuinely reliable. It was billed at the time as Databricks’ most important innovation, even including the Apache Spark engine for data processing.

The need for Delta Lake has emerged from the way businesses have traditionally managed data in massive and unrefined data lakes. Because these repositories contain both structured and unstructured data, at different scales, any operations running on top of this data – such as data analytics or machine learning – might not be optimized.

Databricks designed Delta Lake to transform these messy data lakes into cleaner ‘Delta Lakes’ with higher-quality data. This ultimately ensures any additional processes or operations performed on top generate much better insights, even if performed in real-time.

“Nearly every company has a data lake they are trying to gain insights from, but data lakes have proven to lack data reliability,” said Databricks cofounder and CEO, Ali Ghodsi. “Delta Lake has eliminated these challenges for hundreds of enterprises. By making Delta Lake open source, developers will be able to easily build reliable data lakes and turn them into ‘Delta Lakes’.”

Delta Lake competes with a number of other services on the market that perform similar functions, including Apache HUDI, Azure Data Lake, and Snowflake.

How does Delta Lake work?

Based in the cloud, Delta Lake is a Spark proprietary extension that allows organizations to create Delta tables by default, whether they’re using Apache Spark or SQL. The open source platform extends the classic parquet data file format with a file-based transaction log, which allows for additional functions. 

Delta Lake features include providing atomicity, consistency, isolation and durability (ACID) transactions on Spark, as well as scalable metadata handling. The former ensures there’s never inconsistent data, while the latter takes advantage of Spark’s distributed processing power to handle masses of metadata at once.

Organizations can also take advantage of streaming and batch unification – by allowing both types of data to land in the same sink. This comes alongside some quality-of-life features such as rollbacks and historical audit trails, as well as the capability to reproduce machine learning experiments.

It comes with a variety of open source connectors, including Apache Flink, Presto and Trino, alongside standalone readers/writers that lets clients for Python, Ruby and Rust write directly into Delta Lake without needing a massive data engine like Apache Spark.

In 2021, Databricks launched Delta Sharing, a protocol that lets different companies share massive data sets securely and in real time. This aims to break the notion of vendor lock-in, as well as break down data silos due to proprietary data formats and computing resources that are required to read data. Users can access shared data through Pandas, Tableau, Presto, and other platforms without the need to use proprietary systems.

What’s new in Delta Lake 3.0?

With the latest iteration of Delta Lake, Databricks is hoping to end the ‘format wars’ that’s long posed challenges for storing and managing data in data lakes. 

Most data (99%) is stored in data lakes using the parquet format, but organizations must choose from one of several competing standards to store all metadata: Delta Lake, Apache Iceberg, and Apache Hudi. Each format handles metadata differently are fundamentally incompatible with one another. Once enterprises choose a format, they’re stuck using it for everything.

With Delta Lake 3.0, Databricks has created the universal format (UniForm), which serves as a unification of the three standards. Customers who use a Databricks data lake will find their data stored in the parquet format, as usual, alongside three different versions of the metadata. It means the Delta Lake data can be read as if it’s stored in either Iceberg or Hudi.

“The metadata is very important, and if you get the metadata wrong, you can’t actually access this stuff,” says Databricks CEO Ali Ghodsi, speaking at Databricks AI + Data Summit 2023. “Since all three projects are open source, we just went and understood exactly how to do it in each of them, and now inside Databricks, when we create data, we create data for all three.”

Also new in the latest iteration of Delta Lake is the Delta Kernel, which addresses connector fragmentation by ensuring connectors are built against a core library that implements Delta specifications. This alleviates the burden of having to update Delta connectors with each new version or protocol change. With one API, developers can keep connectors up-to-date and ensure the latest innovations are pushed out as soon as they’re ready.

The final new addition is the Delta Liquid Clustering, which boosts the performance of reads and writes. With a flexible data layout technique, the Delta Liquid Clustering marks a departure from the decades old hive-style table partitioning system, which uses a fixed layout. It means organizations can take advantage of cost-efficient data clustering as their data lake grows with time.

Snowflake has revealed new enhancements designed to make it easier for developers to build apps on its data cloud platform while reducing reliance on third-party tools.

Following its acquisition of Streamlit in March 2022, an open source app framework in the Python language, the company is now integrating the framework with its own technology. It said this will help developers bring their data and machine learning (ML) models to life as secure and interactive applications on Snowflake.

The cloud data company hopes the Streamlit integration will help developers build powerful applications without the complexity of building and deploying web applications. The integration will help developers create applications with Python using their data in Snowflake, and then deploy and run these applications on Snowflake’s platform before sharing them with business teams.

“As we continue to disrupt application development, we’re giving builders the data access and tools they need to accelerate their pace of innovation securely under Snowflake’s one unified platform,” said Torsten Grabs, director of Product Management at Snowflake.

“Snowflake’s advancements provide developers with the capabilities to build powerful applications, pipelines, and models with the utmost confidence, and eliminate complexity so they can drive value across their organisations with the Data Cloud.”

To further reduce the need for developers to turn to third-party tools, Snowflake is also providing access to Python and its open source libraries to all users and teams, with the general availability of Snowpark for Python. This means developers will be able to use Python to query and process data queries in a data pipeline while remaining on the company’s platform.

Snowflake said that since its public preview announcement in June 2022, Snowpark for Python has seen six times growth in adoption, with hundreds of customers building their data using the platform, including EDF, Sophos, and Northern Trust.

Through Snowpark, developers also get a streamlined architecture that natively supports users’ programming languages of choice, including Java, Scala, SQL, and Python. Snowflake is also releasing Snowpark-optimised warehouses, which are currently in public preview in AWS, so Python developers can run large scale ML training and other memory-intensive procedures directly in Snowflake. They can also use Python Worksheets, which is in a private preview, to develop applications, data pipelines, and ML models inside Snowflake.

Lastly, Snowflake is also making it easier to work with streaming data within a single platform and further eliminate silos for customers. Users are able to boost productivity by onboarding faster with Schema Interface, and execute data pipelines with Serverless Tasks on Snowflake’s platform.

As organisations' data and analytics efforts become more mature, they collect more diverse data and perform more sophisticated analytics.

Today most organisations have either moved to the cloud or are planning to move to the cloud to modernise their data and analytics infrastructures. And in a recent IDC survey, nearly 80% of organisations said they stored more than half of their data in hybrid and multi-cloud infrastructures.

Download this report to learn more about multi-cloud environments and how to achieve seamless cloud data management across them.

Provided by

As organisations seek to become more competitive, they are often moving to cloud data warehouses (CDWs) and cloud data lakes (CDLs) to help manage large volumes of diverse data and perform advanced analytics. In many cases they are using multiple cloud providers to fit their needs.

Organisations like the idea of a multi-cloud environment, which avoids vendor lock-in and helps them stay portable. They also want to take advantage of the best features and services of each cloud provider.

Download this study to see why the respondents agree that cloud is the future of data management and realise the five must-haves to modernise data and analytics.

Provided by

Data is a critical asset for every organisation. But putting your data to work efficiently with the right data management platform can be the difference between success and failure. Oracle’s data management platform enables you to reduce costs and complexity, get more value from your data with a data lake, and save time with automation—businesses spend 68% less time on IT admin tasks with Oracle Autonomous Data Warehouse.

Discover10 benefits of Oracle’s data management platform.

Maximise the value of your data.

Provided by

Provided by

Modernising the data-centric technology stack is critical to achieving super data analytics and getting actionable insights.

The traditional stack is too complex, time-consuming, and expensive, but simplifying the data pipeline, lifecycle, and stack can solve these issues and offer businesses scalability and agility. With a modernised stack, your organisation will also have the ability to address business needs in real-time.

Download this ESG whitepaper to find out:

• Top ten challenges with enterprise data warehouses
• The benefits of modernising your data warehouse
• Seven components of a modern data experience
• The capabilities of data platform Looker

Google Cloud has acquired a London-based startup called Dataform that builds tools to manage data flows for enterprise customers.

The terms of the deal haven't been released, but TechCrunch understands that it is an 'acquihire' with Google keen to take on the company's talent.

The company is described as an "operating system" for data warehouses and some of its co-founders are ex-Google employees. Its platform aims to help data-rich businesses draw insights by mining data stored in warehouses.

This is something that usually requires a team of engineers and analysts, but the Dataform system is about making the process simpler and cheaper for organisations.

This is a growing area of data analytics with companies such as Snowflake recently undergoing a successful IPO. Dataform were close to a series A funding round, but have instead chosen to continue its growth under Google.

Under the terms of the deal, Dataform will continue to operate under its management and focus on BigQuery. The Dataform Web will also be made free for all new users from now on with customers transitioned to the free plan immediately.

"After several conversations with the Google Cloud team it became clear that we are deeply aligned on the importance of serving analysts with the right tools and technology in order to fill what we all perceive as a missed opportunity in existing solutions," co-founder and CTO Guillaume-Henri Huon wrote on Dataform's website.

"At the same time, as a team of just seven, in a complex, competitive and rapidly changing market, we had more ideas than we had people or resources to accomplish. There has always been so much more we wanted to do each quarter than we could achieve.

"With the support of the BigQuery and Cloud Analytics teams and our combined thought leadership and efforts, we felt that together we could achieve something bigger than we could separately".

How do you keep track of canine genetic information, breeding data and a quarter of a billion communications records, all spread across different systems? That’s the challenge being grappled with by Guide Dogs for the Blind, a charity organisation dedicated to helping improve the lives of people who are blind or partially sighted.

Like many organisations, Guide Dogs is in the process of unifying its data assets in order to make them as efficient and valuable as possible, and in this week’s episode, we speak to CIO Gerard McGovern about how the project is going. We look at the importance of data warehousing, the benefits of accessible data, and how data can help identify a good dog.

Footnotes

• How data can help SMBs with digital transformation
• The IT Pro Podcast: How to build a data-driven business
• How to improve database costs, performance and value
• What is data and big data mining? An easy guide
• The IT Pro Podcast: Do you need a CDO?
• How to become a data leader
• Data: A resource much too valuable to leave unprotected
• IT Pro Live: How Pro Football Focus became the NFL's secret weapon
• The IT Pro Podcast: How analytics can keep your business afloat
• What is CRM documentation and how can it benefit your business?
• CRMs listed as top technology priority for growing SMBs
• The IT Pro Podcast: Bridging the gender data gap
• What is agile development?
• What is digital transformation?
• IT Pro Live: Data driven digital transformation
• The IT Pro Podcast: How to build a data-driven business
• Bridging the data gap to digitise Britain’s railway system

Subscribe

• Subscribe to The IT Pro Podcast on Apple Podcasts
• Subscribe to The IT Pro Podcast on Google Podcasts
• Subscribe to The IT Pro Podcast on Spotify
• Subscribe to the IT Pro newsletter
• Subscribe to IT Pro 20/20

Data has become the engine that drives modern business, and collating and analysing that data is a crucial component of many IT departments' duties. Most turn to Enterprise Data Warehouse (EDW) technologies, which offer platforms that allow business to centralise their data for easier analysis and processing.

Teradata is among the most well-known EDW platforms on the market, having spent the last 40 years building its reputation providing on-premise EDW hardware and software for customers including General Motors, P&G, eBay and Boeing. It has now transitioned to a cloud-first model and is now available on all three major public cloud providers, following the addition of Google Cloud Platform support on 22 October 2019.

Back in 2017, however, the company's cloud credentials were not so well-established. That's why when healthcare and pharmaceuticals giant Johnson & Johnson (J&J) decided to move its data stores to a Teradata-powered cloud infrastructure, the plan was met with surprise and skepticism. In the years leading up to the project, J&J's senior manager for data and analytics Irfan Siddiqui says, the company became aware its current on-premise platform would not support its burgeoning data analytics requirements demands at an affordable price for very much longer.

"We [had] been experiencing some challenges and thinking about how we transform the traditional data warehouse into a more modern service, particularly around the flexibility, scalability and cost, and we were searching for a solution," he told a Teradata conference in Denver, Colorado earlier this year.

And so, in 2017 it started to look at migrating its enterprise data warehouse (EDW) system to the cloud, eventually landing on Teradata as the most promising solution provider for its problems.

At that time, the offer of Teradata on AWS was not widely considered mature enough for an enterprise environment, Siddiqui tells Cloud Pro.

"When Teradata released its first machine on AWS, and I said I wanted to do a proof of concept for Teradata in the cloud, people who knew Teradata, their first reaction was, 'What? Why? Really?'."

However, the commitment from Teradata to show its systems could work in the cloud was so strong Siddiqui found the confidence to go into a proof of concept. Initial trials showed promise.

The 80-terabyte a-ha moment

"Most of us know doing a capacity expansion or migration to new hardware takes in the order of six months but [with AWS] we were able to spin up a formal system with 80TB of data in just 20 minutes. That was one of the 'a-ha moments' for us which became the driving force for us to take another step," he says.

J&J set itself five goals in lifting Teradata to the cloud, Siddiqui says: to migrate three data environments and all its applications by the halfway point of 2019; to offer the same or improved performance compared with the on-premise system; and to increase flexibility and scalability while reducing cost.

This posed a sizeable challenge for Siddiqui's team, which aimed to support about 300TB of storage, 50 business applications and 2,500 analytics users on to a system capable of handling more than 200 million queries per month.

It also raised some significant questions.

"How are our applications going to perform? How do we migrate? What happens with downtime, and stability and security?" he says. "We had to address these questions, not just for our leadership team, but all the stakeholders across J&J. We had to show how it would benefit each one of us."

Most applications stay on-prem

Although all the data warehouse workloads would be in the cloud, most of the related analytics applications and data visualisation tools, including Qlik, Talend, Informatica, and Tibco, remained on-premise.

Some applications were split between the cloud and on-premise servers. For example, J&J wanted to spin up application development environments in the cloud when they were required and only pay when using them. "That is the flexibility we did not have our own servers," Siddiqui says.

Given the migration had to follow an upgrade to the data warehouse production environment, deadlines became tight. The team worked for three months more or less continuously. But by the end of June of 2019, it was able to decommission the on-premise data warehouse hardware systems.

The hard work has paid off for Siddiqui and his team. Extract-transform-load jobs now take half the time compared to the on-premise system. Large Tableau workload performance has improved by 60% and another application's data loading was cut from more than three hours to 50 minutes.

Beware the desktop data hoarders

Claudia Imhoff, industry analyst and president of Intelligence Solutions, says it makes sense to put enterprise data warehousing in the cloud in terms of scalability and performance, but there are caveats.

"It's a wonderful place if you have all the data in there. But, unless you're a greenfield company, nobody has all of their data in the cloud. Even if most operational systems are in the cloud, there are so many little spreadsheets that are worth gold to the company, and they're on somebody's desktop," she says.

"There are arguments for bringing the data into the cloud. It is this amorphous thing, and you don't even know where the data is being stored. And you don't care, as long as you get access to it. Some of it's in Azure, some of it's in AWS, and some of it is in fill-in-the-blank cloud. And, by the way, some of it is still on-premise. Can you bring the data together virtually and analyse it? Good luck with that," she adds.

To succeed in getting data warehousing and analytics into the cloud, IT must convince those hoarding data on desktop systems that it is in their interest to share their data. The cloud has to do something for them, she says.

Despite the challenges, enterprise IT managers can expect to see more data warehouse deployments in the cloud. In April, IDC found the market for analytics tools and EDW software hosted on the public cloud would grow by 32% annually to represent more than 44% of the total market in 2022. These organisations will have plenty to learn from J&J's data warehouse journey.

Six lessons from Johnson & Johnson's EDW cloud migration

• Identify all the stakeholders involved and begin discussions to identify potential challenges
• Start with a small proof of concept to test all aspects of the potential solution
• Understand as early as possible the network bandwidth and latency between your on-premise and cloud solutions
• Expect some things to go wrong the first time you try them
• Engage a strong project manager, who is good with timelines and risk, to be the single point of contact for communicating progress
• Practise processes over and over again, including failure scenarios

With so much data generated daily that needs to be stored, processed and analysed, traditional data warehouses have struggled to keep up. Waiting days for insights is no longer feasible - especially when your competitors aren’t.

This whitepaper explores how your organisation can easily and affordably harness massive amounts of data and transform it into valuable insights.

Download it now for an overview of the data warehouse, including a comparison of on-premise to cloud-based systems, case studies and steps to get started.

Databases are the cornerstone of any business; they're used to track purchase orders, catalogue customers and manage employee payrolls. On top of that, basically every web technology is built on a back-end database.

The most common form of database within IT is the relational database. These are coded using SQL (pronounced 'sequel') which stands for Structured Query Language. These databases require specialised software to manage, examples of which include Oracle Database, Microsoft's SQL Server family, and PostgreSQL.

Relational databases were first developed in order to standardise the way that databases were constructed and maintained. Prior to this, the way databases were structured and navigated often differed from one to the other, making them hard to develop applications for without being deeply familiar with the database in question.

In order to make this task - and the general process of working with databases - more efficient, relational models were developed to provide a universal, standardised format for databases. These were easier to maintain, more versatile and quicker to learn how to work with.

How do relational databases work?

Relational databases are comprised of multiple interconnected tables which are linked by a shared value. These shared values are identified by 'keys' - the column or columns within a table which contain values which are shared between multiple tables. These keys help the database understand which row of values to pull from each table when handling queries. The primary key is usually the first column within a table, although a table can contain multiple subsidiary keys, or 'composite keys' comprised of multiple combined columns.

Let's look at an example, from a fictional restaurant delivery company that wants to offer UK customers who have been premium subscribers for over a year a voucher for their favourite food as a reward. Their database contains two relevant tables - one with customers' personal details, and one with their subscriber information, as shown below.

Personal details

Subscription details

In order to establish which customers are eligible and to send them the offer, the company needs to establish their name, email address, country, membership tier, the length of their membership and their favourite food. They can do this by using the database's key - which in this case is the email address - to cross-reference the two tables with one another.

This allows the database to establish that, because he has been a premium member for a year and a half and lives in the UK, John Smith is eligible for the reward, as well as allowing it to automatically send a personalised email to him containing a voucher for his favourite burger restaurant.

Relational database schemas

Relational databases are organised according to specific schemas - predefined rules governing how tables are laid out, including what data is contained in each column and the order they come in, as well as which columns are used as keys.

Schemas can be coded in SQL or laid out in charts and graphics. The schema's specifics will differ from database to database, but the format is standardised enough that it's easy to go from working with one schema to another, making them simpler to develop applications for.

Advantages of relational databases

The primary advantage of relational databases over earlier non-standardised methods of database architecture is the high degree of consistency they offer. Relational databases are both internally consistent - ensuring that data is easy to find, because it's formatted and laid out in a uniform way - as well as being much more interoperable with each other due to their use of pre-set schemas.

Relational databases are also rules-based, meaning that you can manage and edit large databases by creating global parameters and applying them to every record and table within a database. Additionally, because the databases themselves are independent of the physical storage on which they reside, database architects can alter the infrastructure the database runs on without affecting its logical integrity.

These databases also have certain advantages over newer database architectures such as NoSQL (which we'll discuss further in a moment). Relational databases tend to be more stable and consistent than their non-relational counterparts, and they're often better-suited to complex queries. The relational model also benefits from a much greater wealth of support resources, information and skilled professionals, due to its longer history.

Relational database alternatives

In recent years, the dominance of the relational model has been challenged by the emergence of non-relational 'NoSQL' databases, typified by providers like Couchbase and MongoDB. NoSQL databases (predictably) do not use SQL, and are instead built using object-oriented programming languages like Python, Ruby, Java and C++.

These databases are predominantly built to handle massive scale, and have found particular popularity as a tool for big data storage and analysis. The main advantages NoSQL offers over relational databases are that NoSQL databases can be expanded horizontally by adding more nodes of commodity hardware, and that they are designed to allow unstructured data, whereas relational databases demand rigid formatting.

Big data represents a shift in the enterprise technology landscape. As the volume and variety of data continues, companies are increasingly turning to Hadoop, NoSQL and other tools to tackle information issues not readily addressable with older data technologies.

This whitepaper outlines the challenges many companies face with big data, and sets out a blueprint for succeeding with these scenarios:

• Data warehouse optimisation
• Streamlined data refinery
• Monetising data
• Getting a 360-degree customer view

Download it now to learn more.