Amazon Web Services (AWS) has announced a series of changes to Amazon OpenSearch aimed at streamlining data log analysis.

A new purpose-built log analytics engine for the service will provide users with marked performance gains and up to 70% lower storage costs, according to the firm.

OpenSearch allows users to search, analyze, and visualize data in real-time, and is typically used for application monitoring, observability, and AI/ML search applications.

The hyperscaler said the update comes in direct response to surging volumes of log data handled by enterprises in 2026, largely due to AI.

Research from AWS shows that log volumes have grown between 30-40% year over year, which is stretching budgets and creating bottlenecks.

Amazon OpenSearch overhaul

As part of the update, Amazon revealed OpenSearch will now store data in the Apache Parquet format. This is a column-oriented data file format designed specifically for more efficient data storage and retrieval purposes.

The service previously relied on inverted indexing. Amazon noted the shift will deliver significant efficiency gains and faster search and retrieval capabilities.

“Apache Calcite parses and optimizes each query, then routes operations to the engine best suited to execute them: Apache DataFusion for analytical operations on columnar data, or Lucene for search predicates,” the company explained in a blog post.

“The two hand off mid-query, so a single query can search log content and aggregate the results without additional roundtrips.”

Amazon noted that OpenSearch now also supports the Piped Processing Language (PPL) and SQL query languages, both of which “execute natively” through the platform.

Amazon eyes optimized log analytics

From a performance perspective, Amazon noted users can expect significant improvements alongside lower costs.

OpenSearch now boasts up to four-times price improvements compared to the existing engine, based on internal benchmarks.

Users can also capitalize on two-times faster analytics queries due to the fact the engine processes data in columnar batches, even across larger data sets. A two-fold improvement on ingestion rates also aims to reduce bottlenecks.

“The optimized engine sustained 1.78 million documents per second at matched concurrency, approximately 2x the throughput of the Lucene baseline, while consuming less CPU,” the hyperscaler explained.

“Both domains ran with zero write rejections. For teams ingesting terabytes per day, the throughput advantage translates to fewer nodes for the same volume, or longer retention on the same infrastructure.”

How to access Amazon OpenSearch

The new log analytics engine is available now for enterprises in AWS regions where OpenSearch Optimized instances are supported, the company noted. Similarly, there will be no additional premium for the optimized engine.

It’s worth noting that the new engine is a setting selected at creation time, meaning users can’t add this to an existing domain or enable it within individual indices or fields within a general-purpose domain.

“To adopt the optimized engine, create a new domain and migrate your ingestion pipelines into it,” the company explained.

“Create a new domain in the Amazon OpenSearch Service console and select Observability as your use case. The optimized engine is enabled by default.”

The in-built console will provide users with a side-by-side comparison of capabilities based on individual needs.

“After your domain is ready, ingest JSON documents through the same Bulk API and client libraries you use today. No changes to your ingestion pipelines or application code are required.”

FOLLOW US ON SOCIAL MEDIA

SAS has announced new synthetic data capabilities and support for digital twins in a push to equip enterprises with concrete platforms for data-driven decision making.

Synthetic data is artificial data generated to resemble real datasets in every way that’s meaningful. These datasets are typically used to train AI and machine learning models without exposing sensitive data.

While generating synthetic data has benefits, obtaining it can be difficult, research shows. Organizations need access to the right algorithms to generate quality data, particularly if they’re looking to train a domain-specific AI model for very targeted use cases.

This is where SAS Data Maker is intended to come in. The service is a low-code/no-code platform that can quickly generate synthetic data in the quantities needed and to a high standard that’s auditable.

SAS first announced Data Maker in 2024 as an in-preview product with an unspecified release date, and has since acquired the UK synthetic data startup Hazy to improve its data generation software. It has now announced a targeted general release of Q3 2025.

The announcement was made at SAS Innovate 2025, the company’s annual conference held in Orlando, Florida. At the same event, SAS elaborated on work it’s doing at the other end of the data pipeline to make better use of existing business data for advanced data simulations, such as digital twins.

It gave the example of a recent project it has undertaken with the paper and wood-based product manufacturer Georgia-Pacific (GP), in collaboration with Epic Games.

The trio have leveraged Epic Games’ 3D Unreal Engine, originally developed for video games, combined with GP's enterprise information and SAS data technology, to simulate an entire factory at Savannah River Mill in Georgia.

Using the factory’s digital twin, GP could analyze the efficiency of its on-site automated guided vehicles (AGVs), as well as lower the barrier to entry for workers seeking analytics data on the factory.

SAS said its cloud-native Viya platform was integral in getting the project up and running, with GP having used tit to translate unstructured factory data into informations systems that formed the backbone of the digital twin.

“With the help of SAS and Unreal Engine, we can create realistic simulations of factory operations,” said Roshan Shah, GP’s vice president of AI & Products.

“Imagine watching AGVs navigate through a bustling factory floor, reacting to proximity alerts, obstacles and rare adverse events in real time," Shah added.

"The powerful analysis and photorealistic simulations delivered by SAS’ enhanced digital twins can enable decision making and boost output at Savannah River Mill. And they hold great promise for improving productivity, safety and efficiency at other GP facilities.”

MORE FROM ITPRO

• What to watch out for at SAS Innovate 2025
• Why AI researchers are turning to nature for inspiration
• Are AI digital twins a match made in heaven?

Data-driven decision-making can help a business predict how a market will change, what demands customers will place on their vendors and suppliers, and how the broader economic climate could impact each business decision.

Every business in the channel can collect vast quantities of data about its customers and the wider markets they operate within. Performing data analysis on this information for tangible and actionable data-driven decision-making should be a core focus for channel businesses looking to achieve sustainable transformation and capture profit over the long term.

While some channel businesses already take advantage of this opportunity, many do not. Ben Nicklen, CEO of workplace analytics firm Tiger, tells ITPro that the channel is not yet fully utilizing the kind of big data analytics necessary to predict future customer trends and requirements.

“Regrettably, it seems that they may not be allocating sufficient resources towards adopting future technologies, as their primary focus often appears to be maximizing profits from existing strategies.”

No business is immune from the economic turmoil we have seen worldwide in recent years. Stubbornly high inflation and interest rates on an upward trajectory have meant businesses are re-evaluating their spending and what this means for future purchases. The channel ecosystem can adapt to shocks in the tech sector while individual channel partners can help maximize growth and deliver cost savings. But what the channel must be actively pursuing is strategies that predict the shape, size, and needs of their markets over the short to medium term.

Building channel resilience is now a top priority and the basis of that resilience is data. In conversation with ITPro Rob Mackle, director and co-founder at Assured Data Protection, points to the essential customer data the channel must be collecting and analyzing.

“Data plays a vital role in not only determining what vendors and software developers build, but what and how service providers and partners go to market,” Mackle tells ITPro.

“Economic factors drive business and channel decisions. It is all about executing today and where the customers will be in the future. For example, businesses are typically driven to cloud due to the concept of cost savings and economic pressures. In reality, is it cheaper, is it as secure oras flexible as on-prem? Data will be a good indication of how the cloud has delivered today and what does the future hold.”

Samantha Sene, channel leader at A10 Networks, outlines its approach to using the data they collect. "At A10 Networks, we send regular surveys to our customers to see how our services and products are benchmarked and what additional product features our clients want. 

“It’s also key to offer flexible pricing models. If the customer grows their business, channel suppliers must be flexible enough to offer adapted pricing structures to ensure it suits their customers’ growth models.”

Big data is big business for the channel

Nearly 60% of businesses report they are driving their businesses with data, according to a report [PDF] by NewVantage Partners which took in responses from executives at 116 leading organizations. However, less than a quarter (20%) stated they have nurtured a data culture within their organizations. This worries channel companies, who need to leverage their data to become more relevant to their customers and market.

Richard Farrell, CIO of Netcall, sees data-driven decision-making as essential for any business in the channel, arguing that the channel is increasingly reliant on the process to anticipate customer needs.

“By analyzing historical sales data, market research, and customer behavior, channel partners can identify emerging technology demands and tailor their offerings accordingly. Predictive analytics can help anticipate shifts in customer preferences and buying patterns, allowing vendors to proactively develop products and services that align with future demands.”

Farrell adds that expertise is necessary to make this happen in the most effective way. He suggested that businesses could benefit from working with smaller vendors for closer collaboration.

“These vendors are more agile and responsive to channel partner feedback, making it easier to co-create solutions that precisely meet customer needs. Smaller vendors often prioritize relationship-building, fostering a deeper understanding of the channel's requirements.”

The value of data-driven decision-making 

The managed services market will increase in value by 13.6% by 2030 according to Grand View Research Its current value is $267 billion. This significant expansion should be food for thought as the channel continues its journey to deliver more integrated services to companies in the future.

For Seagate, data is at the core of their business. Andrew Palmer, UK channel lead for Enterprise Data and Cloud Solutions, explained to ITPro how Seagate uses artificial intelligence (AI) to help track its sales history and manage its inventory. 

“Using AI-powered analytics, we can determine customer sales trends based on their product requirements. AI has helped drive predictions for us regarding building stock and committing to a manufacturing cycle. It's not just the frequency the customer buys, but the types of products."

Palmer also emphasized the power of historical data for driving decision-making. “Information, like when sales peak or dip, can highlight which strategies are the most and least effective. Companies can then use this data to create more effective promotions, following seasonal and long-term trends to adapt to changing markets.”

The channel must become more proactive. In the past, it was often guilty of following market trends shaped by large players such as the hyperscalers. To become truly innovative and essential business partners, the channel must shape its own future by forming strategic plans through data-driven decision-making to speak directly to its customer needs.

“Data is the foundation for differentiation and success in the IT channel,” Farrell adds.

“By harnessing data effectively, UK LCAP and contact center as a service (CCaaS) vendors can stay agile, responsive to market changes, and customer-focused, setting themselves apart from larger US mega vendors. Partnering with a smaller UK-based vendor offers channel partners a unique advantage due to their agility, localized expertise, and commitment to personalized solutions. 

“By leveraging data in collaboration with these vendors, the channel can better address the specific needs and dynamics of the UK market, creating a win-win scenario for both parties.”

Understanding the market environment channel customers are working within is critical if the channel is to remain relevant and have the required products and services available on demand. Here, data is the key. A more detailed definition of each channel customer, their market sector, the pressures they are under today, and how these might evolve into the future, is how the channel becomes an essential partner and not simply a supplier. Data can deliver that level of intimacy and insight.

Copenhagen Airport is colossal. Spread across multiple terminals and passenger concourses, covering an area of 4.6 square miles, the airport employs 1,700 people and there can be up to 83 operations (take-offs and landings) every hour. Some 30 million people travel through the airport each year and this is expected to increase in the coming years. To meet the growing passenger throughput, Copenhagen Airport needed to expand.

Physical expansion isn’t always possible, as airports are constrained by the physical limitations of their site and planning regulations. But space isn’t always used as effectively as it could be. 

And if an airport were to operate more efficiently, it could increase this capacity. For Copenhagen Airport, the answer lay in unlocking the power of big data.

Joining disparate systems under the AIRHART project

Airports are information-dense environments, but data is spread across multiple systems and often in incompatible formats, making it difficult to gain a complete overview of what’s happening. The result is inefficiency and conflicting priorities being introduced.

“Imagine buying a car, and the only tires you could add to that car are from a specific manufacturer,” says Mehdi Motaghiani, CEO of Smarter Airports, a joint venture between Copenhagen Airport and Netcompany.

“We wanted to be able to add in and attach multiple systems and data to one platform that would help orchestrate those processes and create one source of truth that everyone could rely on for decision-making,” he adds.

For ten years, Copenhagen Airport examined how it could combine previously incompatible digital systems into a single platform, enabling total airport management. The solutions they were initially presented with, however, didn’t fully meet expectations, so they looked into creating their own.

The AIRHART project developed a digital platform unifying all information streams. This offered improved airport traffic management, for example enabling them to better predict passenger flows and the associated baggage handling requirements. But this was no easy task. Managing multiple data streams is challenging, especially when they’re generated by specialist software.

Why it was hard to define the project scope

Airports are constantly evolving. As such, the development team was unable to lock the project scope during its three-year development. A level of flexibility was maintained in order to be able to adapt to meet shifting demands.

“You cannot lock the scope, otherwise you would need to do all your thinking up-front; but that cannot be done, because you don’t actually know the details of what you want to achieve,” says Motaghiani. “You need to ensure that you can keep being innovative, while still having control over timelines, systems, and the stakeholder landscape.”

Airports also necessitate high-level security, which Netcompany ensured by developing AIRHART with a methodology based on security by design; they used continuous testing, authentication safeguards, and good programming practices to make AIRHART as impervious to attack as possible. AIRHART’s security was tested prior to launch by penetration testing teams.

The team then rolled out AIRHART with minimum levels of disruption thanks to the ‘shadow production’ distribution model. The existing platform and the new AIRHART were running simultaneously across the networks; by not enforcing full adoption, Copenhagen Airport enabled stakeholders to transition to the AIRHART system at their own pace. 

How the project met regulatory requirements

Airport regulators are understandably risk averse, so one AIRHART’s key challenges was meeting the strict regulatory requirements necessitated by governing bodies. A particular challenge was they were developing AIRHART according to new regulations – which hadn’t been previously mandated.

“There are a set of laws around air traffic management systems, which are new, as the systems put into production after a given date need to comply with this new set of standards and regulations. As there had not been anyone going through this process, we needed to understand that and make it tangible: what did it actually mean when it comes down to software testing and security?”

With hindsight, they'd have liked to be more involved with regulators. Greater collaboration would have let them create a platform with compliance embedded at its core. “What we tried to do was not only develop a platform but to say ‘This is what we think should be a standard’,” explains Motaghiani.

The next stage for AIRHART will be using machine learning to predict events and recommend actions. Utilizing machine learning in this way could help Copenhagen Airport further minimize disruption and improve efficiency. There are also plans in the works to offer the core functionality of AIRHART to other airports and potentially similarly complex sites, such as hospitals.

Liverpool Football Club (Liverpool FC) is one of the world’s most renowned soccer institutions, with a massive global fan base and a rich history of success. As a prominent brand in the sports industry, the club has recognized the importance of digital transformation and embarked on a journey to improve its operations and fan experiences.

One of the key challenges Liverpool FC faced was managing its large digital media workflows and handling vast volumes of content. The club generates and distributes a substantial amount of digital content across various platforms. This includes live match broadcasts, highlights, interviews, behind-the-scenes footage, and promotional materials. The sheer magnitude of Liverpool FC’s digital media workflows posed significant challenges in terms of content creation, distribution, storage, and accessibility.

Understanding the need for change

Traditional on-premise infrastructure Liverpool FC has traditionally relied on proved to be inadequate for efficiently managing the club’s digital media workflows, says Drew Crisp, its senior vice president of digital.

“Initially, we would have done this anyway because all of our media archives are on-premise and were on traditional 24 spinning disks, and actually accessible [only] when you're in the office,” he tells ITPro. It was always the plan to get data into the cloud, Crisp adds, as it would be “faster, quicker, easier to access”. 

Liverpool FC faced numerous challenges with its infrastructure, including slow data access, limited remote access capabilities, performance bottlenecks, and the physical limitations of storing and processing large volumes of content. He adds COVID-19 accelerated plans. “It brought forward a bunch of stuff because hybrid working doesn't lend itself particularly well to accessing archive when it's on a bunch of hardware that's in the office.”

Recognizing the importance of data as a critical asset, Liverpool FC crafted a robust data management strategy to support its operations. Crisp stresses how important this was, adding: "We needed a capability that is remotely accessible, really quick and really easy to use, but also which could enable us to do future things – not just with archived content, but post-production content we could share with partners."

This would also improve the decision-making processes. By harnessing the power of data analytics, Liverpool FC could gain valuable insights into fan behavior, preferences, and engagement patterns. These insights would guide its marketing efforts, fan engagement initiatives, and overall business strategies. A data-driven approach would help Liverpool FC understand its fans better, tailor its content offerings and deliver personalized experiences to its global fan base.

Scoping out a cloud partner

To overcome these challenges and unlock new opportunities for growth, efficiency, and fan engagement, Liverpool FC embraced the cloud by partnering with Wasabi as its storage provider. The club revolutionized its digital workflows through this agreement and maximized the potential of its vast media library.

“I mean, the biggest advantage, as we see with wasabi cloud is we don't have to pay for egress. From a commercial model perspective, as well as speed and efficiency, actually, it does an awful lot for us. Because you don't pay commercially, you can ingest really quickly and then you can get into operation,” says Crisp.

Neither the club nor Wasabi encountered major challenges, and the migration's success means the club is exploring how else it can use the cloud in the future. Ultimately, Liverpool FC’s decision to move to the cloud has provided a solid foundation for future improvements to its end-to-end content workflow.

Wasabi’s cloud storage has allowed Liverpool FC to store and access its media content with ease and speed, eliminating the performance issues and physical limitations of its previous on-premise infrastructure. It's also provided Liverpool FC with scalability and flexibility, allowing it to adjust its storage capacity according to changing needs and demands.

This has translated into measured improvements. Liverpool FC can now share and access data across various teams and departments within the club, which facilitates better collaboration and communication. It’s also streamlined workflows, reducing the time and resources required for content creation, distribution and management.

Crisp says the organization’s editors are “probably three times more efficient in its eight-hour day than they would have been previously waiting for all sorts of things to upload, download and edit on the fly”.

“From an efficiency and time point of view, that means that they can get through more content and create more content significantly faster,” he continues. “I'd say probably, we probably do 20-30% more content now than we did previously.”

Splunk has launched new AI tools for its security and observability platform as well as its new Edge Hub for more streamlined edge data analytics. 

The Splunk AI Assistant - now in preview - is designed to assist users with the authoring of Splunk Processing Language (SPL) queries.  

Generative AI is leveraged via an AI chatbot to write or explain SPL, something the company says will improve time to value and make SPL more accessible. 

Mark Woods, chief technical advisor EMEA at Splunk, told ITPro that training for the model behind the generative AI was based on the company’s own knowledgebase and that the limited release preview had been based on the company’s own internal experts. 

Woods’ words stand to reassure customers concerned that sensitive data might turn up in the AI. 

He said: “We wouldn’t have any visibility of the actual query that you’re putting in place, nor of any of the data that you are taking”. 

As well as the Generative AI tooling, Splunk also previewed ML-Assisted Thresholding to use historical data and patterns to create dynamic thresholds.  

Its IT Service Intelligence also features greater detection accuracy, with the Outlier Exclusion for Adaptive Thresholding aimed at detecting and omitting abnormal data points and outliers. 

A preview of the OpenTelemetry Collector was also announced, aimed at providing a unified view of an enterprise’s infrastructure and services.  

Splunk Edge Hub 

The new Edge Hub device was also shown off and is aimed at simplifying the ingestion of data into Splunk’s analytics platform. 

The compact device weighs in at just over 600g and designed to be mounted in a variety of customer environments. Splunk envisages customers simply plugging it in and using the device to stream environmental data back for analysis. 

Onboard sensors comprise temperature, humidity, light, and sound. The device also features an accelerometer and gyroscope, and it can monitor air pressure and air quality. Typical applications of the latter sensor - according to its manufacturer, Bosch - include spotting unusual gases - indicating a possible leak - and spoiled food detection. 

The device also features a screen with a dashboard showing the current sensor readings and can be connected via WiFi, Bluetooth, LTE, or Ethernet. Power is supplied by either USB-C or Power-Over-Ethernet. 

The Cortex-based processor within the device - a Verdin IMX8M+ Quad with 8GB of RAM - has sufficient power for collection and collation of data from the sensor suite. 

Woods said to ITPro that he expected the device would open customers’ eyes to the data in their OT and IoT environments: “They suddenly start to see the data that they already have all around them”. 

“All those functional domain-specific environments, they become really tangible and valuable to customers.” 

Who is Splunk’s Edge Hub for? 

Woods identified three main user groups for the device. Compared to fitting out an environment with sensors, Splunk feels the Edge Hub represents a relatively low barrier to entry, so the first user group targeted is the one that has yet to use sensors and the data does not exist. 

The second group, according to Woods, might have some sensors installed, but the systems are closed and Splunk’s device represents a useful bridge. 

Finally, there is the group that is already equipped with sensors in their environment - (Woods cited Mars as an example - and the Edge Hub provides additional resilience or a way to experiment without adding a toll to an operation system. 

Splunk said that data from the device could be used as part of its predictive analytics platform to spot anomalies in manufacturing processes as well as detect, investigate and remediate physical threats from a single platform. 

The Splunk Edge Hub is US-only for now - although the company plans to roll it out in EMEA and APAC. It is also only available through the company’s partners. 

Knowledge workers are active across many industry sectors – anywhere, in fact, where a person needs to use their intellect and judgment to synthesize information and make decisions. Artificial intelligence (AI) is proving to be a help to knowledge workers in numerous ways, across a wide range of sectors. 

It might read as rather stating the obvious, but knowledge workers need insights to work with. This only exists when raw data is put through some sort of filter, but processing data can take a human much longer than it can take a machine, even when they’re working with coherent data sets. When more data sets are added in, with variables of differing nature, perhaps also with weighting, the human brain can become overwhelmed.

Not only are computers faster at working with big data, but they also don’t make mistakes due to oversight or fatigue, and can keep going 24 hours a day. With data analytics, machines can identify trends, spot anomalies, and understand nuances – and bring to our attention things we might easily dismiss or overlook. They can flag up the expected, the unexpected, and the unusual. 

The key benefit of industrialized AI is it helps knowledge workers spend more time acting on knowledge, or “insights”, and less on actually discovering that knowledge in the first place. That way, people can optimize their use of time. We look at four very different industry sectors and show how AI helps knowledge workers in them do a better job.

Medical diagnosis

Highly trained specialists need to analyze increasing volumes of data to help diagnose more and more patients with potential illnesses. Indeed, they have a data mountain to climb. The last thing medical specialist want to do is miss out on opportunities to help people as they sift these mountains of data.

In many areas of diagnostic medicine, AI is taking on the heavy lifting. It can do this by reviewing large amounts of data at speed, alerting professionals to anomalies and to specific characteristics it has been asked to identify. 

Moorfields Eye Hospital

At Moorfields, a team has developed a way to use AI to assess the severity of a form of age-related macular degeneration (AMD), as well as how it’s progressing and responding to treatment. Manually doing this work can take up to an hour per eye, and is prone to human error. The AI can match, and even outperform, predictions made by specialists in a fraction of a second. This is one of many ways Moorfields Eye Charity is researching the use of AI. 

Personalized Education 

Education is about the giving and receiving of knowledge. But individuals don’t all learn in the same way or at the same speed, and some of us need more support or nudging than others. At the higher levels, in particular, people can want to be more free to learn at their own pace. Educational institutions need to work differently to support and help them.

AI can be used to deliver educational content at different paces, and in different forms. Students can repeat learning exercises in different ways to help them fully understand concepts or complete learning modules. AI can help identify students who might need support, and provide useful nudges. 

Georgia State University

Georgia State University has been using an AI-enhanced messaging tool called Pounce since 2016. In 2021 the chatbot was integrated into course content, including so it could issue reminders to complete work by deadlines, and to take practice tests. Students getting this help turned in better grades. Using AI in this way helps teachers give their students the best support and assistance.

Mining and exploration

Mining involves complex processes and data analytics to identify the best localities in which to work. It also requires the use of precision equipment, which needs to be kept at peak performance and must maintain the highest possible safety and compliance standards.

Predictive maintenance can be used to help diagnose equipment and ensure parts are replaced before they break and call work to a halt. AI can also be used to analyze different data sets and suggest the most likely locations to mine. The technology might also be able to analyze the environmental conditions inside mines to support compliance – and play a role in maximizing yields by sorting the materials mined.

Streamlining mining logistics

A multi-national resources company, which manages freight of more than $1 billion annually for shipping across the world, wanted to enhance logistics with analytics capabilities. They tapped into Infosys Nia. This is an AI platform that helps companies manage their logistics – and combines data in numerous formats to help improve the success rate of discovery and exploration. It can be used to provide humans such as geologists and engineers with detailed information they can analyze further to help do their own jobs more efficiently.

City planning 

Cities and towns are complex, and people need to be able to move around. The right facilities – housing, work, leisure – need to be in the right places, too. We also are becoming increasingly aware of the need to reduce car travel and pollution and to increase access to open spaces and entertainment locally. Urban planners have their work cut out comparing variables and finding the solutions that work best.

AI-based tools can be used to model different scenarios for any particular area plan, taking into account environmental factors like heat, wind, or rainfall, and making the tweaks necessary to minimize negative environmental effects. In major cities, AI can analyze data on real places to support further development, such as transport planning, and provide analytics humans can use to better allocate resources. 

Transport for Wales

Transport for Wales (TfW) has worked with a third-party AI specialist, Optibus, to analyze bus use, demand for public transport, and movement patterns. Analyzing this data has allowed TfW to improve bus schedules, punctuality, and journey times.  

Using the Optibus software, TfW worked with all 22 local authorities to design a better public transport network with the aim of improving the quality of service and making buses more reliable, especially in rural areas.

Delta Lake is an open source framework designed by Databricks that allows businesses to manage and analyze the data they generate in real-time. 

Taking the form of a transactional storage layer, it provides a foundation for storing data and tables on Data Lakehouses – which Databricks refers to as an open data management architecture. This allows for data warehousing and machine learning operations to be applied directly onto a data lake.

It’s the default storage format for all Databricks operations and is an open protocol, with the organization completely open sourcing Delta Lake with the launch of Delta Lake 2.0 in June 2022. 

With Delta Lake 3.0, launched at Databricks Data + AI Summit 2023, Databricks has launched a new unified file format – UniForm – that solves a longstanding compatibility issue in the way businesses store metadata in their data lakes. This launches alongside a bid to remove data silos by addressing connector fragmentation, and performance improvements.

What is Delta Lake?

Introduced as an open source project in 2019, Delta Lake was created to ensure the data that powers real-time insights that organizations crave is genuinely reliable. It was billed at the time as Databricks’ most important innovation, even including the Apache Spark engine for data processing.

The need for Delta Lake has emerged from the way businesses have traditionally managed data in massive and unrefined data lakes. Because these repositories contain both structured and unstructured data, at different scales, any operations running on top of this data – such as data analytics or machine learning – might not be optimized.

Databricks designed Delta Lake to transform these messy data lakes into cleaner ‘Delta Lakes’ with higher-quality data. This ultimately ensures any additional processes or operations performed on top generate much better insights, even if performed in real-time.

“Nearly every company has a data lake they are trying to gain insights from, but data lakes have proven to lack data reliability,” said Databricks cofounder and CEO, Ali Ghodsi. “Delta Lake has eliminated these challenges for hundreds of enterprises. By making Delta Lake open source, developers will be able to easily build reliable data lakes and turn them into ‘Delta Lakes’.”

Delta Lake competes with a number of other services on the market that perform similar functions, including Apache HUDI, Azure Data Lake, and Snowflake.

How does Delta Lake work?

Based in the cloud, Delta Lake is a Spark proprietary extension that allows organizations to create Delta tables by default, whether they’re using Apache Spark or SQL. The open source platform extends the classic parquet data file format with a file-based transaction log, which allows for additional functions. 

Delta Lake features include providing atomicity, consistency, isolation and durability (ACID) transactions on Spark, as well as scalable metadata handling. The former ensures there’s never inconsistent data, while the latter takes advantage of Spark’s distributed processing power to handle masses of metadata at once.

Organizations can also take advantage of streaming and batch unification – by allowing both types of data to land in the same sink. This comes alongside some quality-of-life features such as rollbacks and historical audit trails, as well as the capability to reproduce machine learning experiments.

It comes with a variety of open source connectors, including Apache Flink, Presto and Trino, alongside standalone readers/writers that lets clients for Python, Ruby and Rust write directly into Delta Lake without needing a massive data engine like Apache Spark.

In 2021, Databricks launched Delta Sharing, a protocol that lets different companies share massive data sets securely and in real time. This aims to break the notion of vendor lock-in, as well as break down data silos due to proprietary data formats and computing resources that are required to read data. Users can access shared data through Pandas, Tableau, Presto, and other platforms without the need to use proprietary systems.

What’s new in Delta Lake 3.0?

With the latest iteration of Delta Lake, Databricks is hoping to end the ‘format wars’ that’s long posed challenges for storing and managing data in data lakes. 

Most data (99%) is stored in data lakes using the parquet format, but organizations must choose from one of several competing standards to store all metadata: Delta Lake, Apache Iceberg, and Apache Hudi. Each format handles metadata differently are fundamentally incompatible with one another. Once enterprises choose a format, they’re stuck using it for everything.

With Delta Lake 3.0, Databricks has created the universal format (UniForm), which serves as a unification of the three standards. Customers who use a Databricks data lake will find their data stored in the parquet format, as usual, alongside three different versions of the metadata. It means the Delta Lake data can be read as if it’s stored in either Iceberg or Hudi.

“The metadata is very important, and if you get the metadata wrong, you can’t actually access this stuff,” says Databricks CEO Ali Ghodsi, speaking at Databricks AI + Data Summit 2023. “Since all three projects are open source, we just went and understood exactly how to do it in each of them, and now inside Databricks, when we create data, we create data for all three.”

Also new in the latest iteration of Delta Lake is the Delta Kernel, which addresses connector fragmentation by ensuring connectors are built against a core library that implements Delta specifications. This alleviates the burden of having to update Delta connectors with each new version or protocol change. With one API, developers can keep connectors up-to-date and ensure the latest innovations are pushed out as soon as they’re ready.

The final new addition is the Delta Liquid Clustering, which boosts the performance of reads and writes. With a flexible data layout technique, the Delta Liquid Clustering marks a departure from the decades old hive-style table partitioning system, which uses a fixed layout. It means organizations can take advantage of cost-efficient data clustering as their data lake grows with time.

Virgin Media O2 has saved “millions of pounds” in costs by enhancing data center efficiency through the introduction of flash storage drives, according to a senior company official. 

Ajit Sharma, business optimization manager at the telecoms giant, told attendees at Pure Accelerate 2023 that the integration of the devices in data center operations followed a deal with Pure Storage to adopt its FlashArray//X drives. 

This move, Sharma said, delivered significant operational and financial benefits, including a 98% reduction in its data center footprint. 

The integration of flash drives also resulted in a 96% reduction in power consumption, Sharma said while hailing the improvements made to the company’s data architecture. This has enabled the firm to reduce the impact of rising energy costs amid the cost-of-living crisis. 

“In the UK we’ve seen our power costs increase by nearly five to ten times depending on who you’re co-locating with in the data center space,” he explained.

“We don’t have our own property, but most of our own data centers are co-located with Equinix and other companies. Unfortunately, we’re at their mercy due to what’s going on across the sector as well as what’s going on across the world as well.”

A survey of data center professionals from Aggreko last year found that 60% reported energy bill increases of up to 40%. 

Given the rise in energy costs over the last year, Sharma said that by reducing power consumption, the company has experienced a “huge reduction in costs”. 

Maximizing data center space

As part of the transformation program, Virgin Media O2 reduced its storage racks from 42 to three, and runs its storage on a select number of flash arrays which are able to handle over four petabytes of data. 

A key factor in the firm’s ability to reduce its footprint lies in the compact nature of flash storage drives compared to traditional storage devices, such as hard disks, Sharma said. 

Reducing this footprint represents a long-term strategy and was more than about delivering immediate cost savings. Looking ahead, this will cut the need to invest in additional data centers across the country. 

“The UK is a much smaller country in terms of size,” he said. “So by being able to re-use and improve sustainability, we don’t need to go and build new data centers.”

“So we’re helping the UK and the community to ensure that we’re not using more space than is required. We’re saving millions of pounds by not building new data centers.”

Re-investing cost savings

The cost savings from this data center transformation strategy means that Virgin Media O2 has also been able to reduce e-waste levels and re-invest in other areas of the business, creating a positive cyclical process of optimization, cost savings, and re-investment. 

This has been especially critical for the firm as it prepares for the eventual roll-out of 6G technologies, Sharma added. 

“In the last year alone we got rid of over 48,000 tons of e-waste,” he said. “What we’re seeing from the money that’s being saved in e-waste and the ability to reduce our data center costs, is that it’s been refunded into other parts of the business.”

“We’ve seen 4G and 5G, and 6G is going to be out soon as well. So we need to invest our money in the right places - this helps us on that journey.”

We all know data is the new currency, and the applications for data analytics are seemingly endless. Industries across the economy have been keen to adopt big data and analytics, but have done so to varying degrees of success. 

Football is an area in which data is everywhere. Indeed, the use of data analytics is rife – whether it’s quantifying different phases of play throughout a match or measuring player performance, or even identifying transfer targets.

There are countless data points available to coaches, presenters, commentators, pundits, and executives throughout the industry, who comb through the numbers and translate them into insights. 

But there are metrics off the pitch the football industry has less success in translating to real insights. Seeking to capitalize on the opportunities of on-pitch and off-pitch data is LaLiga, the premier division in Spain.

Tapping into rivers of data

To make the most of the untapped rivers of data flowing through football, the league created LaLiga Tech as a subsidiary that strives to capture and process the data generated throughout each facet of the annual competition, with the organization providing detailed analytics to the groups that make up the league.

“We know now that data is power – but if data is isolated, and you have silos of data that are unconnected, then you are not able to provide value for the clubs, or the fans,” says Fermin Martinez, senior data engineer for business intelligence and analytics at La Liga Tech. 

“But it's not only about having the data, you have to be able to join all the data together. In order to get real insights, you need to put a lot of different sources together and mix them.

“The real motivation is to be able to give value to clubs and fans. In order to do something with this data, we need a platform. We need the data lake we are using from Databricks, but we also need to apply data processing techniques.”

Collaborating with data off the pitch

Before the inception of LaLiga Tech five years ago, clubs were processing their own data and creating analysis in isolation. While this was useful for each club in isolation, it couldn’t provide insights that a larger data model would be capable of, Martinez says.

With 20 clubs all providing up to 25 data points a second throughout matches, this volume of data needed a place to be stored and processed. In the case of LaLiga Tech, this came in the form of Databrick’s data lake where, as Martinez puts it, the data originating from different sources can be “mixed” and insights can be generated.

“Before we started with Databricks, most departments inside LaLiga worked with different providers; these providers were the only ones able to access the data. As a result, the data was there but there was no connection between one source and the other because there was no connection between teams.

“We talk about data and how data is important. It's obvious that we need to get all this data together somewhere. The data lake is the place where all the data goes. Using Databricks, we are able to separate the data we receive into several states. We have a staging area inside this data lake, a prepared area where we start to prepare the data, and we have a final consumer area where the data is presented.”

Powering the sporting world with data

With these analytics, LaLiga Tech is able to provide insights to players and management around the probability of a player getting injured, the style of play when a goal was scored and, during the pandemic, how likely it was for players to transmit COVID-19 to each other during a match.

For the clubs and TV broadcasters, the data that’s generated provides the best time to start promoting season tickets for the next year. The insights will also reveal which events during a match – like yellow cards and goals – might cause people to tune into a game.

According to Martinez, these insights are not just reserved for football and can be applied to any sport from rugby, to tennis, to basketball. “We’ll need to use some different data sources,” says Martinez, “but the process is going to be more or less the same. We are now able to use what we have built to apply it to different leagues around the world.

"We want to package what we have built and to be able to apply it to a tennis competition, or the Olympic Games. If we talk about tracking the position of a player 25 times a second, this is something that a lot of sports could use. The conclusions we get may be different but the treatment of the information stays the same.”

Explaining or predicting future trends and outcomes is not a straightforward task. Regardless of quality data science or determination, it's rare for predictions to be 100% accurate. In the modern world, this is where two practices come into play - namely, machine learning and statistics. 

Historically, humans have been a necessary part of the statistical process as it was we who were ultimately capable of examining and interpreting data. Through machine learning, however, computers can now answer questions and make predictions in the same way.

Yet despite the fact AI and analysis models are fast becoming ubiquitous, there are still common misconceptions about the benefits of using a machine learning approach as opposed to a statistical approach. Educational direction and career opportunities are just some of the differences here, with key differentiating factors between courses in machine learning and courses in statistics. 

Machine learning vs statistics: What is machine learning?

Machine learning uses computers to identify patterns in data, without the computers needing to follow explicit instructions. Instead, they are programmed with initial algorithms and models, and they learn to adapt these based on the data inputted to offer answers. However, this also means there’s less human control over the results.

Dr Michail Basios, co-founder of TurinTech, says: “Machine learning models are built for providing accurate predictions without explicit programming. Machine learning models can provide better predictions, but it’s more difficult to understand and explain them.”

Machine learning vs statistics: What are statistics?

Statistical models make their predictions from numerical data. This is often gained from questioning people and asking them to make a choice; for example, it’s how political and election polling works, or consumer shopping surveys. Human involvement is usually more involved with the interpretation of this data. “Although some statistical models can make predictions,” Basios explains, “the accuracy of these models is usually not the best as they cannot capture complex relationships between data.”

Machine learning vs statistics: What are the major differences?

Despite the fact statistics are entered into computers as part of the machine learning process, there is one overarching variation between both methods. This difference lies in the objective or desired outcome rather than in how the data is analyzed, according to Paul Clough, professor at the University of Sheffield Clough and head of AI and data science at business intelligence and data science firm Peak Indicators. 

Providing an example, Clough describes a comparison between the amount of chicken feed consumed by a hen and the number of eggs that chicken produces. Where a statistical analysis would look to depict the correlation between the amount of feed and the number of eggs, Clough says, a machine learning analysis would act differently. The data would instead be used to predict what number of eggs will be produced in the future.

Machine learning vs statistics: When would you use machine learning over statistics?

Clough’s egg analogy is useful in exploring this further. “The machine learning model doesn’t tell you anything about running a more efficient farm,” he suggests, “whereas the statistical model would be unwieldy if you owned tens of thousands of chickens.”

When looking at huge data sets, machine learning can be more optimal. Many concede statistics are unable to offer a deeper analysis on the relationships and correlations between the data when the levels of data are high.

They also cannot be relied on for causation, probability, and certainty because the danger is they might be misinterpreted or intentionally misused to back up a particular argument. As the famous saying goes: “There are lies, damn lies, and there are statistics.” Human involvement presents a weakness in statistics.

Basios, however, explains you can apply statistical modelling when you understand “specific interaction effects between variables” and “have prior knowledge about their relationships”. Machine learning, meanwhile, can be used when aiming for “high predictive accuracy”.

It tends to be ‘deployed’ more at the source of the data; as this is gathered and grows, algorithms automatically begin to provide intelligence. In manufacturing, for instance, it can predict when machines will need maintenance, reducing disruption. It can also analyse one option when compared to another – predicting which outcome would be better.

Machine learning vs statistics: When would you choose statistics over machine learning?

Conceptualizing problems is a good use case for statistics, especially problems that demand close consideration. According to Graham Upton, chief architect of intelligent industry at Capgemini, statistics are also great for the creation of metrics, tables, and key performance indicators (KPIs). Statistics is focused on humans and easy to explain, AI writer and researcher Eleanor Watson adds, as opposed to machine learning.

Machine learning has, in theory, greater power in more complex scenarios, Watson goes on, such as those that involve more variables than a human would be able to deal with. The most powerful tool is not always required, though. Watson adds that many organizations tend towards the overpowered “jackhammer” of AI for surgical problems in which traditional data science would be more cost-effective and more efficient. Data science has become an “unsung hero,” Watson says, as in reality it props up the world’s current economy to a much great degree than machine learning. 

Machine learning vs statistics: Is there a role for both machine learning and statistics?

Governments, businesses, and organisations are increasingly becoming reliant on machine learning as well as traditional statistics to help them make better and faster decisions. Each method, though, has its own individual drawbacks to consider.

“Statistical models have limited predictive accuracy since, sometimes, the underlying assumptions of the model are far too strict to represent reality,” Basios says. “Today’s businesses are adopting hybrid methods combining characteristics of statistical modelling and machine learning to understand in-depth how the underlying models work as well as to generate accurate predictions.”

Machine learning vs statistics: What does the future hold?

Decision intelligence (DI) is the space to understand in the future, highlights Dr Lorien Pratt, chief scientist at Quantellia. She explains it bridges both technology and the everyday decisions we all make hundreds or thousands of times to construct our world.

“There is a certain class of problem – ranging anywhere from day-to-day business strategy to the realities of climate change – that involves intangibles, complex systems dynamics, and human-machine interaction,” she says. “Machine learning, as a ‘supercharged’ form of AI, can cut through the noise and make a real difference.”

Pratt continues: “Rather than starting by thinking about the data we have, the key is to begin by asking what outcomes we want and what actions can be taken to achieve those outcomes, and then to fit the data into this action-to-outcome pathway.”

Data is among the most valuable assets many organisations have, and if you have the right training there's a world of high paying, challenging work waiting for you – two of the most prominent are data scientist and data analyst.

And while the two have some similarities, the common pathways into careers in each area can be quite different, and candidates have a wealth of choices around types of study. The responsibilities and workloads of each are different, as are the further career prospects.

Two of the most popular career paths in data are those of the data scientist and data analyst. While similar in nature, on the face of it, they have varying routes of entry – including data science courses, data science bootcamps, as well as university degrees. They also have differences in terms of roles and responsibilities, as well as varying career prospects.

What does a data scientist do?

Online information systems are coming into being and growing so fast, the need to analyse data is exploding. In 2023, we're set to produce 120 zettabytes of data, up from 97 zettabytes last year and set to be outstripped next year with 147 zettabytes, according to Statista.

But companies and organisations that collect that data have to store and action it in particular ways for it to have any value. Enter the data scientist, who sets up frameworks for others to synthesise and extract insight from the data that's been gathered.

A data scientist combines computer science and mathematics to do this, with a particular focus on statistical tests and distributions. The companies that collect data need to ask it questions, and the data scientist figures out how best to frame the question and translate the answer. Like any business asset, data can be used to provide a financial or social advantage, and the data scientist helps to reach that goal.

Data science is the art of collecting, collating, processing, analysing and interpreting data in both structured and unstructured environments, creating frameworks that standardise it for further interrogation. Their arsenal includes machine learning or AI, data mining, statistical algorithms and more to 'smooth' data into a comprehensible form.

What does a data analyst do?

A data analyst takes a dataset from multiple sources, cleans it, and reorganises it for analysis. The tools of the trade include automated systems for unearthing unstructured data as well as stripping out or fixing corrupted or garbled files.

Whether it's designing and building an analytics system or delivering results from the analysis performed, a data analyst can enter the process anywhere along the workflow (or perform it entirely), including training other staff in conducting analyses that relate to their own business units.

A data analyst also needs to be a presenter, putting reports and summaries of results and interpretations into formats executives or other stakeholders can digest.

What skills do data scientists and data analysts need?

Data scientist skills

As with other scientific fields, a qualified data scientist is usually at least a holder of a Master's degree or PhD, and typically an expert in both technical and non technical principles.

The non-technical skills include critical thinking – clients and employers will have specific questions the data needs to answer, and the data scientist must structure the workflow so the available information can do so in direct relation to the industry or company concerned.

Data scientists also need to be competent communicators, able to put arcane concepts into relatable language for other business units, and the ability to think on your feet to solve problems as they come up is critical.

When it comes to the more technical aspects of the job description, the data scientist needs to know how to prepare datasets for the best analysis tools and methods. They need to know how to write effective code to do so, applying maths, statistics, and machine learning on a daily basis.

It also calls for a thorough understanding of database programming languages like Python, R Programming, SQL, Scala, MongoDB, MySQL, and more.

Data analyst skills

The responsibilities of a data analyst have some overlap with those of a data scientist. The data analyst similarly needs to be proficient in maths, which is applied more to the modelling and analysing of data once it has been prepared for treatment (possibly by a data scientist).

Every data analyst needs to have a deep knowledge of the various techniques needed to clean data, so analysis produces the best possible results, calling for high attention to detail.

When a project is complete, they then need similar interpersonal skills as a data scientist in order to communicate and present work to colleagues, particularly the ability to create visual or graphical representations of findings.

Formats and frameworks data analysts need to know include statistical programming languages like R or Python, data management, machine learning/AI, and systems like SQL.

Data science vs data analysis: Training pathways and career prospects

It's easy for outsiders to confuse the two professions because both are concerned with properly preparing and interrogating data for organisational insight. But there are quite stark differences.

Firstly, the data scientist is a more senior position, with more generalised responsibility in applying programming knowledge to create data models used for multiple purposes. A data analyst, by contrast, designs examinations of the data according to the established aims of other business units.

A career in data science

Becoming a data scientist will usually start with a degree in a computing, statistical, or scientific field, which will prepare you for the generalised knowledge of data science.

Depending on your area of focus, industry or the employer(s) you want to target, data analysis certifications like the CAP or vendor-specific study (Cloudera Data Platform Generalist Certification, Microsoft Azure Data Scientist Associate, SAS Certified Big Data Professional, etc) can be a good next step to specialise further.

According to Payscale.com, the salary range of a data analyst in the UK in 2023 is around £22K-£44k, with an average of £29,699. For data scientists, the range is £28k-£62k, with an average salary of £41,483, although seniority or further specialisation can put you over the £100k mark.

A career in data analysis

To many, the best way into data analysis is to get an entry-level job in the  field, giving you invaluable exposure to datasets and the tools you'll apply to them from real-world business processes.

You don't strictly need qualifications in maths or statistics to find work as a data analyst, but it certainly won't hurt to have them. Once you have a foot in the door, further training to boost your knowledge is common, and that can only expand your career advancement possibilities.

That might mean anything from the entry-level Associate Certified Analytics Professional (ACAP) to the Certified Analytics Professional (CAP), a vendor and technology-agnostic qualification that's more specialised and suits data analysts further into their careers.

Financial services organisations are facing a crucial moment. Niche financial categories, like peer-to-peer payments, have been completely disrupted by nimble SaaS entrants. FinTech startups are capturing the attention - and bank accounts - of a younger, more discerning audience.

As millennials and Gen Z become the next primary financial consumers, current industry leaders must quickly adopt a new marketplace strategy. The competition is gaining momentum through highly personalised digital experiences, all thanks to behavioural analytics.

Download this e-book to learn more about:

• Industry challenges and trends caused by digital transformation and consumer preferences
• The importance of first-party data, behavioural analytics, and a product-led approach
• How behavioural data drives personalisation across the entire customer lifecycle
• Tactics and strategies being used by the most successful disruptors in the segment

Provided by

Digital customer experience has become imperative for business success. Those responsible for driving this - data and analytics leaders - use web, product and digital experience analytics to analyse customer and user behaviour, digital product performance and usage patterns to improve the digital customer experience.

This guide shares how the digital analytics markets are blurring the lines of digital experience and customer journey analytics as these categories begin to overlap, and offers advice on how to choose a solution to complement business needs.

Download this guide for insights into the current vendor landscape and learn how your business can build contextually relevant customer experiences.

Provided by

The Master Data Management (MDM) market is evolving to meet the needs of a broader range of stakeholders who are seeking the business agility afforded by a single, trusted, enterprisewide view of master data. Data and analytics leaders should use this research to inform their MDM solution choices.

Gartner defines MDM as a technology-enabled business discipline in which business and IT work together to ensure the uniformity, accuracy, stewardship, governance, semantic consistency and accountability of an enterprise’s official shared master data assets.

Download this whitepaper to discover the pros and cons of the current vendors in the MDM solutions market.

Provided by

Master data management (MDM) is the gold standard of data management and analytics, but many organisations struggle to select the right solution to meet their needs. With a host of MDM vendors offering packaged solutions, it is sometimes difficult to understand how they differ, or which MDM platform would be the best fit to achieve an organisation’s goals.

Therefore, it is important to first identify your desired business outcomes and understand what features are needed so you can select an MDM solution that is the best fit for both short-term and long-term needs. Choosing an MDM solution that’s not aligned properly can result in wasted time, resources, and money.

Download this whitepaper to learn the key considerations in selecting an MDM platform.

Provided by

IT infrastructure services provider Kyndryl has announced a new strategic partnership with Teradata, a multi-cloud data platform for enterprise analytics.

The collaboration aims to help customers apply artificial intelligence (AI) and data modernisation techniques across their IT environments to boost the performance of various projects and speed up time-to-value.

Kyndryl’s data and AI services, as well as its infrastructure management tools, will be paired with Teradata’s high-performance cloud analytics and data platform in a move the pair claim will help customers migrate their on-premises data warehouses to the cloud with hyperscaler platforms.

In an announcement, the companies said they will help customers harness AI to gain better insights from their hybrid and multi-cloud data environments by designing, enabling, and supporting enterprise data modernisation solutions.

“We are thrilled to partner with Teradata who is a trusted and valued technology partner to many customers, providing a best-in-class analytics and data platform for enterprises across the globe,” said Nicolas Sekkaki, applications, data and AI global practice leader at Kyndryl.

“We look forward to deepening our partnership so we can deliver greater value and results that help companies transform and compete in their industries and around the world.”

London-based Teradata has created VantageCloud, a connected multi-cloud analytics platform for the enterprise, designed to help businesses leverage data and analytics to drive performance.

The firm says this new strategic partnership will help customers to leverage Kyndryl’s expertise – which includes advisory, implementation, and managed services – to get the most out of the platform.

There will also be a focus on delivering industry-specific analytics services geared toward customer modernisation challenges across key verticals such as manufacturing, communications, financial services, retail and CPG industries.

“This joint expertise will enable our customers to drive more actionable insights and higher value from their data through advanced analytics and AI/ML initiatives,” said Lisa Stewart, SVP of worldwide partners and alliances at Teradata.

Formed just last year as a spin-off from IBM, IT infrastructure services provider Kyndryl designs, builds, and manages complex mission-critical information systems for customers across more than 60 countries.

The firm’s partnership with Teradata is the latest in a string of collaborations struck since it went solo at the tail end of 2021. Recent deals include a hybrid cloud and edge partnership with Lenovo, a cyber resilience collaboration with Dell, as well as a delivery partnership with Oracle, which was announced back in June.

The company also has agreements in place with the three largest cloud providers, AWS, Google Cloud and Microsoft.

As organisations' data and analytics efforts become more mature, they collect more diverse data and perform more sophisticated analytics.

Today most organisations have either moved to the cloud or are planning to move to the cloud to modernise their data and analytics infrastructures. And in a recent IDC survey, nearly 80% of organisations said they stored more than half of their data in hybrid and multi-cloud infrastructures.

Download this report to learn more about multi-cloud environments and how to achieve seamless cloud data management across them.

Provided by

71% of CIOs say that the increasing amount of data resulting from cloud-native technology stacks exceeds human management capabilities, according to a new study.

According to the results, CIOs are increasingly concerned about the strain that managing technology stacks puts on their team members, and advise strategy change now before cloud complexity overwhelms teams entirely.

Software intelligence and digital transformation firm Dynatrace announced the results after a survey of 1,303 senior IT professionals, including CIOs and IT operations managers, across five continents. The survey indicates that complexity is already at unmanageable levels, with the average technology stack relying on seven separate platforms and systems, and each transaction involving an average of 35 different technologies from end to end.

Lack of overall insight into issues within the technology stack stood out among IT professionals as a problem arising from the scale of stacks. 97% of organisations state that obstacles are preventing them from achieving full-stack observability, or from adequately expanding security across their multi-cloud environments. On average, surveyed organisations had end-to-end observability of just 9% of their cloud environment.

Pressures of the coronavirus pandemic led to the widespread acceleration of digital services across the UK and the world, in both the private and public sectors. This has led to many benefits, particularly for small and medium-sized businesses (SMBs) whose transformation allowed for far greater predictive analytics.

But respondents also identified the rapidly changing nature of their environment as a source of strain, with 77% of CIOs indicating that their IT environment changes every minute or less. Amongst UK CIOs, this figure was even higher, at 93%.

The general lack of observability, the results make clear, is having a large knock-on effect on organisations, particularly over their ability to properly carry out and utilise big data analytics. Over half (57%) of the surveyed organisations state that data is frequently unavailable for analytics on demand, and just under two-thirds (63%) point to rehydration delays, in which data is located and decompressed for use, as preventing businesses from unlocking the value of their data.

Many respondents indicated that to improve efficiency and utilise team skills more effectively, a far greater level of back-end automation is required. 59% of CIOs state that without increased automation, their respective teams could become overwhelmed by the sheer scale and complexity of their stack, and the majority of CIOs (93%) argue that artificial intelligence in the form of AIOPs is becoming ever more integral to staving off burnout.

IT teams indicated that 40% of their time is currently spent simply “keeping the lights on”, and that automation will allow this time to be put to better use elsewhere. To use automation at its limitless potential, however, CIOs also indicate that an overhaul of data repositories is required. Almost half (47%) of surveyed CIOs want to see more adoption of the data lakehouse, a relatively new repository model that combines data warehousing and data lake techniques to allow for greater searchability through raw data that requires far less DevOp time to be spent on manual retrieval or rehydration.

“Multicloud and cloud-native architectures are critical to helping organizations achieve their digital transformation goals,” stated Bernd Greifeneder, Dynatrace founder and CTO.

“While organizations benefit from the flexibility and scale that these technologies bring, the explosion of observability and security data they produce is increasingly hard to manage and analyze.

“Existing tools – and there can be dozens of them – keep data in silos, making it difficult and expensive to unlock insights when organizations need them. As a result, they struggle to achieve the highest standards for security and performance across their digital services.”

Financial services have always been ahead of many industries in their use of consumer data technology. However, financial services are now facing challenges from increased regulation, high expectations from tech-savvy consumers, and more agile competitors.

Embracing further technology advancements will allow financial companies to remain competitive, and reduce costs while delivering even better customer service, but the question remains for CIOs on where to begin.

Download this guide which provides a clear direction for thought leaders on how to leverage big data analytics and how to grow share in the financial sector through digital capabilities.

Provided by

Cyber security provider Proofpoint has released a new Intelligent Compliance Platform designed to provide enterprises with modern regulatory safeguards while simplifying corporate legal protection devices.

Leveraging the company’s machine learning engine, the offering serves up AI-powered collection, classification, detection, prevention, search, eDiscovery, supervision, as well as next-gen predictive analytics – all while meeting complex compliance and information governance obligations.

The platform can classify and predict risks across a range of digital communications channels, files, email, and endpoint activities.

Proofpoint says this enables teams within compliance, IT, information management, and legal to gain visibility and access information with superior fidelity, as well as preventing risks in real time.

“We understand today’s organisations are overwhelmed with growing volumes of data that are incredibly difficult to manage,” explained Kevin Leusing, senior vice president and general manager of Compliance at Proofpoint.

“For Compliance and Legal staff, that means having to manually search and review petabytes of messages or files from regulatory compliance, supervisory, or investigation review queues.”

Instead, Proofpoint’s new Intelligent Compliance Platform pulls from several of the firm’s solutions to tackle the issue, including Capture, Patrol, Track, Archive, Discover, Supervision, and Automate.

Existing customers can leverage the platform in two different ways. Those looking for efficiency can reduce supervised communications content review time by up to 84%, the company says.

For teams more focused on effectiveness, sentiment analysis of communications data and a variety of machine learning models enable early detection of organisational behavioural misconduct.

Either way, Proofpoint says customers can easily manage communication content from platforms such as Microsoft Teams, Zoom, Meta, and Slack, thanks to its unified data collection and preservation.

Customers can also automate risk detection in real time communications, reduce time spent on data review by bypassing pre-flagged content, as well as leverage AI-powered data discovery with case management.

“The new Intelligent Compliance Platform vastly improves the detection of non-compliant communications and content while quickly pinpointing supervised insider risks,” Leusing added.

Asset-intensive organisations deploy APM solutions to improve asset maintenance decisions, and reduce the risk of failure for critical assets & equipment.

Although asset management strategies are advancing across industries, selecting the right solution is critical to success as it directly impacts the bottom line.

Download this guide for insights on APM offerings, so that your CIO can make an informed technology decision.

Provided by

KPMG has signed a 'strategic alliance' with Californian tech firm Alteryx that will aim to boost the accountancy giant's data analytics services.

With Alteryx, KPMG will seek to push forward plans to automate its own systems, while simultaneously ramping up its data handling platforms.

The two firms will combine their expertise in developing data strategies and digital transformation programs to help companies deploy "automated analytics" into everyday business processes so they can make "smarter and faster decisions". All of this will be powered by Alteryx's unified end-to-end analytics automation platform, which is rolling out across KPMG's systems over the coming months.

The two companies will be able to deliver actionable insights and also reduce operational costs, according to KPMG, with shared clients able to accelerate digital transformation plans with data-driven insights powering automation.

"As we adapt to the digital age, we are seeing a cultural shift across the firm to one founded in technology and data," said Emily Watkins, KPMG's head of deal analytics. "But trapping our capabilities in our technology teams is simply unsustainable. Working with Alteryx will put the power of data science and automated analytics in the hands of our people and our clients - empowering the next generation of data workers to capitalise on what their data is telling them."

Alteryx, which was founded in 1997 by the software entrepreneur Dean Stoecker, was originally known as 'SRC LLC'. The company provided demographic mapping systems to the US government for use in its yearly census.

The term 'Alteryx' was a type of data mapping software the company released in 2006, which it adopted as its name four years later. The company went public in 2017, floating on the New York stock exchange, and is headquartered in Irvine, California, with a development centre in Colorado.

It’s clear that embedded analytics as a core part of most enterprise software is a useful tool. However dashboards and data reporting have often been bolted on to existing software in the past, rather than being a core component of the application.

This can mean limitations when analysis is reviewed separately, causing disruption in workflows, lack of context, and friction in the user experience.

Download this whitepaper to better understand:

• What contextual analytics is and how it can enhance your business software
• How the maturity model can help speed up modernisation
• Why contextual analytics should be part of your next innovation

Provided by

While there is a broad understanding that successful organisations need to be data-driven, what does that mean in practice and how can your business achieve this?

This whitepaper lays out the key steps a company needs to take to become a truly data-driven institution that bases strategic business decisions on concrete evidence, and how agility, digital transformation and continuous innovation tie into this project.

Download now to learn how other companies are making the best use of their data and what that can teach your organisation.

Provided by

This IDG eBook examines how data can be overwhelming to organisations large and small and demonstrates how a modern analytics and machine learning (ML) approach will help make data accessible to your entire organisation, opening up new ways to put your data to work and reinvent your business.

Download now to read how organisations are applying modern analytics with tips on how you can get your data to work for you.

Provided by

Palantir’s decision to hire former NHSX AI director Dr Indra Joshi has been described as "deeply worrying" by a leading privacy expert amid the controversial tech company competing for a £240 million NHS federated data platform (FDP) contract.

Joshi, who left NHSX on 31 March 2022, oversaw the creation of the NHS AI Lab prior to being appointed to the UK government’s Scientific Advisory Group for Emergencies (SAGE) in May 2020. During her time at NHSX, she oversaw a number of significant tech deals as the UK’s healthcare system expanded its AI capabilities.

Jim Killock, executive director of the Open Rights Group which campaigns for digital rights and freedoms, described Palantir’s "recruitment of this senior NHS executive" as "deeply worrying".

Palantir is currently seeking a deal to manage the NHS's new £240 million FDP announced earlier this month. Described as "an ecosystem of technologies and services" and "an essential enabler to transformational improvements across the NHS", the system will manage population health insight, care coordination, elective recovery, vaccines and immunisation, as well as the NHS supply chain.

Killock, however, told IT Pro "the NHS should not be relying on them [Palantir] as a supplier".

The US data analytics company has long garnered controversy due to its alleged role in the Cambridge Analytica scandal, as reported by CNBC in 2018. Founded by US entrepreneur Peter Thiel as a service that supported the CIA's counterinsurgency and intelligence operations in Afghanistan and Iraq, Palantir has been accused of seeking "unprecedented" access to NHS patient data. Apart from the NHS, the company also manages sensitive data for the UK’s Ministry of Defence and Cabinet Office.

Due to Joshi’s close ties with the NHS, meanwhile, her hiring also has been criticised for representing a "conflict of interest" by critics, including ​​MBI Healthcare Technologies partner and former NHS divisional manager, Barry Mulholland. "It’s scandalous," he said. "What’s more is that the two people leading the procurement have been cheerleading the Palantir product across the NHS for the last [six] months. Clear conflict of interest," he wrote in a Twitter post.

Joshi’s hiring could potentially break the NHS’ strict employment regulations, too, which bar former staff members from lobbying or recruiting old colleagues for six months after joining the private sector.

A Palantir spokesman told IT Pro that the company is "delighted that Indra has agreed to join the team, which we are aiming to grow by 250 in the UK this year".

"Indra has extensive experience of cutting-edge technologies and their real-world application. She has helped lead standards and best practice in data and AI governance with both the UK and international bodies. Indra will work with Palantir’s many customers to help improve their operations and transform their businesses. We are proud of the work we have done with [the] NHS over the last two years in response to the COVID-19 crisis, which included helping to power the NHS' COVID-19 vaccination programme," they added.

If you want to win in the channel, B2B e-commerce is a tool you can't afford to ignore, and, according to McKinsey, it's now the leading edge of successful B2B sales.

Indeed, larger channel players like Westcoast, Tech Data and SCC have been working on their e-commerce propositions for years, anticipating further potential as data and analytics are further integrated into the technology landscape. That's despite persistent myths surrounding B2B e-commerce – including notions that customers aren't ready; that it's only for simple repeat business; and that customers always prefer face-to-face interactions.

Ian Briffett, director of sales process optimisation at Tech Data, confirms the distie continues to invest in digital platforms as part of its own digital transformation strategy, spurred on by the last two years of disruption. “If nothing else, that made people sit up and look for where we need to make a difference,” he tells IT Pro.

Briffett, however, “broadly agrees” with McKinsey that B2B e-commerce strategies can suffer from misperceptions, relying on outdated notions of online as being all about an advertising shopfront.

“If you go back about two years, many people did look at B2B e-commerce as being either marketing or purely transactional; it was a very different era,” he explains. “What's changed is it's more about how we enable a broad channel and provide additional capabilities and value to help customers grow their businesses.”

Fully integrated platforms will drive success

From working out how to increase software licensing and recurring revenues to providing the right levels of customer service, platforms that can drive processes will enable success.

Most B2B companies (65%) now offer e-commerce, according to McKinsey, meaning they can fully transact sales online – up from 53% in early 2021. Would the channel not be further ahead than non-tech sectors? Tech Data reportedly earned 50% of its revenues via e-commerce five years ago; back in 2017.

Canalys chief analyst Alastair Edwards says their research suggests 40% offer “some form of e-commerce online”, with another 20% planning online sales channels.

“It’s complex. What is a channel digital marketplace, as opposed to an e-commerce site?” asks Edwards. “We see large distributors and bigger partners, however, building digital marketplaces as a necessity as more software as a service (SaaS), cyber security and cloud-based services are transacted digitally.”

Data, and its growing role in digital transformation, poses questions to businesses that can be answered with e-commerce solutions, creating data that drives further insights, prompting further questions, and so on. From addressing immediate problems and pain points, e-commerce moves into driving new business incrementally.

“Software vendors, too, are pivoting away from selling traditional perpetual licences now to a subscription model,” Briffett adds. “Once you've sold something to a customer, you have to retain them, get them to renew.”

By helping vendors or distributors put the right information in front of the right people – at the right time – they should be better positioned to go and drive the “right cadence” of activities, he says.

“The other piece is to migrate these customers now to this subscription licence,” he continues. “We, therefore, need to be offering alternate quotes and help customers make informed decisions in an appropriate amount of time. That’s good for us, good for customers and good for vendors.”

For maximum benefit, information can be provided in a number of ways, including through portals, social media and other channels. If a channel partner needs a quotation to go from A to B, Tech Data publishes it automatically, for instance. Information can also be targeted to specific customers.

“We know that customers that got red boxes can move to blue boxes – or if they got one product from a vendor that they can move to another product in that portfolio,” Briffett says.

Will Omnichannel become omnipresent?

For each stage of the sales process B2B customers now employ a roughly even mix of traditional, remote, and self-service sales using e-commerce and digital portals, McKinsey suggests. Now, 94% of decision-makers say omnichannel sales are “as effective or better” than pre-pandemic models.

Many partner e-commerce forays are mostly for marketing, addressing different points in the customer journey. SCC, for instance, just partnered with IBM on a series of “virtual engagement” centres.

Tech Data, meanwhile, offers specific platforms and services for different tasks – such as for cloud services set-up with “appropriate” automated billing. Traditional account management, however, will remain important, not only for in-person calls or meetings but support and guidance. E-commerce platforms should facilitate, rather than take away, the entire burden.

“Partners, however, aren't there because we're beating them with a stick,” says Briffett. Canalys' Edwards warns, too, that face-to-face interactions can be preferred for complex projects or systems. Sophisticated e-commerce engines, meanwhile, perhaps embedded with AI analytics, mainly suits the scale of the Bechtles, Dustins, Ateas or Insights of the world. Without sufficient digital presence, however, partners can miss customer opportunities.

“Closed intranet-type procurement models are one potential differentiator, where pricing and discounts are pre-agreed,” Edwards says. “Capturing customer usage data from digital platforms can drive targeting that supports customer lifecycle strategies.”

To achieve cost-effective performance at scale and leverage multiple public clouds at the same time, Dell EMC PowerScale for Multi-Cloud has proved itself to be one of the most easy-to-use cloud file services today.

Enabling high availability for business continuity, data resiliency and flexible scalability, this technical review, as part of the Multi-Cloud Data Services for Dell EMC PowerScale, demonstrates how your organisation can solve your data challenges in the public cloud effectively.

Download this whitepaper to learn how Dell EMC PowerScale for Multi-Cloud can enable:

• Compute cost savings
• Storage cost savings
• And analytics efficiencies

Provided by

UltimateSuite offers four stages to its task mining, from data capture to pre-processing and enrichment, through to data analysis output, and they stand out from competitors with automation of much of the manual analysis that the more typical tools offer.

Download this whitepaper to learn more about UtlimateSuite and what this task mining tool can offer your business, as well as how easy it is to set up.

Provided by

The French city of Toulouse is famously known as ‘La Ville Rose’, the Pink City, due to the sheer volume of red bricks that run through its streets and buildings, creating a breathtaking spectacle. It’s also, however, home to a burgeoning startup scene with a focus that's a little more out of this world.

Space technology has almost taken over the fourth largest city in France; many of its news tech businesses are focused on satellite data, which has caught the attention of the French Space Agency (CNES).

Towards the end of 2021, CNES chose Orange Business Services (OBS) to lead an industry consortium to design, deploy and maintain a new storage system for its scientific spatial data. This is, in part, to help CNES modernise its storage infrastructure, but also find ways it can use the plethora of satellite data it receives in better ways. When completed, it’ll resemble a private cloud platform, and will be based in Toulouse.

“Toulouse is really the capital of space in France,” according to Etienne Bonhomme, France’s director of cloud activities at Orange Business Services. “This is where you've got all the companies in the space market – they are located in Toulouse.”

Oceans of data

The topic of space is interesting in itself, Bonhomme explains, but what’s more exciting is the data coming from the many satellites orbiting Earth. OBS aims to use the data CNES captures for the benefit of society, and to boost the economy, citing examples such as transportation and tourism. The French space sector employs roughly 20,000 to 30,000 people, but their work with data can also help the two million that work on tourism, he explains, touching on how the sector can bed roots into other sections of the economy.

“This is why CNES has launched this project,” he says. “A private cloud auto-scaling infrastructure, which is being built in one place; that they expect will be a new data zone designed to take advantage of the split between compute and storage. When you’re looking for data, you won’t be slowing down the user with silos. You have, instead, each individual element that can scale when necessary.”

CNES has specifically drafted in OBS to build a data lake because the volume of data coming from this space is “exploding”, Bonhomme explains. So much so, the firm foresees the volume of data could actually double by 2025 because there will be more powerful satellites relaying even more data to ground stations, which, in turn, inevitably means more data to analyse.

Bonhomme describes Toulouse as the capital of space in France

The reasoning behind a data lake is very much tied to the potential for satellite data because it removes certain barriers for users. A data lake is a type of centralised repository that's different from a data warehouse, in that it doesn’t require a holder to store data with any specific ideas about its intended use – it accepts both structured and unstructured data. One of the reasons people sometimes have structured and unstructured forms of data is that it often gets stored in different locations – on-premise and the cloud – with the data lake offering a single virtual facility to house it. This is arguably the best place to store data collected from satellites, which have, seemingly, endless possibilities.

The next space race

“The purpose of the data lake is actually at the heart of the digital transformation of the French Space Agency,” Bonhomme explains. “Beyond storage, the challenge is to process and transform the information in order to accelerate innovation cycles and tasks, and use data in various data initiatives it carries out.”

The new data lake, technically, will be capable of hosting 100PB of special data. Just 1PB is the equivalent of 500 billion pages of standard printed text or 11,000 4K movies. Much of the data collected will have an economic value, particularly for geolocation and imagery, but it will also hold information of importance to technologies that monitor climate change. These include systems monitoring rising sea levels, or machines that analyse the ozone layer. Geospatial data is also important for emerging technologies such as driverless cars or for other, more traditional services, like digital maps and even urban planning.

In supplying the infrastructure for CNES, OBS is helping to facilitate the advancement of scientific understanding, but also aid the emergence of new services and products within the industry. Having such data stored in a single repository, that doesn’t require stipulations to structure it, is also a powerful capability.

“The dream for every organisation is to be able to transform this data into gold,” Bonhomme adds. “This is also the case in the space market – they want to be able to transform their data into something that's billable.”

Google has expanded its open-source differential privacy (DP) platform to support the Python programming language, widening availability to millions more developers and data analysts.

The announcement makes Python the fourth language supported by the project after initially launching in 2019 with support for C++, Java, and Google-created language Go, sometimes referred to as Golang.

It comes after Google reported a significant number of developers contacting the company expressing their interest in using the open-source library for their Python projects. Google worked for more than a year with OpenMined on Python support and said numerous projects have already used its DP library, including Australian developers who have accelerated scientific discoveries through analysing medical data in a private way.

DP is a system used by data analysts to preserve the privacy of the individuals whose data is used in an analysed data set. Work to develop strong DP dates back decades, but only in recent years have tech giants such as Google and Apple embraced the system.

One of the key areas of DP development for Google in the past year has been on providing a tool for developers within the library to fine-tune the 'epsilon' - a mathematical measure of privacy. Finding the optimum epsilon requires a great deal of trial and error to perfect and having a tool within the library that allows developers to make adjustments to yield a lower epsilon, which indicates a more private release, means individual projects are able to be tuned as privately as possible.

Google said now Python is supported, the DP library is now available to nearly half of all developers worldwide which means more developers and researchers will be able to analyse data and make new discoveries while preserving the privacy of users to whom the data belongs.

Python is among the most popular programming languages currently in use and won 'Language of the Year 2021' from the TIOBE index, which ranks programming languages based on their popularity. Python is useful for a wide range of programming activities but is especially well-known for its capabilities in data analysis, making it a natural progression for Google's DP library.

As part of the launch, Google has released a new web-based product, pipleinedp.io, which allows any Python developer to analyse their dataset with differential privacy. Google also said it has seen organisations experimenting with new use cases such as showing a website's most visited web pages by country, in an anonymised fashion.

The library is compatible with leading large data processing engines, the Spark and Beam frameworks, and Google will be launching an additional tool to help users "visualise and better tune the parameters used to produce differentially private information".

"We encourage developers around the world to take this opportunity to experiment with differential privacy use cases like statistical analysis and machine learning, but most importantly, provide us with feedback," said Google announcing the news. "We are excited to learn more about the applications you all can develop and the features we can provide to help along the way.

"We will continue investing in democratising access to critical privacy-enhancing technologies and hope developers join us in this journey to improve usability and coverage. As we’ve said before, we believe that every Internet user in the world deserves world-class privacy, and we’ll continue partnering with organisations to further that goal."

What is differential privacy?

Differential privacy is a tool that has gained acclaim in recent years as data and identity protection have become focal points for researchers, businesses, and regulators alike.

Some argue it is fundamentally necessary in data analytics to preserve the privacy and hide the identity of people whose data is being analysed. For technology companies especially, it has been at the forefront of how their users expect them to handle the data they hold on others.

DP works by adding 'controlled noise' to datasets so that people cannot be individually identified by the data they provide to the dataset. For example, if residents of a neighbourhood supplied data for analysis involving their salaries which were then represented as an average, and one resident left the neighbourhood, their salary information could be tied to their identity by looking at the difference in the data pre- and post-move.

Similarly, if two databases were analysed, one with a single data point on 50 people and one with a single data point on 51 people, the analysis results for both would have to be indistinguishable from each other to avoid identifying that 51st person, in order to qualify as differentially private.

Adding controlled noise to a dataset would remove the possibility of identifying an individual by skewing the statistics just enough to remove the element of identification, without significantly compromising the accuracy of the results.

All major Big Tech firms have embraced DP in different ways. Microsoft's AI Lab works with Harvard University on projects to facilitate DP-enabled research. Apple has used DP on its products since macOS Sierra and iOS 10, and Facebook and Amazon also have experience working with the system too.

The future of artificial intelligence (AI) in medicine is in Cambridge – and it’s superfast. Nvidia has unveiled Cambridge-1, the most powerful supercomputer in the UK and among the top 50 globally, installed during the pandemic at a cost of $100 million (£73 million) to the GPU giant.

The aim isn’t just to sweeten Nvidia’s acquisition of British chipmaker ARM, but to show off its DGX SuperPOD scalable computing setup for AI – and help power the future of medicine. “Cambridge-1 will empower world-leading researchers in business and academia with the ability to perform their life’s work on the UK’s most powerful supercomputer, unlocking clues to disease and treatments at a scale and speed previously impossible in the UK,” said Jen-Hsun Huang, CEO of Nvidia, at the launch of the new supercomputer.

“The discoveries developed on Cambridge-1 will take shape in the UK, but the impact will be global, driving groundbreaking research that has the potential to benefit millions around the world.”

Cambridge-1 is named in honour of the city where genomics and computing got its start, but it’s also where Nvidia plans to base its UK HQ. “Cambridge-1, and our purchase of ARM, is our giant investment in the UK, giving us a platform to work with the amazing universities, companies, and over 1,000 AI startups in this community,” said Huang.

Scalable supercomputing

With eight petaflops of Linpack performance, Cambridge-1 is the UK’s fastest supercomputer, just pipping the previous leader run by the Met Office. How has Nvidia, more famous for its gaming GPUs, achieved this feat? The answer lies two decades in the past, when researchers struggling with machine learning started trialling Nvidia’s hardware for their AI projects, sparking a boom in AI that continues today. Nvidia has since started developing custom hardware for AI tasks, including the Ampere processor at the core of Cambridge-1.

The key building block is Nvidia’s AI-system-in-a-box, the DGX. It features eight A100 Ampere GPUs as AI accelerators, alongside a 128-core AMD CPU, and includes networking and storage as well as AI software and models. The idea is that an organisation can easily scale its AI efforts by plugging together multiple DGX modules; the SuperPOD upon which Cambridge-1 is based stacks 80 together in one package, for a supercomputer in a box.

What’s intriguing about Cambridge-1 is that it’s the first Nvidia supercomputer that’s been made available for use outside of the company itself. The chipmaker has previously built supercomputers for its own uses, such as developing new GPU technology, but the Cambridge-1 is the first created for outside partners.

The Cambridge-1 isn’t actually in Cambridge, however. To enable the supercomputer to be powered by renewable energy, it’s situated at green hosting provider Kao Data’s location in Harlow, Essex.

Future of medicine

AI is increasingly assisting medicine in myriad ways, from scanning images to spot symptoms of cancer to sifting through data to identify patterns that can guide research into diseases such as dementia. It also develops models to help create new drugs. However, ethics and safety concerns mean development must be done carefully.

Right now, there are “two core problems”, said Sebastien Ourselin, head of the School of Biomedical Engineering and Imaging Sciences at King’s College London. One is access to data, and the other is access to the computing power needed to process it.

On the first problem, industry and the NHS need to work closely together to increase accessibility to relevant data, an area that’s been fraught in the UK because of privacy concerns. Ourselin noted that it’s key to ensure that AI is developed using unbiased datasets that represent the demographics of the local population. Otherwise, it won’t work for everyone.

The second problem is a growing one. The latest AI systems can do incredible things thanks to ever-more sophisticated models and massive datasets. But they demand a lot more computing resources than older, simpler systems. “The power of your computer limits you,” said Lindsay Edwards, vice president of data science and AI, respiratory and immunology, biopharmaceuticals R&D at AstraZeneca. “If it takes you three weeks to train a model, your ability to hone that model is really difficult – you need to be able to iterate quickly.”

Hence the eagerness of medical organisations to have a go on Cambridge-1. Access to this supercomputer could spark a leap forward in advanced medical modelling, finally allowing faster breakthroughs in drugs, genomics and understanding of diseases.

The first projects to access Cambridge-1 include drug discovery and medical imaging with AstraZeneca, genetics and health predictions with GSK and genomic sequencing with Oxford Nanopore Technologies. It will also generate synthetic brain images for King’s College London and Guy’s and St Thomas’ NHS Foundation Trust to power research into dementia, strokes, cancer and multiple sclerosis.

“Some of the first projects slated for the supercomputer are using AI to sequence genomics, discover novel drugs, and to unlock the mysteries of dementia by studying MRI scans,” said Huang.

AstraZeneca and drug discovery

AstraZeneca has had plenty of attention lately thanks to its production of the Oxford-developed COVID-19 vaccine, which was being jabbed into arms less than a year after the pandemic hit – and AstraZeneca is hoping that Cambridge-1 will accelerate future drug decvelopment. The pharmaceutical giant is using a new transformer-based neural network architecture to predict reactions to drugs and optimise the development process. It’s a pioneering approach because the network will effectively train itself: just as a natural language processing AI will learn the basic rules of language by “reading” as much text as possible, this system is aiming to learn the rules of chemistry, Edwards explained.

AstraZeneca is also running a trial looking at digital medical images, hoping to apply a trained algorithm to annotate images of tissue samples, while also looking for anything that could signal a drug reaction. “One of the challenges with digital pathology is that the images are huge. The files are incredibly large,” Edwards said. “What we’ve done in the past is break those files up and train models on those – but that comes with a whole host of problems. Computation on the scale of Cambridge-1 allows us to do whole-slide image processing, which is really transformational.”

Genetics, GSK and Oxford Nanopore

Elsewhere, GSK and Oxford Nanopore are both working on genetics and genomics projects, in the hopes of better understanding disease and potential treatments. GSK is using AI to better understand biology using genetic evidence, to not only develop drugs more quickly but also better understand when and how to use treatments. “We know that medicines which have genomic evidence behind them are twice as likely to succeed and result in successful medicines for patients,” said , Steve Crossan, VP for AI and machine learning at GSK.

Oxford Nanopore meanwhile is using Cambridge-1 to run a model originally developed for speech recognition, to examine biological markers that aren’t yet well understood. “There’s an enormous amount of work that goes into not just accelerating understanding what those markers are, and how you call them and how you get them working reproducibly, but how they impact health,” said Sinclair Dokos. The hope, she added, is that access to Cambridge-1 will let Oxford Nanopore speed up its algorithm development, leading to more accurate genomic analysis.

Brain scans for research

King’s College London and Guy’s and St Thomas’ NHS Foundation Trust are working to better understand brain diseases such as dementia, stroke, multiple sclerosis and cancer, with the aim of speeding up diagnosis and improving treatment. The model will be trained on tens of thousands MRI scans to teach it how to simulate brain images with features such as age or disease, giving researchers more material to work with.

“By generating so many images, you can create a situation where you are absolutely certain that you have all the possibilities of how a brain will look if it’s healthy,” explained Ourselin. “But what you could do as well is start to generate the same amount of data for a specific disease, and then start to understand the progression of the disease.”

These are only the initial projects: the aim is for Cambridge-1 to push British AI-based medical research to the forefront. “Cambridge-1 [allows us] to try things out that we couldn’t try before,” said Edwards. It could even help develop the field of personalised medicine, according to Ourselin: “By comparing in-depth information of a specific patient with the entirety of the population, and 20 years or 30 years of data, we will be able to do better precision medicine,” he said.

Nadine Hachach-Haram, clinical innovation lead at Guys and St Thomas’, predicts that finding new drugs and treatments will unquestionably change the pharmaceutical industry. But there’s more: “The even more exciting thing for me is reimagining healthcare where we’re bringing together cutting-edge life sciences innovation and compute power, to ultimately enable us to do what we’re all here trying to do: deliver the best care to our patients.”

The digital arm of the Department for Work and Pensions, DWP Digital, has launched a new Chief Data Office (CDO) in an effort to improve the services it offers to the public.

Led by its chief data officer, Paul Lodge, the focus of the CDO will be "small" for the first year, with the main aim to drive up quality, compliance, and consistency in data sharing.

Other data management policies will follow on from there, the DWP has confirmed.

The data available to DWP Digital comes in an "increasingly complex" variety of formats, it said, so the first order of business will be to ensure the "consistent" use of data in a secure, legal, and ethical way.

A challenge it will immediately face will be to manage the various independent systems used to collect data, which have created inconsistencies in the past and has made it difficult to reuse data across multiple systems.

To support the CDO's ambitions, a data strategy for the department has been developed by its head of data strategy and enablement, Katharine Purser. This will set out the principles DWP Digital will use to manage data more efficiently.

"DWP is custodian of more than 200TB of data," Purser said. "That's the equivalent of approximately 67 billion single-spaced A4 pages of text, and this is growing in both volume and diversity at a rate of about 10% a year - including new data types such as voice and video.

"We know that one of the best things we can do to accelerate our efforts in the area is to improve our ability to re-use data. Good data can help us understand the environment we operate in and the way we operate within it, the needs of the citizens who come to us, and the impact our services have."

Purser will also be discussing the challenges around establishing the CDO in a digital leaders webinar on 18 October.

HPE has announced a series of new GreenLake offerings which signal its entrance into the analytics and data protection markets.

HPE GreenLake for analytics is a set of open and unified analytics cloud services that aim to modernise data and applications stored on-premises, at the edge, and in the cloud. It will enable analytics and data science teams to scale up Apache Spark lakehouses and speed up artificial intelligence (AI) and machine learning (ML) workflows, according to HPE.

HPE GreenLake for data protection offers backup cloud services as well as disaster recovery sourced from HPE’s recent acquisition of Zerto. It provides restore times of as little as minutes, allowing organisations to recover from ransomware attacks without impacting business operations, regardless of the scenario.

HPE has also annunced the Edge-to-Cloud Adoption Framework, which aims to support customers in creating an effective cloud operating model by being able to evaluate it using categories such as Strategy and Governance, People, Operations, Innovation, Applications, DevOps, Data, and Security.

Beside it's three main new offerings, HPE has also showcased a new addition to its AI Ops for infrastructure, HPE InfoSight. Known as HPE InfoSight App Insights, the tool is capable of detecting application anomalies, providing recommendations, and preventing disruptions in application workloads. Customers looking to make smarter, data-based IT decisions across edge-to-cloud will also benefit from the new HPE CloudPhysics.

Commenting on the announcements, HPE president and CEO Antonio Neri said that “data is at the heart of every modernisation initiative in every industry”.

He adds, however, that many organisations "have been forced to settle for legacy analytics platforms that lack cloud-native capabilities, or force complex migrations to the public cloud that require customers to adapt new processes and risk vendor lock-in."

According to Neri, the big data and analytics software market, estimated by IDC to be worth $110 billion by 2023, “is ripe for disruption, as customers seek a hybrid solution for enterprise datasets on-premises and at the edge”.

“The new HPE GreenLake cloud services for analytics empower customers to overcome these trade-offs and give them one platform to unify and modernise data everywhere," he said. Together with the new HPE GreenLake cloud services for data protection, HPE provides customers with an unparalleled platform to protect, secure, and capitalise on the full value of their data, from edge to cloud.”

Customers can benefit from the HPE Edge-to-Cloud Adoption Framework starting today, while HPE GreenLake for analytics and HPE GreenLake for data protection will become available in the first half of 2022.

Oracle and the University of Oxford have developed a global pathogen analysis system (GPAS) to help governments and medical communities cope with new COVID-19 strains.

Powered by Oxford's scalable pathogen pipeline platform (SP3), Oracle APEX, and Oracle cloud infrastructure (OCI), GPAS allows standardized genome data analysis and comparison for the SARS-CoV-2 virus.

Early adopters of the system include the University of Montreal Hospital Centre Research Centre, the Institute of Public Health Research of Chile, Oxford Nanopore Technologies, and more.

GPAS, which is available as a free resource, has also been integrated into Public Health England's New Variant Assessment Platform.

Researchers and governments are reportedly using the system to stream pathogen data in an effort to track new COVID-19 variants. With users’ consent, test results can be made accessible to participating labs worldwide.

The cloud-based system can also integrate with existing sequencing infrastructures and data repositories to enable seamless tracking of pathogens. GPAS also features built-in computational infrastructure, relieving labs of building and operating their own.

According to reports, the upcoming version of GPAS will include enhanced scalability and data-sharing functionalities.

"GPAS is the first industry standards-based service anywhere in the world, offering a standardized sequence data analysis service for users on the cloud," said Derrick Crook, professor of microbiology at the University of Oxford's Nuffield Department of Medicine.

Crook added, "Users will be able to access, upload and process their sequence data fully under their sovereign control and receive back fully analyzed data in as little as 20 minutes of successful upload. If they select to share data, they will contribute to electronic dashboard visualizations of global data revealing the daily changes in the way the pandemic is progressing and how the virus is changing. This will enable continuous assessment of the pandemic and help guide national and global interventions to curb the impact of the virus."

Workflow specialists Appian has announced the latest version of its 'low-code' automation service at its European conference on Tuesday.

The update to the Appian Low-Code Automation Platform promises to be faster and more efficient for building enterprise applications through advancements in low-code data and improvements to the user interface.

Appian hopes its new improved platform can make it easier for companies to access and transform their data with new services to sync, store and view it. In addition to new features that aim to improve automation, connected systems and application deployment.

Ahead of the announcement, CEO Matt Calkins spoke to IT Pro about the improvements and how low-code has evolved over the course of the pandemic.

"We went public as the first low-code firm in the world, back in 2017, and at that time, low code meant pretty much EPM or workflow," Calkins said. "It was drawing instead of coding your application, but we have expanded it. In the years since then, we expanded downstream into automation.

"So you can now learn about the processes in your enterprise before you automate them. You can even translate the process mining map into a process model, and then edit it. So we've expanded both downstream and upstream from the original workflow centre of low code and now we believe that we've redefined what workflow needs to be to be viable in 2021."

This new approach to workflow is about change and the need to be adaptable and change quickly has been "reordered" by the pandemic. According to Calkins. In response, Appian has made agility "imperative" and decided that it would make sense to gather a suite of products to make it as easy as possible to build a new app, a new process and to run that process.

"We know how important it is now, to create change, to be empowered to change, and so we're going to give you this platform that is the best possible way to define your new patterns and execute them. And we know that that's what the market wants right now because they're all desperate for ways to change for the better."

A regiment in the British Army has been globally recognised for its work on a "groundbreaking" data platform that was used to support the UK's COVID response.

The platform, built by 42 Engineer Regiment, is a geospatial intelligence system (GIS) capable of pulling together data from NHS facilities, and those belonging to its partners, to inform real-time decisions.

The work, which is considered the first to be the first example of a military project being used to support a civilian emergency, has won the 2021 Esri Special Achievement in Geographic Information Systems (SAG) Award.

Information system specialists Esri recognises organisations that have set new precedents and improved the world using geographic information systems. In the case of the 42 Regiment, its work helped to inform decision makers and planners with a dynamic common operational picture of all aspects of the COVID crisis in the UK.

It broke new ground, according to Esri, by being the first data platform capable of overlaying COVID incidence data with NHS infrastructure, hospital bed capacities, population demographics, partner facility locations and available personnel numbers. The extent of its reach was also unique to the UK's pandemic response, given it joined up multiple organisations and over 1,000 users, spanning the Cabinet Office and government departments to the NHS, SAGE, local authorities, first responders, and military personnel.

The 42 Regiment is said to have drawn on its experience using GIS for humanitarian operations to create a secure, web-based 'COVID Viewer' using Esri's ArcGIS service within a matter of days. This gave instant access to real-time data to all parties to make faster and more informed decisions.

In turn this helped to ensure effective deployment of limited resources and ultimately, saved lives, protected the NHS, and combated the pandemic, according to Esri.

"42 Engineer Regiment (Geographic) is proud to have played a key role in enabling the government to make faster, well-informed decisions in the battle to protect the NHS and save lives," said 42 reg support lead captain Luke Parker. "The online GIS platform helped lots of different organisations to work together to make the big decisions about how the UK responded to the pandemic".

Spire Global and Mantle Labs are teaming up to develop integrated risk assessments solutions for agri-commodity companies.

The partnership is also aimed at providing early alerts on crop health, weather patterns, and pest activity to farmers.

Spire’s low-orbit nanosatellites record data from every layer of the atmosphere in real-time. Over 110 satellites monitor atmospheric temperature, pressure, and humidity at over 10,000 intervals per day, allowing for an accurate 3D weather profile.

Geobotanics by Mantle Labs is an end-to-end crop-monitoring platform that includes crop stress warnings, pest forecast, and production risk analysis. Powered by artificial intelligence (AI), Geobotanics also offers plant nutrition and seed recommendations.

Per reports, the radio occultation data collected by Spire’s satellites will complement Mantle Labs’s crop health data.

Additionally, Spire's data analytics will allow Mantle Labs to access critical weather data from regions lacking reliable weather infrastructure, including Southeast Asia, India, South America, and Africa.

Combining their expertise, Spire Global and Mantle Labs aim to develop a robust crop health forecast.

“We operate in many countries where there is a lack of reliable weather monitoring. This leads to issues for industries like agriculture where weather data and forecasting are vital for risk mitigation,” explained Swapnil Baokar, co-founder of Mantle Labs.

Baokar added: “Our cutting-edge Geobotanics platform, powered by AI, is used by leading customers in food and agriculture. Adding Spire’s revolutionary technology will offer unprecedented access to high-quality weather forecasts at a global scale and offer greater visibility into weather-related risks, helping the world’s farmers build climate resilience.”

Commenting on their strategic partnership, Spire vice president and general manager of global data services John Lusk said: “We are proud to be partnering with Mantle Labs to help farmers, banks, crop input, insurance, and agri-commodity companies access previously difficult weather data in under-observed areas around the world in one of our most complementary partnerships to date.

"It is critical that technology reach these locations to empower farmers on the ground, maximize their profit, and properly anticipate risk assessment, especially in the face of rapidly accelerating climate change.”

Cricket may have something of an old-fashioned reputation, with its genteel attire and somewhat languid pace, but just like every other sport, it’s in the process of being revolutionised by increasing use of digital technologies and data. Nowhere is this more evident than in Australia, where cricket remains an almost universal passion.

In this week’s episode of the IT Pro Podcast, we talk to Michael Osborne, general manager of technology for Cricket Australia, to find out how the organisation - in collaboration with technology partner HCL - is using data to support volunteers, develop players and connect with fans of the game.

Highlights

“Over the summer there's over 700,000 registered players in the country who are competing at every level from the under-eights, all the way through to the seniors. In this day and age, we have about 60% of those games that are now live scored on a mobile app. So the old days of the big paper book, sitting by the sideline, keeping track of all the statistics; for 60% of those games - which happen every weekend - It's live scored, streamed across to our MyCricket app, where then parents, brothers, friends, siblings, can follow in real-time how the scores are coming together.”

“One [sport we’ve been inspired by] is the NBA. We've had the opportunity to connect with them a couple of times as well. But the NBA does such a good job of engaging the youth demographic. If you can believe it, the NBA itself is one of the most popular sports in Australia from a fandom perspective, particularly amongst that youth demographic. And none of those kids, or very few of them, will ever get to go and see a live NBA game. And in fact, we're gonna have to follow these games at inconvenient times of the day and night. Then yet, they're able to reach them, to connect with them, they're able to amplify their player brands, provide solutions into markets that really get people engaged. So a lot of what the NBA is doing, from a direct to consumer perspective, is really interesting to us”.

Read the full transcript here.

Footnotes

• Australia plans to become a leading digital economy by 2030
• Australian government invests $8 million to boost cyber skills
• How AWS took over sports
• How big data can give you a competitive edge in sports
• Sports industry urged to reform cyber security after £1m Premier League phishing scam
• IT Pro Live: How Pro Football Focus became the NFL's secret weapon

Subscribe

• Subscribe to The IT Pro Podcast on Apple Podcasts
• Subscribe to The IT Pro Podcast on Google Podcasts
• Subscribe to The IT Pro Podcast on Spotify
• Subscribe to the IT Pro newsletter
• Subscribe to IT Pro 20/20

IBM has announced it will be acquiring Spanish hybrid cloud consultancy firm Bluetab Solutions as it continues to flesh out its new hybrid cloud and AI strategy.

The financial details of the deal have not been revealed, but it's expected to be completed by Q4 of 2021.

Bluetab will become a key part of IBM's hybrid cloud and AI strategy as a data services consulting practice. The firm, which is based in Madrid, was founded in 2005 and has brand partnerships in banking, telecommunications, and energy and utilities industries across Spain, Mexico, Peru, and Colombia.

It works with enterprises by migrating their on-premise systems to hybrid multi-cloud data platforms using a combination of public cloud providers and technologies, such as Red Hat OpenShift.

"The outside-in digital transformation of the past is giving way to the inside-out potential of using company-owned data with AI and automation to generate business value and create intelligent workflows," said Mark Foster, senior vice president of IBM's Services and Global Business Services.

"Our acquisition of Bluetab will fuel migration to the cloud and help our clients to realise even more value from their mission-critical data."

Bluetab's expertise in data and cloud migration services includes specialised data strategy, data fabric, and advanced analytics, according to IBM.

The firm's data experts will also be joining IBM's Global Business Services to capitalise on opportunities in the rapidly growing market for data services, which is estimated to reach $232 billion by 2024, according to Gartner. That's double the $123 billion it reached in 2020.

"The key to solving data challenges for our clients has been the exceptionally talented and experienced team we have been able to build as well as the value-added accelerators we have developed," said José Luis López, co-founder of Bluetab. "We could not be more excited by the opportunity that IBM offers us to continue to grow our team, to build on our accelerators and to help more clients achieve leadership positions by leveraging their data."

Approx. read time: 1 hour, 20 minutes

Discover how you can make huge cost savings and minimise complexity by lifting and shifting your existing Oracle applications and workloads to the cloud. Infrastructure-as-a-Service (IaaS) gives you the predictable performance that you get on-premises , but with increased scalability, better security, and accessible, real-time data analytics.

Read this guide for an in-depth understanding of IaaS and how the cloud can help you provide value to the wider business, plus learn how to eliminate manual tasks and reduce costs while running your high-performance workloads.

Google Cloud is launching three database services across its analytical portfolio that aim to unify its data services.

Dataplex, Analytics Hub and Datastream are all 'previews' that aim to reduce silos and securely predict business outcomes in 'dynamic' digital environments.

Datastream is a serverless offering and replication service that enables customers to replicated data streams from Oracle and MySQL into Google Cloud products like BigQuery or Cloud Spanner.

The Analytics Hubs is a service where companies can create, curate and manage analytics tools. This includes sharing data insights, dynamic dashboards and machine learning models both internally and externally. Google said it builds on BigQuery's existing sharing capabilities, which have already proven popular with organisations.

The third new service is called 'Dataplex' and is described as an '"intelligent data fabric". It meshes the best of Google Cloud with open source technology to enables secure and rapid curation, integration, and analysation of data at scale.

"Data must be thought of as an ability that integrates all aspects of working with it," said Gerrit Kazmaier, VP and GM of Databases, Data Analytics and Looker, at Google Cloud. "Every industry is accelerating their shift of being digital-first as they recognise data is the essential ingredient for value creation and the key to advancing their digital transformation."

"At Google Cloud, we're committed to helping customers build the most powerful data cloud solution to unlock value and actionable, real-time insights, needed to future-proof their business."

Equifax is an early user of the new previews, working with Google to incorporate Dataplex into the company's core analytics platform. The firm is hoping to simplify its workloads and build a single interface for policy management and governance across all its analytics data offerings.

"Google Cloud has been a critical part of the Equifax journey, helping us protect our customers' sensitive and proprietary data," said Bryson Koehler, Equifax CTO. "Google Cloud allows us to create a rich, unified and trusted data ecosystem between business units and partnerships - one in which everyone gains immediate value."

Commonwealth Bank of Australia (CBA) has launched a joint venture with data analytics firm Quantium to provide data insights to customers and automate decision-making.

The venture is called CommBank iQ and will provide fast, current and relevant data insights to help institutional clients deliver better products and services for their customers and stakeholders. It will begin trading in the second half of 2021 and all transaction data handled by Commbank iQ will be aggregated and fully de-identified.

The venture will see the creation of a number of data-driven products, decision support tools and AI decision engines that harness de-identified data to automate decision-making, helping institutions act and deliver change based on unprecedented customer insights, according to the companies.

“As Australia’s biggest bank, we see more transactions than any other institution in the country,” said Andrew Hinchliff, CBA’s group executive of institutional banking and markets. “By working in partnership with Quantium, we can harness the power of this peerless dataset to help Australian governments, businesses, investors and NGOs predict and adapt to the changing needs and wants of their customers.”

Quantium had previously worked with National Australia Bank (NAB) to build insights for clients but approached CBA a year ago as it recognised it held a larger pool of transaction data to work with, according to the Australia Financial Review. The move highlights CBA’s objective of moving ahead of its competitors, like Westpac, and position itself to win new business in sectors like housing.

News of the joint venture comes after the Australian government announced it’s investing almost $1.2 billion AUD (£670 million) in its digital future as part of a strategy to transform the country into a modern and leading digital economy by 2030. The investment will see the formation of a National Artificial Intelligence Centre, support for digital skills training, an overhaul of myGov and supporting emerging aviation tech like drones.

Palantir Technologies, a controversial US software company, has officially been named as a supplier on the Crown Commercial Service (CCS) Back Office Software (BOS) Framework.

The BOS framework is a new £1.2 billion CCS contracting route for the provision of software across central government departments and all other public sector bodies, including local authorities, health, police, fire and rescue, education and devolved administrations.

It has to date been awarded to 31 suppliers, including companies such as Fujitsu, Oracle and SAP.

This will allow public sector organisations to purchase software subscriptions and licence support for back office systems directly from Palantir, in a deal that will run from 6 April 2021 to 10 October 2023, with the potential to extend for a further 18 months.

Palantir has been associated with a number of controversial deals in the past, including work with the US' Immigration and Customs Enforcement (ICE), and faced a parliament enquiry in 2018 over alleged meetings with the now-defunct data analysis firm Cambridge Analytica. It was also criticised in 2018 for picking up Project Maven, a contract with the Pentagon to develop drone technology powered by artificial intelligence, when it was dropped by Google following employee walk outs.

"This completes Palantir's relentless deployment into UK government and public institutions' operations," said Lucie Audibert, legal officer at Privacy International, a digital rights and privacy advocacy group.

"With public scrutiny of Palantir's products and services still lacking, and reports of Palantir's products having been used for human rights violations around the world, this is of great concern," she added. "And it comes just a couple weeks after the government conceded to seeking public consultation on Palantir's involvement with the NHS," she told IT Pro.

Privacy International believes Palantir has been discreetly building and extending its reach inside and across government departments without proper public scrutiny.

“We know they have contracts with various government departments, including the Cabinet Office, for development of the “Border Flow Tool”, a post-Brexit border management system,” said Audibert.

She said that this tool is meant to integrate a wide variety of government data, from the Home Office to the Department of Social Care, but the public is unaware of exactly what data that is.

“The privatisation of public responsibilities is deeply problematic when deployed without the oversight and safeguards required to ensure human rights are not quietly abused,” added Audibert.

Audibert added that public-private partnerships are dodging rules around transparency and accountability, as any disclosures and processes to which the public is entitled are limited and bypassed on account of “commercial interests confidentiality”.

Digital rights group Open Democracy has previously claimed the government was forced into a “major U-turn” on its NHS deal with Palantir. The company was awarded an “emergency” contract in March 2020 by the NHS to supposedly assist with the COVID pandemic’s handling, which Open Democracy said was “quietly” extended in December and gave it a bigger role in handling NHS data. The extension was reportedly for two years and affected other areas beyond COVID, such as Brexit.

Open Democracy launched a legal challenge in February, which it withdrew and claimed victory in April as it stated that the NHS had committed to not extending Palantir’s contract beyond COVID without consulting the public.

The NHS said the group had to drop the case as it “was apparent even to them that the NHS has always acted in accordance with its legal responsibilities”.

The UK government has made a "major U-turn" after facing a lawsuit from Open Democracy over a £23 million NHS data contract awarded to controversial US firm Palantir.

Open Democracy claimed that the government "finally caved" in the face of the lawsuit even though the case was withdrawn, in a blog post on Tuesday, adding that the NHS has now committed to not extending Palantir’s contract beyond COVID without consulting the public.

It has also agreed to engage the public, via patient juries, about whether firms like Palantir are appropriate for a long-term role in the NHS at all.

“It’s a major U-turn at a critical moment. The NHS, with its unique trove of structured health data, is powerfully attractive to tech corporations. Palantir and other US tech firms clearly stand to profit from managing or accessing this asset, estimated to be worth £10bn a year,” wrote Mary Fitzgerald, Open Democracy’s editor in chief and Cori Crider, founding director of FoxGlove.

“The NHS datastore is the largest pool of private health data in NHS history, and that raises questions too important to be settled in secret deals. Should it survive the pandemic? On what terms? Should Palantir manage it, or are there more trustworthy alternatives?”

Open Democracy stated its mission is not complete. It is still seeking full transparency on the Palantir deal, wants to know what data is being fed into Palantir’s datastore, and is also keen to ensure that the public consultation the government has promised is far-reaching, not just a “box-ticking exercise”.

Fitzgerald and Crider also wrote: “We also need to fully understand and debate Matt Hancock’s long-term plans for our NHS. Our case was launched amid a major proposed shake-up of laws governing the NHS and its patient data.

"Proposals in the government’s recent white paper for health and social care would centralise NHS control under Matt Hancock, and could prepare the ground for larger slices of the NHS – including its £10bn/year health data assets – to be contracted to private tech firms.”

However, in a statement provided to IT Pro, an NHS spokesperson disputed Open Democracy's claims and said the group "have had to drop their court case unilaterally as it was apparent even to them that the NHS has always acted in accordance with its legal responsibilities.

"They, therefore, stood no chance of succeeding in their completely spurious claim. It would be more honest if they actually came clean with their crowdfunders that far from 'winning' this case they had no choice but to drop it when they realised they hadn’t a leg to stand on," the spokesperson added.

Palantir was awarded an “emergency” contract in March 2020 by the NHS to supposedly assist in handling the COVID pandemic. However, openDemocracy accused the government of “quietly” giving the company a contract extension in December and a bigger role in handling NHS data. The extension was for two years and affected other areas beyond COVID, such as Brexit and more, according to the organisation.

Open Democracy stipulates that the government has a duty to consult the public and users of the NHS before signing major NHS contracts with big tech, which led to the organisation launching the legal challenge in February.

This article originally stated that Open Democracy had won a lawsuit against the NHS. This was incorrect and the article has been updated to reflect this.

Organisations might be drawn to cloud by the lure of gaining new information and insights from the data they hold. Perhaps they will learn how to retain customers more successfully, or get a better understanding of profit centres – or cost centres. The twin powers of artificial intelligence and machine learning (AI and ML), deployed in the cloud can work wonders. But they need their ‘food’, the raw data they work with, to be in tip top condition, and for many organisations preparing that food requires time, effort, and attention to detail.

Data strategies and clear priorities

There is no doubt that AI and ML can provide information that organisations can’t come by in any other way. Data-crunching can, if done well, help organisations boost productivity and profits. But optimal success doesn’t come overnight. As Yann Lepant, MD for Accenture Technology in the UK tells IT Pro: “It’s easy to get tempted by the extensive set of cloud based new data and AI technologies as it’s an exciting new playground to venture on. However, it’s also a place where it’s easy to consume wastefully and get lost in delivery.”

A key factor in avoiding the pitfalls is to have clear priorities and a solid data strategy. Lepant tells IT Pro that having these will mean “each initiative is done with purpose and outcome, contributing to a journey of maturation through the cloud”.

Ingrid Verschuren, head of data strategy at Dow Jones, whose global news database Factiva grows by a million articles every day, from approximately 33,000 sources, tells IT Pro: “It doesn’t matter how good the technology is if the data feeding it is poor quality. The first and most important step to get the most out of your data is to ensure you are using the right data – and that it is structured in a way that will answer the questions you want to ask.”

So what do clear priorities look like? Paul Clough, professor of search and analytics at the University of Sheffield tells IT Pro: “Identifying the use cases is critical and needs to be based on issues and problems identified in the business by business stakeholders, rather than by data scientists or the IT department.” This approach keeps the priorities very practically focused, he says.

Cleaning and refining

Another pair of related tasks that may need to be undertaken before AI and ML can work their magic are cleaning up old data and refining what new data is collected.

Part of the cleaning task will involve bringing data out of silos. This can be tricky, but worth the effort and organisations undertaking the task can take some comfort from the fact that they’re not alone. Often, data silos are a result of historical practice and how things have grown organically over time, and it takes a concerted effort to undo in a relatively short time what’s built up over a much longer period. But as Lepant points out, there are really no short-cuts. “From getting a 360 degree view of the customer to end-to-end supply chain management, via fraud prevention and intelligent forecasting, the list of business outcomes enabled by the removal of data silos is endless,” he says.

Cleaning data is another task that can take time, but the effort is definitely worth it in the longer term. Verschuren provides some useful advice to help keep the cleaning task focussed, telling IT Pro: “Before you start cleaning your data, you need to determine how that data will be used and what insights you need to generate. Ask yourself, what does perfect look like? From there, you need to define the data fields that will be part of your data set, and the input for each of those fields.” Effectively, work backwards from the outcomes you want to get, and determine what data you need to get them.

And finally, along come analytics and insights

With clean data, a clear data strategy in place, and perhaps some new data collection streams in place too, an organisation is finally in a position to start using AI and ML to gain those valuable insights it craves.

But that’s not quite the end of the story. To continue to gain insights over time, data should take its place, unsiloed, front and centre. Or, as Lepant tells IT Pro when sharing key advice, organisations should “implement a programme to update the ways of working, culture and data literacy to help the business become more data driven and self-served”.

The NHS is being taken to court over a data contract awarded to controversial US data firm Palantir.

The Open Democracy has announced the legal action after a contract extension was awarded to the US "spy-tech" company for long-term analysis of large quantities of public health data.

Palantir was awarded an "emergency" contract by the NHS in March to assist in handling the COVID pandemic. The firm was once funded by the CIA and is known for its involvement with defence projects and immigration agencies.

The Open Democracy is accusing the government of "quietly" giving Palntair a contract extension in December and a "major, long-term" role handling NHS data. The new two-year contract reaches beyond COVID and into areas like Brexit and more, the organisation said.

There is also a suggestion that this deal has been in the works for much longer. Palantir has spent years courting top UK and NHS officials over cocktail dinners in London, San Francisco and Davos, according to the Bureau of Investigative Journalism.

"Emails seen by the Bureau show that discussions between Palantir and NHS chiefs about how the company could work with patient data were underway throughout the second half of 2019, months before the pandemic hit - and indeed that by January 2020, Palantir's London team were already working on a product 'exclusively focused' on the UK's healthcare market," the Bureau states.

Health secretary Matt Hancock is already facing calls to resign from a number of politicians after a court ruled that his department had not published details of contracts awarded during the pandemic. The health secretary was found to have "breached legal obligations" over the spending of taxpayer money.

The Open Democracy said it had a duty to consult the public and users of the NHS before striking "massive deals which affect the future".

The group also pointed out that data protection impact assessments were critical in ensuing health information and human rights were protected, something they claim the government hadn't done with the Palantir contract.

"Health secretary Matt Hancock and his advisers must have known it wouldn't look good," the Open Democracy wrote.

In a statement given to IT Pro, an NHS spokesperson said: "The company is an accredited supplier to the UK public sector, the NHS completed a Data Protection Impact Assessment in April 2020, and an update will be published in due course."

Mindtree has announced it has achieved the Application Development Partner Specialisation in Google Cloud's Partner Advantage Programme.

The 'specialisation' is an acknowledgement of the cloud firm's expertise and success in building apps and services using the Google Cloud Platform.

Mindtree is an Indian-based tech with offices in Banglore and the US. The firm provides services for outsourcing, data analytics, e-commerce, mobile applications and cloud migrations.

Google Cloud's Partner Advantage Programme is designed to provide its customers with access to qualified companies that have demonstrated technical expertise and is designed for firms that show particular success in specialised service areas. For Mindtree, that specialism is in driving digital transformations with cloud migrations and application adoption.

"Mindtree is committed to helping enterprises grow and scale their business leveraging Google Cloud's world-class infrastructure and robust set of cloud solutions," said Radhakrishnan Rajagopalan, the global head of customer success, data and intelligence at Mindtree.

"This recognition instils further confidence in enterprises seeking to migrate their legacy applications and workloads onto Google Cloud that Mindtree can effectively help an organisation drive their cloud adoption initiatives forward."

Through its collaboration with Google Cloud, Mindtree has boosted its status as cloud migration and digital transformation specialists. For Google, it is further evidence of its cloud strategy to priorities migration and analytical services as it looks to cement its place as the third-biggest cloud provider.

While they cannot match the sheer breadth of services offered by AWS, other providers like Google and IBM have taken different approaches and sought to specialise in areas of cloud computing,

In 2020, Google Cloud acquired a number of analytical firms and also migration specialists, such as Cornerstone Technologies and Looker. In May 2020, Google also announced that Splunk services were available in beta, with a full rollout later in the year. That partnership was also aimed at providing fast and reliable analytical services to Google Cloud customers.

Google Cloud Platform has signed a multi-year agreement with Twitter that will see the social media company shift its offline analytics, data processing and machine learning workloads to Google’s Data Cloud.

The deal means Twitter will be able to more quickly process trillions of data points generated by every tweet, retweet and like send on its platform to generate insights that, in turn, can be fed into improvements to the core product.

Twitter's data platform currently consumes hundreds of petabytes of data and runs tens of thousands of tasks over a dozen data clusters each day. It previously struck an agreement with Google Cloud Platform as part of its ‘Partly Cloudy’ strategy in 2018, when the firm moved its Hadoop clusters to Google’s infrastructure.

With this expanded partnership, Twitter will adopt a number of Google services including BigQuery, Dataflow, Cloud Bigtable and machine learning tools. Ultimately, it’ll allow the firm to expand its data ecosystem and generate insights much faster, as well as allowing for deeper machine learning-driven innovation.

"Our initial partnership with Google Cloud has been successful and enabled us to enhance the productivity of our engineering teams," said Twitter’s CTO Parag Agrawal. “Building on this relationship and Google's technologies will allow us to learn more from our data, move faster and serve more relevant content to the people who use our service every day.

"As Twitter continues to scale, we're excited to partner with Google on more industry-leading technology innovation in the data and machine learning space.

Twitter is also hoping to ‘democratise’ data access by offering a range of data processing and machine learning tools to better understand and improve how Twitter features are used. Previously, engineers and data scientists developed large custom processing jobs, although these can now be queried faster using SQL in BigQuery.

The partnership, generally, will make it easier for both technical and non-technical teams to study data generated from the usage of Twitter, and gain insights from these.

"Helping customers manage the entire continuum of data - from storage to analytics to AI - is one of our key differentiators at Google Cloud," said Google Cloud CEO, Thomas Kurian.

"It's been phenomenal to watch this company grow over the years, and we're excited to partner with Twitter to innovate for the future and deliver the best experience possible for the people that use Twitter every day."