In the enterprise world in 2026, vector databases are increasingly under discussion in the same breath as semantic search, retrieval-augmented generation (RAG), and other ‘AI-ready’ ways of finding and using information inside an organization.

The basic driver is straightforward: teams want systems that can retrieve results based on meaning, and similarity, rather than relying solely on exact keywords, rigid schemas, or purely structured querying.

To understand the sector, ITPro spoke to Devin Pratt, research director at IDC and Noel Yuhanna, VP and principal analyst at Forrester, about what a vector database actually is, where it fits compared to other enterprise options, and what organizations should weigh up.

Vector databases have gone from a specialist concept to a practical talking point for IT teams in fairly rapid fashion, largely because so many modern AI projects now depend on finding the right information quickly and reliably.

What is a vector database for?

At its simplest, a vector database is a system designed to make “similarity” a first-class way of searching and retrieving data.

Instead of treating a query as a strict match, it returns items that are closest to what you asked for, based on how the underlying information is represented, making it particularly relevant for organizations dealing with large volumes of unstructured data – documents, support tickets, product catalogues, and so on – where the intent of a question matters.

“A vector database is designed to search by semantic meaning rather than exact matches, making it fundamentally different from relational databases, key-value stores, or search engines,” Yuhanna explains.

“Vector databases are built from the ground up to perform fast, large-scale similarity search using specialized indexing and vector embeddings,” he says. “It makes them especially suited for AI use cases like semantic search, RAG, and modern AI applications.”

How do vector databases work?

In practical terms, vector databases rest on the idea that content can be represented in a form that makes “closeness” measurable, so a system can return results that are similar to a query, even when the wording is different.

This setup is what enables meaning-based retrieval across messy, unstructured information, where a keyword search might miss useful context, or return too much noise.

The database’s job, then, is not just to store these representations, but to retrieve the nearest matches quickly enough to be useful in real applications. A major reason vector databases can do this at scale is indexing. As Pratt notes, vector databases “implement advanced indexing techniques like HNSW, PQ, or DiskANN to optimize similarity searches.”

The details vary by product, and that is where performance and cost trade-offs appear.

Yuhanna points out that “different products use a variety of indexing methods… which can have significant implications for query speed, accuracy, and efficiency depending on the workload.”

What can you use a vector database for?

Most organizations look at vector databases when they need better retrieval over unstructured information. The common goal is to help people and systems find the most relevant material even when the query does not share the same wording as the source.

In that sense, vector databases are often less about 'adding AI' in the abstract and more about making search, discovery, and information access work properly at scale.

A related capability is combining similarity search with more traditional constraints, such as filtering by department, document type, customer segment, date range, or permissions.

Yuhanna argues that “The ability to perform hybrid searches that combine vector similarity with non-vector data, such as master data or metadata, will become essential for most enterprise use cases.”

Vector databases are often compared to relational databases, key–value stores, and traditional search engines, but the more useful comparison is usually between a purpose-built vector database and a general-purpose platform with vector indexing.

In other words, the decision is not always “vector database or nothing” – it is whether vector search is central enough to justify adopting a dedicated system, or whether you can meet your needs within the database and operational model you already run.

Yuhanna is clear that a general-purpose approach can be sensible in the right circumstances: “A general-purpose database with a vector index is sufficient when vector search is secondary, data volumes are moderate, or the application needs to combine vector search with non-vector data to provide a broader context.”

Native or integrated vector databases?

In 2026, most organizations will encounter vector search in two broad forms.

The first is a purpose-built, native vector database, where similarity search is the core design goal. The second is an integrated approach, where vector indexing and query capabilities are added into a general-purpose platform, often to align data, governance, and operations.

In practice, the right choice tends to depend less on ideology and more on what you are building, as well as how central vector search is to the application.

Pratt draws a clear line between the two options, noting that in a native approach “the vector store is purpose-built and optimized specifically for managing high-dimensional vectors,” while integrated platforms are increasingly common because teams want consolidation.

“Integrated vector databases are the clear platform of choice for storing and querying vector embeddings, as organizations favor unified, governed environments for agentic AI,” he adds.

More broadly, Pratt explains that vector search is increasingly paired with RAG as an embedded function within mainstream database architectures.

Pratt also makes the trade-off more concrete. Integrated options can “embed vector stores within a broader framework that combines traditional structured data management with vector processing capabilities”.

Vector databases: Pros and cons

In simple terms, the case for vector databases is strongest when an organization needs more effective retrieval across unstructured information, and when “similarity” is a better fit than strict matching.

Implementing this can unlock experiences where search feels less brittle, and where applications can pull back the most relevant context even when users do not phrase a question in the same way the source content was written.

In that sense, vector databases are often less about replacing existing systems and more about adding a capability that traditional services struggle with.

However, the trade-offs tend to appear after deployment, when systems move from a proof of concept to a service that has to meet performance, security, and operational expectations.

“The biggest risks organizations face after deployment typically include performance bottlenecks at scale, declining search relevance, insufficient data security controls, operational complexity, skills shortages, and a lack of clarity about what vector databases can deliver,” Yuhanna warns.

From Pratt’s perspective, governance concerns are also front of mind.

“Security vulnerabilities (26%) and data privacy breaches (24%) are the leading concerns when deploying agentic AI,” he says, which makes it important to treat vector search as an enterprise data problem, not just an experiment

How to choose and deploy safely

Choosing a vector database is rarely a matter of picking the “best” product in the abstract. Yuhanna urges leaders to filter by “security controls, governance capabilities, TCO, hybrid search, scalability, performance, customization options, and search relevance”.

He also cautions against over-simplifying the category: “Common misconceptions include assuming that all vector databases scale equally, that all can perform vector and non-vector searches in a single query, or that they all use the same vector indexing algorithm.”

Leaders keen to implement vector databases can also fall into the trap of focusing too much on the technology and not enough on assessing their specific organizational needs.

Pratt acknowledges this and warns that vector capability is easy to over-simplify during procurement: “The most misunderstood point is treating ‘vector support’ as a checkbox rather than validating fit across governance needs, indexing requirements, and performance at target scale.”

In practice, reducing risk means treating deployment as an ongoing operational discipline, rather than a one-off build.

“Companies mitigate these risks by implementing stronger security controls, continuously monitoring vector performance, managing the lifecycle of vector data, selecting appropriate indexing strategies, and focusing on specific, relevant use cases that vectors can support,” Yuhanna adds.

Vector databases in 2026 and beyond

Over the next year and beyond, the most important change is likely to be less about a single breakthrough feature and more about convergence.

As mainstream database platforms add vector capabilities, and more teams try to run vector search alongside existing operational and analytical workloads, the category is starting to look less like a standalone niche and more like an expected part of the broader data stack.

Techniques such as database sharding will be used to partition datasets across nodes, for the faster, distributed similarity searches relied upon by tools such as AI agents.

Both experts predict a convergence of native and integrated vector databases, with the core motivation being better support for AI deployment. But even as the market converges, Yuhanna argues the practical success factors will look familiar to anyone who has put a database into production.

“Enterprise-grade security, governance, and compliance, along with platforms that can scale automatically and reliably, will be critical to succeed.”

As enterprises grow their operations and customer base, it can become more difficult to keep on top of the amount of data that needs to be stored and managed. In particular, the traditional database server can become a bottleneck when multiple users try to save, access, and retrieve data and in extreme circumstances the system might crash or deliver a compromised user experience.

Database sharding is the process of splitting a large database into multiple smaller, independent, and manageable databases distributed across multiple machines or nodes within an enterprise network. It exists to alleviate the problems that come with storing enterprise datasets, including big data.

"Picture a massive library split into separate rooms,” says Paul Done, field CTO Modernization at MongoDB, one of the largest database companies to implement sharding.

“Instead of cramming every book into one room, books are spread across different rooms. Finding what you need remains quick as the number of books increases over time, and there isn’t a single room that becomes overcrowded."

A typical database server stores data in multiple rows and columns. Each shard contains unique but interrelated rows and columns of data, which are always a subset of a larger database.

There are three components of shards:

• Logical shard: a set of data that has been systematically partitioned into smaller units - the shard we just defined.
• Physical shard: the actual hardware that “physically” holds logical shards is known as a physical shard.
• Shard key: a field that determines how data is divided and distributed across shards.

Business benefits of sharding

Databases can be scaled vertically and horizontally. Vertical scaling expands the capabilities of a single database server by using more memory, cloud storage, CPU power, and parallel processing. A vertically scaled database can handle large volumes of data.

Horizontal scalability: Sharding can horizontally scale databases. Adding more shards, volume data handling, memory load distribution, and higher transaction throughput support horizontal scaling for big data and intensive deep learning workloads.

Cost-cutting: Enterprises increase storage and computational power by spending a significant portion of their IT budgets. “For firms, database sharding is far more cost-effective than upgrading one large server, which at some point isn’t possible to go any further with,” Done explains.

Improved response time: Even if a single database server tries to store large chunks of data, it can process a limited number of queries and return small amounts of data. By splitting the database into smaller parts, queries need to search fewer rows and columns. Query response times improve drastically, making it easier for enterprises to retrieve data.

Fault tolerance: In monolithic databases, hardware malfunction, software downtime, or even planned maintenance would limit database operations and availability. It results in a single point of failure (SPOF), a total service outage that compromises business continuity. Data sharding is built on a shared-nothing architecture, where each shard is an independent data block. Failure of a shard’s host node does not impact the entire database, which allows the rest of the database to remain functional for enterprise operations.

How to perform database sharding

Database sharding isn’t a silver bullet, nor is it the method to use in every circumstance. “Sharding methods should be selected based on specific enterprise needs,” says Radoslaw Szulgo, senior product manager at database company Percona. In addition, the choice of shard keys affects the performance and scalability of database management systems.

Let’s consider a dataset: Customer_id, name, age, and city to understand sharding.

Dynamic sharding

Also known as range-based sharding, dynamic sharding splits database rows into a range of values, based on any field from the database. By setting a predefined range, data from the larger database is split into shards.

The system starts with fewer shards. As entries increase, multiple shards appear. When a single shard becomes too large or busy for user queries, it automatically splits into two or more.

For example, shard A contains all customers from 1 to 5000. When shard A becomes busy, it splits into two shards, A1 containing customer entries from 1 to 2500, and A2 containing customer entries from 2501 to 5000.

The automatic splitting property and quick implementation make dynamic sharding an ideal choice for enterprises in industries with fast-moving, large-scale information such as healthcare and finance.

Hashed sharding

Hashed sharding is also known as algorithmic sharding because it applies an algorithm, known as the hash function, to the shard key in each row. The value returned by the hash function determines the number of shards and the assignment of entries.

For example, let’s suppose that the customer ID is the shard key. The hashing function can be Customer_ID % 4.

There will be four shards, numbered 0, 1, 2, and 3. The table shows the assignment of customer IDs 1000 and 1004 into shard A and so on.

"The most common sharding method we’ve seen at enterprises is 'hash-based' sharding because it reliably offers uniform data distribution,” Szulgo says.

Geolocation sharding

Geolocation sharding, used by large-scale multinational corporations, considers location as the shard key. Physical shards are geo-located.

As data is stored near the customer location, latency and compliance risks remain low. Due to extensive use in certain regions, some shards can become overloaded while others in less active locations remain near empty. These ‘data hotspots’ limit the ability of sharded databases to perform load balancing and run fast queries.

To get around this, enterprises can further divide popular locations into distinct shards, or fine-tune the distribution of shard keys.

Shard keys must have high cardinality, which are unique values that facilitate even distribution of data. Another factor is the frequency of the shard key, which defines how often a value appears in a database. The frequency of a shard key must not be too high or low, but balanced.

Enterprise strategy: sharding with partitioning and replication

Data replication maintains identical copies across multiple physical nodes in the enterprise network. When data replication is implemented with sharding, each shard is a replica set so that if one shard becomes unavailable, the database remains functional. Due to replication, shards that run on multiple nodes are consistent with a similar schema or design. Hence, data can be retrieved from another shard.

Data partitioning is another database management strategy that divides data into manageable segments, known as partitions. The difference between partitioning and sharding is that partitioning runs on the same database server while shards run on different servers and machines. Enterprises implement partitioning for databases that handle a high volume of read operations, whereas sharding is used for write operations.

Enterprises implement partitioning for databases that handle a high volume of read operations, whereas sharding is used for write operations. Done properly, they can work in tandem to radically improve performance. But enterprises need to carefully consider their data architecture and implementation lest they accidentally overcomplicate systems and introduce data silos.

"Frequent queries and cross-system requests require intricate coordination, sometimes leading to ungoverned local copies and security risks,” explains Tom Peirson-Webber, VP Engineering at Harbr Data.

"The solution is architectural – centralizing access and governance while keeping storage distributed.”

Enterprises need to consider whether their database is large enough to be sharded. Big data, in the order of multiple terabytes and petabytes, must be sharded because vertical scaling would exhaust bandwidth and increase latency. The overhead of managing multiple complex database partitions would burn CapEX.

Database sharding can lower latency radically, but the architecture needs a router to check multiple shards to return queries. Higher volume may burden routers in communicating with multiple shards and collecting data. This results in query overload and increased workload on a single physical node.

While growing databases call for high computational power, operational overhead, and increased costs, database sharding improves data storage patterns and contributes to effective big data management.

UK channel partners are prioritizing data-driven innovation in their quest for new growth in 2026, according to the latest research from tech distribution giant Westcon-Comstor.

As part of its Future Ready program, the company recently quizzed 500 IT strategy decision makers and influencers at partner organizations in the UK, Spain, UAE, Australia, and Singapore on their latest channel priorities.

The results revealed that almost half (47%) of UK firms plan to invest in data-led offerings to help drive new opportunities and increase profitability over the coming year, outpacing the global average of 40%.

Additionally, 24% of UK respondents view data-driven consulting services as the single most promising revenue stream arising from digital transformation.

This heightened focus on data edged out other priorities such as market expansion and pivoting to managed services or subscription models, the distributor revealed.

"UK channel partners are leading the charge on data-driven innovation, with nearly half prioritising analytics to fuel growth in 2026," commented Rene Klein, Westcon-Comstor's executive vice president for Europe. "It's a clear sign that the UK market is setting the pace for smarter, insight-led strategies that unlock new opportunities and deliver real competitive advantage."

Real-time data

Across all five countries surveyed, a majority of partners were found to already be leveraging real-time data to drive customer experience improvements and inform their business decisions.

Top applications include tracking customer behaviour and preferences (57%), refining marketing and sales strategies (50%), followed by identifying market trends (42%).

Westcon-Comstor said the findings coincide with increased adoption of its own analytics-driven sales program, Intelligent Demand (ID), which uses predictive analytics to drive growth, customer value improvements, as well as scope new markets for partners and vendors.

During 2025, Intelligent Demand supported more than 1,200 channel campaigns – up from 540 the previous year and marking a 132% year-on-year increase.

'Critical driver'

Commenting on the findings, Atul Damani, Westcon-Comstor's chief data officer said that data-driven innovation has become "embedded as a critical driver" of AI growth and differentiation across the channel.

"In the era of lifecycle selling, leveraging AI-powered data and analytics can enable partners to save time, uncover highly targeted sales opportunities, and focus their energy on where they'll get results," he explained.

"As channel evolution accelerates, the importance of data will only grow, and at Westcon-Comstor, we feel distribution has a vital role to play in ensuring partners are equipped with the insights and capabilities they need to thrive."

IBM predicts it will reach “quantum advantage” in the next 12 months, meaning the time at which quantum algorithms can begin to complement traditional computing for widespread benefits.

The technology giant has been working on quantum computing for decades, but in recent years has signaled that quantum computers that can deliver tangible benefits to businesses are just around the corner.

Further, the firm predicts return on investment (ROI) in quantum across select industries within the next three to five years.

For example, HSBC has partnered with IBM to develop a new machine learning (ML) model for trades, with the intention of improving its margin via better predictions of buying and selling prices.

IBM took a year’s worth of HSBC’s data and applied a quantum algorithm for ‘feature selection’, a process that highlights the most relevant aspects of a dataset to improve model performance.

In tests, IBM and HSBC tracked a 34% increase in successful trades.

At a briefing on the breakthroughs, Adam Hammond, business leader at IBM Quantum EMEA, told assembled media that the model is not yet in production and still needs more training.

Nevertheless, it acts as a tangible example of how quantum algorithms could improve business performance. Outside of banking, Hammond said IBM is hoping to apply quantum computing to a wide range of sectors.

His team is working with Moderna on mRNA vaccines, the Wellcome Sanger Institute on genomics, Boeing on surface chemistry, and Bosch on superconducting hardware, as well as organizations in the farming, life sciences, and electric vehicles sectors.

A seismic shift in compute complexity

At the center of IBM’s latest quantum roadmap is Nighthawk, its new quantum processor which can process up to 120 qubits linked together for a 30% overall increase in complexity.

“One of the key factors in getting to quantum advantage and to useful quantum is being able to fit the size of your problem into the capabilities of the machine that's running it,” Hammond explained.

IBM measures quantum performance in ‘operations per circuit’ (OPC), with today’s chips capable of 5,000 OPC. The Nighthawk chip will use these 5,000 operations more efficiently, Hammond explained, providing a boost in the complexity of tasks that Nighthawk can complete.

On a technical level, IBM has unlocked this capability by moving from a chip layout in which each qubit was entangled with two or three others to one where each qubit is reliably entangled with four qubits.

Hammond noted that this new configuration could lead to breakthroughs in chemistry in particular, as its lattice-like structure mirrors the makeup of the chemical world.

By 2028, IBM believes it will be able to reach 15,000 OPC, tackling problems three times larger than today’s. The next year, in 2029, it is targeting full quantum fault tolerance, at which point its quantum systems will be able to produce reliable results.

The company said it will achieve this with its new large-scale Starling quantum computer, powered by its Loon chip, which will represent a seismic leap forward for the quantum sector.

It’s aiming for this system to run at 100 million OPC, the computational equivalent of one quindecillion of the world’s most powerful supercomputers – a one followed by 48 zeroes.

When ITPro asked Hammond to explain what this would mean for the 2030s, he explained that it was incredibly difficult to predict the future.

“Three years ago, the largest machine we had was 27 qubits – you can simulate a 27 qubit machine on your laptop,” he said.

“It's only when you get above 100 qubits that actually you can't even simulate that on the largest supercomputer in the world. In fact, actually, if you network them all together, you wouldn't be able to simulate it.

“So the hardware has made a really dramatic increase in capability in the last three years. Three years ago, I wasn't talking about real use cases with clients. We were talking about doing experimentation. Now we're really actively exploring real use cases in preparation for machines being large enough to run those in production.”

Despite this, Hammond was resolute that classical computers and AI algorithms will continue to play a huge role in the future of computing. This is because quantum computers are best suited to complex, linear equations – and worse at tasks that today’s computers excel at.

“So in the same way a GPU solves matrix algebra, what a quantum computer does is it solves complex sets of linear equations – and it so happens that that's something that classical computers really aren't very good at doing,” he said.

“So optimization tasks actually simulating nature at a sort of a molecular level, is a complex set of linear equations, linear algebra, if you're doing computational fluid dynamics. So I think what will happen is that some of those tasks that we currently use super computers to do today may well get offloaded to quantum processes.

“But that's all quantum machines do. You're never going to have Excel running on a quantum processor – you might have a particular bit of maths in your Excel spreadsheet that a quantum processor can help you with, but they're never going to replace it.”

Going forward, IBM is also looking at how these complex quantum algorithms can be integrated in existing enterprise architectures.

Hammond said in the case of HSBC, it would need to consider how its machine learning model for trading could be made to follow its checks for security, privacy, performance, and resilience.

MORE FROM ITPRO

• Understanding quantum-safe AI and how to create resilient defenses
• Preparing for the quantum revolution
• Post-quantum cryptography is now top of mind for security leaders

SAP and Google Cloud have expanded their partnership to include a comprehensive new open data offering, designed to simplify data landscapes and help enterprises view their entire data estates in real time.

The new integration will help customers build an end-to-end data cloud that pulls together data from across the enterprise landscape using both SAP Datasphere and Google’s data cloud, without the risk of duplication.

By combining critical business data – such as supply chain information, financial forecasting, and human resources records - with non-SAP data from Google Cloud, organizations can leverage a “fully-defined data foundation”, the long-time tech collaborators said.

In an announcement, Christian Klein, CEO and member of the executive board of SAP SE, said the development introduces “entirely new opportunities” for organizations to “derive more value from their full data footprints”. 

“SAP and Google Cloud share a commitment to open data and our extended partnership will help break down barriers between data stored in disparate systems, databases, and environments,” he said.

“Our customers not only benefit from the business AI already built into our systems, but also from a unified data foundation.”

This disparate business data is pulled together using SAP Datasphere and sent to Google Cloud’s fully managed, serverless data warehouse offering BigQuery, where users can perform further analysis.

That includes actioning queries to blend data from both SAP and non-SAP software, eliminating common data siloes from various sources across the likes of marketing, sales, finance, and the supply chain.

Additionally, users can generate trusted insights using Google Cloud’s artificial intelligence (AI) and machine learning (ML) services to train data models from both SAP and non-SAP systems, as well as perform advanced analysis to explore financial and business outcomes.

“SAP and Google Cloud now offer an incredibly comprehensive and open data cloud, providing a foundation for the future of enterprise AI,” said Thomas Kurian, Google Cloud CEO. “Few resources are as important to digital transformation as data. 

“By deeply integrating SAP data and systems with our data cloud, customers will be able to utilize our analytics capabilities, as well as advanced AI tools and large language models to find new insights from their data.”

The companies also revealed they plan to partner on joint go-to-market initiatives for enterprises’ largest data projects, enabling customers to adopt data products from both SAP and Google Cloud.

Less than half of data and analytics (D&A) leaders believe their team is effectively delivering value to their organisation.

A survey of more than 500 D&A leaders globally found that just 44% reported their teams are unlocking tangible long-term benefits for their firms despite rising budgets and growing expertise.

D&A budgets have risen steadily as organisations seek to deliver greater value from data, streamline operations, and enhance operational agility. The mean D&A budget now stands at $5.41 million (£4.4 million), the research from Gartner found, while 44% of D&A teams also increased in size over the last year.

However, while rising budgets reflect increasing confidence in the data function, this is placing additional pressure to deliver impactful results, resulting in a negative impact on both teams and leaders, said Donna Medeiros, senior director analyst at Gartner.

“The demands being placed upon D&A, as well as increased investment, reflect a growing confidence in CDAO’s abilities and recognition of the data office as an indispensable business function,” she said.

“However, this leads to more work as pressure grows for D&A to achieve tangible business results.”

Gartner’s study found that D&A functions are receiving increased investment across a range of areas. Nearly two-thirds (65%) of leaders reported increased investment in data management while 63% and 60% reported investment in data governance and advanced analytics capabilities respectively.

Growing responsibilities present challenges

Chief data and analytics officers (CDAOs) now find themselves tasked with an expanding range of responsibilities within their organisation, Gartner found.

60% said they are now responsible for defining and implementing data strategies, with 59% also taking responsibility for oversight of said strategies.

Management of data-driven culture change was also a key responsibility highlighted by 54% of respondents.

This widening array of responsibilities means that some CDAOs are struggling to meet increasingly complex leadership demands, however.

Leaders that committed time to their own professional development were found to be far more effective than those who did not, highlighting the critical need for continual learning and evolution to accommodate the changing demands of the role.

Nearly half (43%) of “top-performing” D&A leaders reported increased effectiveness in both their own ability and the broader team by committing time to professional development, compared with just 19% of “low performers”.

“Successful CDAOs must be elite leaders,” said Alan Duncan, distinguished VP analyst at Gartner. “Top-performing CDAOs invest in their success by developing skills to thrive in ambiguous circumstances, articulate compelling value stories and identify D&A products, and services that can drive business impact.”

Human-related challenges to success

Difficulty meeting the complexity of demands placed on D&A teams is being exacerbated by an acute shortage of talent, Gartner noted.

More than one-third (39%) said that a lack of available talent had become a major impediment to success in the data function.

In fact, the top six key roadblocks for D&A leaders were all human-related challenges, including poor data literacy across the broader organisation and culture change acceptance.

Automation can enable greater efficiencies for organisations, with streamlined processes and better experiences for both users and customers. On the journey to digital improvements, there are a few business options that offer different advantages, benefits, and drawbacks.

This guide is designed to help purchasers of robotic process automation (RPA) technology make a more informed decision. It explains the benefits and limitations of RPA, where it fits in the automation landscape and how to choose the right level of RPA.

Read this guide for tips for choosing the right RPA provider and how to get started.

Provided by

Sustainability has a become a business imperative for organisations globally, with business leaders increasingly needing comprehensive data intelligence suites that can address sustainability issues more holistically.

This recent IDC paper examines how sophisticated technology trends can process data from various sources, curate it, and translate it into usable formats that meet the requirements for standardised, quantitative, metrics-driven ESG information.

Download now to realise the full value of sustainable intelligence solutions.

Provided by

Research from the Enterprise Strategy Group (ESG) reveals that IT professionals struggle to identify the most valuable data available to them, so priority investments include the centralisation of data with increased security controls, as well as comprehensive analytics to identify complex attacks. As a solution, organisations are investing in SIEM infrastructure but admit they find it difficult to manage it effectively.

In this technical product review of IBM Security QRadar SIEM, ESG makes several observations, including QRadar's ability to prioritise events, lower costs, integrate with both IBM and 3rd party solutions, and more.

Download the full report for more details and how to overcome the most challenging attributes of SIEM for your business.

Provided by

The data integration tools market is seeing renewed momentum driven by requirements for multi-cloud and hybrid data integration and data fabric design patterns. Data and analytics leaders should use this research to evaluate suitable vendors for their existing and upcoming data integration use cases.

This whitepaper evaluates the core capabilities and functional requirements of current data integration tool vendors, ensuring they can offer organisations consistent access and delivery of data across a variety of data sources.

Download now to learn how the vendors measure up and why IBM is considered a leader in this field.

Provided by

Cloud computing firm Appian has championed its newly-released Appian Data Fabric system, claiming the system's real-time read/write feature offers capabilities that aren't offered by any competing vendor.

In the keynote that opened its annual Appian Europe conference, the system, which launched as part of Appian 22.4 last week, was the central talking point. The solution combines the automation and low-code development for which Appian is known across a single virtual data model, allowing customers to access unified data and identify areas in which to implement systems such as robotic process automation (RPA).

Appian insisted that while rival data fabric systems are available, none can offer its ease of adoption. The firm said 45% of its customers have already leveraged Data Sync, a process for caching source data that is at the heart of the data fabric system, and stated that for all its benefits Appian Data Fabric is no step up from these existing systems.

The workloads, automations, and queries set up by customers already laid the groundwork for Appian Data Fabric, as metadata from all of this can be used to optimise data management within the platform.

For example, by analysing how a customer builds its processes, what it automates, and which queries that analysis will require, Appian’s data fabric can automatically repartition and re-index data within its virtual model to ensure it can be accessed with maximum efficiency.

“When other clients say ‘oh yeah, we have a data fabric,’ what they're talking about is probably much more like a data lake,” said Michael Beckley, CTO and a founder at Appian.

“They might be able to analyse data from different locations. But being able to write transactions to the fabric in real-time, and read from it, that’s what the Appian data service can do, that no one else can.”

In his keynote address, Beckley also stressed that the data fabric is an improvement to the system that customers already use rather than a new product, and comes with no new training requirements or added cost.

He also stated that the company sees Data Fabric as an alternative to rip and replace models, or any vendor lock-in, giving the example of a data fabric that connects Oracle, Microsoft, and Salesforce alongside the Appian database.

“The answer is not just Appian. The answer is using Appian to take advantage of the technology investments you already have. And with Appian's data fabric, we're confident you'll be able to do that.”

A data fabric is a data management architecture that integrates multiple data pipelines or cloud environments in an end-to-end data environment, allowing customers to access their data through an optimised access point that requires little to no code.

The firm claimed that Appian Data Fabric has doubled search speeds, requires customers to use 75% less code to build charts, and allows zero-code apps to be built up to ten times faster than previously possible.

Last year, Gartner named data fabric as a key technology trend for 2022, noting its potential for adding to existing infrastructure. Since then, a number of companies have offered data fabric systems, such as Google Cloud.

Down the line, Appian said its existing data system could also be used to measure parameters such as carbon impact across a number of silos.

Gartner recently published findings that marked sustainability as a key technology trend for 2023, and that businesses investing in it are doing so to tackle disruption, and reap its short and long-benefits.

Malcolm Ross, deputy CTO at Appian, told IT Pro that regulatory pressure increases in a recession, and that environmental, social, and corporate governance (ESG) will be core to this.

“ESG is definitely the next GDPR-level type of regulatory oversight that companies will have to report their carbon footprint. They will have to report how their governance modelling, their board, is accurately collecting the culture of that organisation or that country as well.”

Ross also stated that efficiency savings through efforts such as automating the back office are unlikely to provide meaningful results with sustainability in mind. He argues that only holistic oversight, allowing for proper identification of weaknesses through processes such as big data analytics, will be necessary.

“You're not going to become better at ESG and sustainability, by just doing automation what you need is visibility.”

Head of process mining Karina Buschsieweke told IT Pro that Appian’s data fabric is already capable of providing this visibility, and that growing regulatory pressures might lead to more specific focus on this process down the line.

“How process mining works is, you have attribute data, so you have a process and you can add as many attributes to that as you like, so can be location, product, etc.," said Buschsieweke

"But it could also be CO2 as a further data attribute. Now, of course, today the major challenge for all companies is where to get the CO2 impact data from, because that's also a whole different issue - how to standardise it, and make it comparable.”

“However, if you have the data already, you could do this with our solution today. And then probably, there could be features that might even make it easier for customers than this. I think at some point in the future, companies will be required to report on CO2 the same way as they have to report their euros.”

Backup, recovery and data management solutions provider Veeam has been selected as the data protection provider of the new Lenovo TruScale backup as a service (BaaS) platform.

Veeam becomes the first data protection solution added to TruScale and will provide Lenovo users with ‘pay as you go’ BaaS via Lenovo’s infrastructure as a service (IaaS) platform.

Announcing the move, the firm said the solution aims to modernise enterprises’ data protection strategy, help manage growing data volumes, tackle increasing performance and multi-workload demands, as well as lowering total cost of ownership (TCO) with a flexible cost model.

“As a leader of modern data protection, we’re proud to be a part of Lenovo TruScale to deliver expert-built and managed BaaS and disaster recovery as a service (DRaaS),” said John Jester, chief revenue officer at Veeam.

“This new Veeam and Lenovo solution will help enterprises reduce the complexity of modernising their data protection strategy from legacy solutions and ensure confidence in the protection of their data.

“They can focus on what matters most – their customers, innovation, and strategic initiatives to propel their business forward.”

The new collaboration builds on the companies’ existing relationship, which has already seen the pair offer intelligent data management solutions for Lenovo software-defined infrastructures since 2020.

This enterprise-level data protection now combines Lenovo ThinkSystem Servers and its DE and DM storage series with Veeam’s backup and replication functionality to maximise availability and simplicity for customers.

Optimised for daily IT requirements, the offering promises reliable recovery and reduced management time, digital resiliency for rapid recoveries, data sovereignty, as well as protection for the data centre, edge, and cloud.

General availability of Lenovo TruScale backup as a service with Veeam is expected to arrive in 2023.

“Lenovo TruScale delivers the devices, infrastructure, services and solutions, from pocket to cloud, that our customers need, with the simplicity and flexibility of a scalable consumption model,” said Dale Aultman, VP and GM, TruScale and managed services, solutions, and services group at Lenovo.

“Veeam is the first data protection partner being added to TruScale for data management portfolio, allowing customers to control their data while letting Lenovo manage the infrastructure. This partnership provides customers with robust and reliable data protection and data loss prevention, future-proofing their businesses as they digitalise.”

Fujitsu and a Japanese university research institute have published research on the concept of a 'Trustable Internet', which aims to make it easier to verify the authenticity of data on the internet.

The Japan-based tech company teamed up with the Keio Research Institute at Keio University to publish a whitepaper on the Trustable Internet concept. The companies said that the verification of large amounts of information on the Internet remains a difficult task, highlighting the need for a more general-purpose method to prove the authenticity of data.

The pair's proposal would aim to help verify the authenticity of data by adding an interface, which they have dubbed the 'Endorsement Layer', which would theoretically help users confirm the credibility of various data based on additional information related to the generation and processing of data.

This includes information on confirmation and evaluation by third parties that have viewed the data, and information on physical spaces related to the data including from sensors, for example.

“This enables users to judge the credibility of data from a variety of perspectives to ultimately prevent the use of unverified data and the spread, unintentional or otherwise, of false and unverified information ('fake news'),” said the companies in the research.

An example that the companies gave in a video was a person seeing a river swollen by heavy rain and consequently posting on social media that a flood is about to occur.

The companies said the message could be misinterpreted that a flood is about to occur, or has occurred, generating anxiety among the general public as well as triggering social and economic confusion.

If the companies’ technology was applied in this case, it could inform the users reading the message about the source of the data and the process by which it was generated, as well as extra information like the actual water level of the river, collected from sensors, or the fact that no evacuation is needed since no message has been issued from the authorities.

“In this way, a chain of verifiable data is established, reinforcing the certainty of the data,” said the narrator in the video. “The confirmed data is then presented to the viewer in a way that is easy to be understood. Viewers can compare the posted message to the confirmed data and act with common confidence without confusion.”

The two companies suggested the proposed Endorsement Layer should sit between the existing 'Internet Layer' and the 'Web/App' layer. This configuration can be implemented without impacting the existing internet, they said, and allows users to use the internet and applications as usual.

If necessary, they could also obtain additional information to verify the authenticity of data and information.

The additional information can be used to verify the authenticity of data and can be provided by people and devices at the time of data generation, including information on the name, affiliation, and qualifications of the creator of the data, such as the person or company, or the device used, place, and date and time.

Extra information also includes verifications and evaluation from third parties such as independent experts and information obtained from physical spaces including sensor measurements that can be added after the data is generated.

Users would be able to search and check additional information necessary to verify the authenticity of data on the internet in the Endorsement Layer, where information would be stored as graph data.

Through this, the companies hope users would be able to evaluate data authenticity from a different perspective, to make data-driven decisions more accurate, and help to prevent the spread of unverified data.

“By providing a mechanism that enables users to verify and judge the reliability of data available on the internet from various perspectives, we contribute to the health of society and economic activities,” the companies stated. “This is what we are aiming for a trustable internet that people can use information securely.”

Gartner Peer Insights is a free peer review and ratings platform designed for enterprise software and services decision makers. Reviews go through a strict validation and moderation process in an effort to ensure they are authentic.

The “Voice of the Customer” is a document that synthesizes Gartner Peer Insights’ reviews into insights for IT decision makers. This aggregated peer perspective, along with the individual detailed reviews, is complementary to Gartner expert research and can play a key role in your buying process, as it focuses on direct peer experiences of implementing and operating a solution.

In this document, only vendors with 20 or more eligible published reviews during the specified 18-month submission period are included. Reviews from end users of companies with less than $50M in revenue are excluded from this methodology.

Provided by

The Master Data Management (MDM) market is evolving to meet the needs of a broader range of stakeholders who are seeking the business agility afforded by a single, trusted, enterprisewide view of master data. Data and analytics leaders should use this research to inform their MDM solution choices.

Gartner defines MDM as a technology-enabled business discipline in which business and IT work together to ensure the uniformity, accuracy, stewardship, governance, semantic consistency and accountability of an enterprise’s official shared master data assets.

Download this whitepaper to discover the pros and cons of the current vendors in the MDM solutions market.

Provided by

Organisations deploy a master data management (MDM) strategy to gain a single, trusted source of reliable information to deliver business value. Until recently, those seeking an MDM solution had to choose between a fullfeatured on-premises master data management platform or a more limited cloud-based solution.

For the agility to deliver results at the speed of innovation, you need an all-in-one MDM solution with the robust capabilities to meet all your end to-end requirements in the cloud.

This eBook describes why organizations are increasingly looking to move their MDM strategy to the cloud, and what to look for in a cloud based solution that can successfully support your MDM strategy.

Provided by

Data quality has traditionally been mandated to fulfill compliance and governance requirements and to reduce operational risks and costs. It's a competitive advantage that data and analytics (D&A) leaders must continuously engage with in order to achieve those goals.

The data quality management practice has been maturing in recent years and the vendor landscape is focused on addressing many of the market requirements. It covers much more than just technology. It includes workflow, roles, collaboration and processes (such as those for monitoring, reporting and remediating data quality issues).

Download this paper to learn why evaluating and selecting data quality solutions is much less of a specialized IT task than it was formerly. It now requires greater collaboration with business leaders and users.

Provided by

IT infrastructure services provider Kyndryl has announced a new strategic partnership with Teradata, a multi-cloud data platform for enterprise analytics.

The collaboration aims to help customers apply artificial intelligence (AI) and data modernisation techniques across their IT environments to boost the performance of various projects and speed up time-to-value.

Kyndryl’s data and AI services, as well as its infrastructure management tools, will be paired with Teradata’s high-performance cloud analytics and data platform in a move the pair claim will help customers migrate their on-premises data warehouses to the cloud with hyperscaler platforms.

In an announcement, the companies said they will help customers harness AI to gain better insights from their hybrid and multi-cloud data environments by designing, enabling, and supporting enterprise data modernisation solutions.

“We are thrilled to partner with Teradata who is a trusted and valued technology partner to many customers, providing a best-in-class analytics and data platform for enterprises across the globe,” said Nicolas Sekkaki, applications, data and AI global practice leader at Kyndryl.

“We look forward to deepening our partnership so we can deliver greater value and results that help companies transform and compete in their industries and around the world.”

London-based Teradata has created VantageCloud, a connected multi-cloud analytics platform for the enterprise, designed to help businesses leverage data and analytics to drive performance.

The firm says this new strategic partnership will help customers to leverage Kyndryl’s expertise – which includes advisory, implementation, and managed services – to get the most out of the platform.

There will also be a focus on delivering industry-specific analytics services geared toward customer modernisation challenges across key verticals such as manufacturing, communications, financial services, retail and CPG industries.

“This joint expertise will enable our customers to drive more actionable insights and higher value from their data through advanced analytics and AI/ML initiatives,” said Lisa Stewart, SVP of worldwide partners and alliances at Teradata.

Formed just last year as a spin-off from IBM, IT infrastructure services provider Kyndryl designs, builds, and manages complex mission-critical information systems for customers across more than 60 countries.

The firm’s partnership with Teradata is the latest in a string of collaborations struck since it went solo at the tail end of 2021. Recent deals include a hybrid cloud and edge partnership with Lenovo, a cyber resilience collaboration with Dell, as well as a delivery partnership with Oracle, which was announced back in June.

The company also has agreements in place with the three largest cloud providers, AWS, Google Cloud and Microsoft.

Salesforce has announced the arrival of Salesforce Genie, its new hyperscale data platform that creates real-time customer profiles to help businesses deliver personalized experiences.

Detailing the platform at its Dreamforce ’22 event, Salesforce said Genie powers the entire Customer 360 CRM platform to provide a “more automated, intelligent, and real-time” experience.

Complete with its data lake, the platform is designed to help businesses deliver highly personalized experiences across sales, services, marketing, and commerce, that continually adapt to customer needs in real-time.

“Every business leader wants to take advantage of real-time data to create compelling, personalized customer experiences — milliseconds matter in this new digital-first world,” commented David Schmaier, President and Chief Product Officer, Salesforce.

“That’s why we built Genie, our most significant innovation ever on the Salesforce Platform. Genie makes every part of Customer 360 more automated, intelligent, and real-time.”

Genie gathers and stores real-time data streams at scale and combines them with Salesforce transactional data. It does this using built-in connectors that bring in data from across all channels – including mobile, web, and APIs – as well as legacy data through MuleSoft and historical data from proprietary data lakes.

It then transforms this data into a real-time customer graph which acts as a unified customer profile record. Everything in this graph is visible and actionable across the entire Customer 360, industry solution, AppExchange, and customer apps.

Salesforce says the offering enables Einstein AI and Flow automation services to create more responsive and dynamic actions and engagement based on real-time data. The platform also runs on the firm’s public cloud infrastructure Hyperforce, which provides data residency, security, privacy, and regulatory compliance controls.

Partnerships also expand Genie’s abilities, Salesforce added. These include real-time data sharing between Salesforce and Snowflake, tailored AI model building with Amazon SageMaker, first-party advertising with the likes of Amazon Ads or Meta, as well as a new Genie collection on AppExchange.

Penny Gillespie, VP and analyst at Gartner, said Genie has the potential to bring real-time personalization across the entire enterprise.

“It harmonizes disparate data with privacy consent and compliance, automation, intelligence (Einstein), and speed,” she said. “It enables the discernment of millions of records to have one unified customer profile which is neither easily achieved nor highly available today with suggested real-time actions—be it a presentation layer change for a digital experience or coaching a sales or service employee.”

With the ability to know and discern every aspect of what a customer is trying to achieve, Gillespie added that the days of asking customers a lot of questions to help them may now be surpassed.

“Coupling the power of customer knowledge with AI has the power to solve the customer’s problem before the customer has even framed the problem—regardless of whether the area is service, sales, etc,” she explained.

“The result is truly effortless customer experiences which we know foster customer satisfaction, loyalty, and repurchase.”

Salesforce’s Customer Data Platform, which is powered by Genie, is available now, alongside Genie for Industries, Tableau + CDP Connector, and MuleSoft Anypoint Connector. Further integrations will be released from December and into 2023, the company said.

71% of CIOs say that the increasing amount of data resulting from cloud-native technology stacks exceeds human management capabilities, according to a new study.

According to the results, CIOs are increasingly concerned about the strain that managing technology stacks puts on their team members, and advise strategy change now before cloud complexity overwhelms teams entirely.

Software intelligence and digital transformation firm Dynatrace announced the results after a survey of 1,303 senior IT professionals, including CIOs and IT operations managers, across five continents. The survey indicates that complexity is already at unmanageable levels, with the average technology stack relying on seven separate platforms and systems, and each transaction involving an average of 35 different technologies from end to end.

Lack of overall insight into issues within the technology stack stood out among IT professionals as a problem arising from the scale of stacks. 97% of organisations state that obstacles are preventing them from achieving full-stack observability, or from adequately expanding security across their multi-cloud environments. On average, surveyed organisations had end-to-end observability of just 9% of their cloud environment.

Pressures of the coronavirus pandemic led to the widespread acceleration of digital services across the UK and the world, in both the private and public sectors. This has led to many benefits, particularly for small and medium-sized businesses (SMBs) whose transformation allowed for far greater predictive analytics.

But respondents also identified the rapidly changing nature of their environment as a source of strain, with 77% of CIOs indicating that their IT environment changes every minute or less. Amongst UK CIOs, this figure was even higher, at 93%.

The general lack of observability, the results make clear, is having a large knock-on effect on organisations, particularly over their ability to properly carry out and utilise big data analytics. Over half (57%) of the surveyed organisations state that data is frequently unavailable for analytics on demand, and just under two-thirds (63%) point to rehydration delays, in which data is located and decompressed for use, as preventing businesses from unlocking the value of their data.

Many respondents indicated that to improve efficiency and utilise team skills more effectively, a far greater level of back-end automation is required. 59% of CIOs state that without increased automation, their respective teams could become overwhelmed by the sheer scale and complexity of their stack, and the majority of CIOs (93%) argue that artificial intelligence in the form of AIOPs is becoming ever more integral to staving off burnout.

IT teams indicated that 40% of their time is currently spent simply “keeping the lights on”, and that automation will allow this time to be put to better use elsewhere. To use automation at its limitless potential, however, CIOs also indicate that an overhaul of data repositories is required. Almost half (47%) of surveyed CIOs want to see more adoption of the data lakehouse, a relatively new repository model that combines data warehousing and data lake techniques to allow for greater searchability through raw data that requires far less DevOp time to be spent on manual retrieval or rehydration.

“Multicloud and cloud-native architectures are critical to helping organizations achieve their digital transformation goals,” stated Bernd Greifeneder, Dynatrace founder and CTO.

“While organizations benefit from the flexibility and scale that these technologies bring, the explosion of observability and security data they produce is increasingly hard to manage and analyze.

“Existing tools – and there can be dozens of them – keep data in silos, making it difficult and expensive to unlock insights when organizations need them. As a result, they struggle to achieve the highest standards for security and performance across their digital services.”

Oracle has entered into an agreement with TikTok to receive the entirety of the social media firm's US user data, as well as perform data management and auditing services for the company.

Under the arrangement, first reported on by Axios, all US TikTok data will be sent through Oracle’s cloud infrastructure, with an eventual goal for TikTok to delete US user data from all its servers and fully migrate this to Oracle cloud servers.

Oracle will also be given access to TikTok’s content algorithms, as well as conduct audits on both manual and automated content moderation systems. With over 80 million users in the US, the deal represents a serious undertaking for Oracle.

Oracle regularly audits data, but governance on this scale is unusual. TikTok’s stated goal to hand all US data to Oracle’s servers and continually work with the cloud giant to improve its data signals a willingness by the company to more firmly situate itself in the US tech sector, and move away from its central hosting in Singapore.

The deal comes almost two years after TikTok was forced to enter discussions with Oracle after former President Trump mandated the company to consider divestment options or face a total ban in the US over national security concerns.

TikTok and its parent company ByteDance have been the subject of controversy in recent months. In June, the FCC commissioner publicly urged Apple and Google to delist the app from app stores, citing concerns over TikTok’s access to sensitive data, while in July Buzzfeed News alleged that China-based ByteDance employees have accessed US tiktok user data.

TikTok reasserted its dedication to transparency in July, with the announcement of a new API for researchers.

There had initially been discussions over Oracle buying US operations of TikTok outright, alongside a proposal by Microsoft for a similar acquisition. In June, TikTok made a blog post addressing its governance strategy with regards to user data and the agreement with Oracle.

“For more than a year, we've been working with Oracle on several measures as part of our commercial relationship to better safeguard our app, systems, and the security of US user data,” the company stated.

“In addition, we're working closely with Oracle to develop data management protocols that Oracle will audit and manage to give users even more peace of mind.”

IT Pro has approached Oracle for comment.

Datadog has announced its Database Monitoring service now supports Microsoft SQL Server and Microsoft Azure database platforms.

The expanded support enables engineers and database administrators to swiftly identify and resolve database performance issues including slow queries, incorrect indexes in SQL Server and/or Azure databases, and application bottlenecks.

"Microsoft Azure SQL Database and SQL Managed Instance are fully managed database services that feature built-in security controls, automated maintenance and are always up to date," said Ramnik Gulati, senior director, product marketing, data & AI at Microsoft.

"The expanded support of Datadog's Database Monitoring product further strengthens this collaboration by providing Microsoft customers with deep insights into their managed and self-hosted SQL Server, PostgreSQL and MySQL databases, enabling them to build and scale workloads with confidence," added Gulati.

Notable Datadog Database Monitoring features for Microsoft SQL Server and Azure database platforms include valuable query metrics, explain plan analysis, and centralized query, database and infrastructure metrics.

"We launched Database Monitoring last year because we wanted to help our customers reduce database costs, troubleshoot performance inefficiencies and increase collaboration between engineers and database administrators," said Omri Sass, group product manager of application performance monitoring at Datadog.

"By adding support for SQL Server and Azure database services, Microsoft users are better able to accomplish these goals and discover and implement the right database improvements while saving time communicating and reconciling information."

The majority of UK business leaders (81%) expect industry disruption from quantum computing by 2030, according to EY research. That’s a bold claim for a technology that’s yet to have a proven business case. The UK, though, is betting the near future will see the practical application of quantum computing in a wide range of industries.

Last year also saw the establishment of the National Quantum Computing Centre (NQCC). Commenting on EY’s research when it was published, Dr Simon Plant, deputy director for Innovation at the NQCC, said: "Quantum computing is expected to significantly speed up the time to solution for certain tasks, addressing computational problems that are currently intractable using conventional digital technologies. As a result, the pace of development is accelerating, and the question is how and when – not if – quantum computing can address industrially-relevant use cases.”

Sparking quantum disruption

We must first quantify the claim that quantum computing will be disruptive, and define this within the context of business processes. The pharmaceutical, finance and automotive sectors, for instance, are likely to be the first adopters. Moving away from classical computing architectures could revolutionise the development of new drugs, for example, radically cutting the typical ten-year development cycle and slashing the $2 billion average cost of bringing a new drug to market.

Quantum advantage, however, is fast approaching for all businesses, according to managing director of Accenture Technology in the UK and Ireland, Maynard Williams. “The power that this new generation of machines will create will start to make these core challenges achievable, with quantum at the pinnacle of next-generation problem solving. The single biggest watershed moment for computing will be when quantum computers solve the problems that were considered quite literally intractable – making the impossible possible.”

The key to harnessing quantum computers will be to place them into context and use these machines alongside binary computers. Classic computers, indeed, have their own place in the computing landscape in such a way that all enterprises, no matter their size, can take full commercial advantage of the next technological wave.

“I hope the detailed quantum strategy will put a strong focus on industry skills in addition to research and development,” says Richard Hopkins, distinguished engineer and fellow of the Institute of Engineering and Technology from IBM. His company is one of many striving to achieve quantum supremacy alongside the likes of D-Wave and Google. "There's a lot of value the UK could derive by being the first to apply quantum computing to real-world industrial problems, but that will need us to invest in the associated skills.

The business case for quantum

The popularity behind the potential of adopting quantum commuting is vast, as research by Fujitsu and the Tecknowlodgy Group – looking closely at how quantum computing could be applied across a diverse business landscape – found. The study saw 81% of business leaders across sectors including manufacturing, life sciences, retail, transport, and utilities saying optimising business processes can help them to tackle digital transformation challenges. It’ll also allow such businesses to remain competitive in a fast-changing market.

It’s telling that there's been a profound shift in the quantum computing discourse, with the conversation moving from theory to how to apply the technology in practice. Now the worst of COVID-19 has passed, businesses must also be more agile and innovate at speed, which includes establishing how to adopt emerging and once-gimmicky technologies like quantum computing. Business leaders now see this new frontier for computing as a tool in the arsenal towards boosting innovation, including the likes of BASF and Volkswagen, which are already using quantum algorithms within their businesses.

In March this year, HSBC detailed its intention to join the IBM Quantum Accelerator programme, which would give the bank access to the company's 127-qubit processor known as Eagle. “This technology has the potential to transform how we run areas of the bank by addressing challenges which classical computers may never be able to solve alone,” commented HSBC Bank plc and HSBC Europe chief executive officer (CEO), Colin Bell, on the announcement.

Accenture’s Maynard Williams also advises that businesses should begin their strategic planning to shape their quantum future. "The quickest action business leaders can take on the journey to quantum readiness is to begin evaluating how these technologies will shape the operations of their enterprise,” he says. “This will allow them to identify their knowledge and operational gaps, and begin filling them in before it's too late.”

Striving for a quantum advantage

The cost of investment has, until today, positioned quantum computing almost exclusively within the corporate realm. Smaller and medium-sized businesses (SMBs) and startups, however, will also be keen to tap into its potential. Is the quantum door closed to them?

Access to technology has always shaped business cases. What we are seeing in the marketplace right now is the major cloud service providers developing quantum as a service (QaaS) platforms that offer the high level of computing capability to the broadest possible audience. IBM has been offering its quantum computers through the cloud for some time, with the likes of AWS offering a rival Braket quantum cloud service and Microsoft offering Pasqal machines through Azure.

Digital Annealer, produced by Fujitsu, is another quantum-inspired technology architecture designed to help businesses solve complex combinatorial challenges beyond the capabilities of today's computers. Although not an actual deployment of quantum computing, this technology does show the way to the quantum future many businesses can see as part of their development over the next decade.

In some sectors in which there are many smaller businesses – notably financial services and fintech especially – quantum computing is seen as a potential business differentiator, and certainly a technology that could deliver commercial advantages. Despite the progress made in recent years, however, the real-world practical applications of quantum computing won’t arrive until 2030, according to McKinsey. Until then, quantum-inspired algorithms will continue to be developed and deployed in association with high performance computing (HPC) models.

Do all businesses have a quantum future? “Given the difference in the nature of this technology, organisations may wish to ‘phone a friend’ to help them develop their quantum strategy and start their initial investigations,” Deloitte’s Scott Buchholz, national emerging technology research director and CTO for government and public services tells IT Pro. “Done with the right partner, that approach will save time and speed up the learning process.”

Jan Beitner, lead data scientist at Boston Consulting Group, also advises businesses should take practical action today: "Businesses should start by examining their value chain and identifying the use cases most relevant to them. From there, they need to quantify them and, with the support of experts, work out when computers can address them. From that starting point companies can then build a roadmap to prepare for the application and integration of quantum as its potential gets unlocked by advances in the technology.”

As businesses look to re-draw their digital transformation roadmaps in the wake of COVID-19, quantum computing should be a component of that journey. As development continues, we’re not far from a breakthrough that will see quantum computers enter perhaps not the mainstream, per se, but become a transformative technology all businesses can access. The key is having a clear deployment strategy and defined business cases to harness quantum computing's unique properties.

Oracle and Microsoft have teamed up to deliver Oracle Database Service for Microsoft Azure, designed to provide Azure users with direct and streamlined access to Oracle databases on Oracle Cloud Infrastructure (OCI).

The deal allows users to provision, access, and monitor enterprise-grade Oracle Database services in OCI in a way that is familiar, Microsoft said in an announcement.

Users can migrate or build new applications on Azure and then connect to Oracle’s high-performance, high-availability database services.

“Microsoft and Oracle have a long history of working together to support the needs of our joint customers, and this partnership is an example of how we offer customer choice and flexibility as they digitally transform with cloud technology,” explained Corey Sanders, corporate vice president, Microsoft Cloud for Industry and Global Expansion.

“Oracle’s decision to select Microsoft as its preferred partner deepens the relationship between our two companies and provides customers with the assurance of working with two industry leaders.”

The solution builds on Oracle Interconnect’s core capabilities to enable customers to easily integrate workloads on the cloud platform with Oracle Database Services on OCI.

In a matter of a few clicks, users can connect their Azure subscriptions to their OCI tenancy, with the service then configuring everything required to link the two cloud environments and incorporate Azure Active Directory identities.

There are no charges for using Oracle Database Service, Oracle Interconnect, or data egress or ingress when moving data between OCI and Azure, either. Instead, customers will only pay for the other Azure or Oracle services they consume – such as Azure Synapse or Oracle Autonomous Database.

Clay Magouyrk, executive vice president of Oracle Cloud Infrastructure, said the solution gives customers the ability to “easily test and demonstrate the value of combining Oracle databases with Azure applications".

“There is no need for deep skills on both of our platforms or complex configurations – anyone can use the Azure Portal to get the power of our two clouds together,” he said.

Bespoke, data-rich and highly available applications are at the heart of the next wave of digital transformation. These applications will be key in giving enterprises a deeper understanding of their customers and their needs and helping them to build new digital experiences to engage those customers with.

They could provide real-time visibility and analytics across their business. Yet development of these applications is being hampered by issues relating to the provisioning and management of data architecture – in particular, the traditional data architecture being run from on-premises servers. Legacy database technology is holding innovation back.

Why is this? The fundamental reason is that traditional, SQL-based databases and the architecture they run on were designed to support the needs of very different applications, built for industries and markets that didn’t involve the same quantities of data or operate at today’s speeds. Traditional on-premises data architecture is slow to provision and slow to scale, which is a problem when you need to be able to scale upwards rapidly to meet peaks of demand or scale down to keep costs and complexity under control.

Enterprises are already handling larger volumes of data, and those volumes threaten to get larger still. IDC’s StorageSphere Forecast 2021-2025 predicted that worldwide data storage capacity would grow from 6.7 zettabytes in 2020 to 16.1 in 2025, meaning an increase of 240% in the infrastructure and services required to store and manage it. On the level of the individual enterprise, that means more servers and infrastructure to manage, as well as a wider range of different architectures to cover different applications and requirements. Server sprawl is becoming a serious issue.

This in turn demands even more of database administrators (DBAs), which is an issue when there’s a growing skills shortage in the field. A 2021 report by EFG Research puts DBAs within the top 10 most common IT roles facing shortages. The UK government’s May 2021 paper, Quantifying the UK Data Skills Gap, puts database management as the seventh most in-need skill for UK businesses. The growing complexity of the role is only exacerbating the shortages. As a recent article for Global Banking and Finance Review puts it, “It is common that database professionals have been tasked with developing a working knowledge of multiple database systems, but they might not know each of them well enough to be considered an expert, and that’s what is needed.”

DBAs are also having to contend with the growing challenges of database security in environments where instant, high-speed availability is crucial, and where more advanced applications in AI, analytics and machine learning need to handle complex datasets in real time. Ransomware threats are on the rise, while many organisations are struggling to build a robust data retention strategy. Overall, traditional architectures are proving difficult to manage and secure, while actively slowing down enterprises’ path to insight. Is it any wonder that many enterprises are looking for a smarter way to work?

As with so many IT challenges, the answer lies in the cloud. Back in 2019, Gartner predicted that, by 2022, 75% of all databases will be deployed or migrated to a cloud platform, and that only 5% of these would ever be considered for repatriation to on-premises hardware. Flexera’s 2022 State of the Cloud report shows data warehousing, relational database-as-a-service (DBaaS) and NoSQL DBaaS making up three of the four most used platform-as-a-service (PaaS) cloud services. What’s more, open source technologies are taking off. Gartner has also predicted that 70% of new in-house applications will be developed on an open source database management system (OSBDMS) or OSDBMS-based database-platform-as-a-service (dbPaaS).

Next gen apps need a modern architecture

These open source technologies and cloud-based database platforms are a better fit for this new generation of applications. MongoDB is built for high-speed analytics and innovation. You can spin up new applications and services without wasting precious time configuring and tuning the architecture. MySQL is fast, scalable, highly customisable and cost-effective. As an object-relational database system, PostgreSQL has powerful features that increase its flexibility and make it easier to organise and model data with a higher level of consistency. Meanwhile, Redis and Apache Kafka are ideal for caching and streaming data securely, at high speeds, for real-time applications, while OpenSearch provides fast access, analytics and search with integrated visualisation tools.

What these platforms all have in common is that they’re highly scalable, both up and down, and that they enable innovation, with rapid deployment, high security and high availability. What’s more, as they’re open source, there’s no vendor lock-in. While switching from one platform to another is never trivial, the architectural features and available tools actively support it.

Yet simply moving to a modern, cloud-based database architecture is no silver bullet; DBAs and IT teams still face challenges in managing and securing them, particularly with multiple systems in place, working with existing, traditional architecture within a hybrid estate. That’s why it makes sense to work with off-the-shelf and custom DBaaS solutions from a provider such as OVHcloud. OVHcloud Managed DBaaS solutions give you instant provisioning across an infrastructure designed for modern data architectures, with scalability, security and high availability built-in. The platforms are true open-source, with no vendor lock-in, and offer a market-leading, highly cost-efficient price/performance ratio, with transparent, predictable pricing.

Perhaps most importantly, there’s no need for any time-consuming infrastructure management, meaning DBAs and IT teams can spend less time and manpower just maintaining back-end operations, and more time delivering solutions that add value. OVHcloud manages all logs and metrics, backups, database management system (DBMS) installations and updates, not to mention operating system installations and updates. OVHcloud handles all the hardware and server maintenance, and is responsible for delivering availability, power and network access according to the terms of the SLA. Everything is in place to spin up new nodes – or spin them down – within minutes rather than days. OVHcloud has been one of Europe’s leading cloud and infrastructure providers since 1999, enterprises can be confident that they won’t run into CLOUD Act issues, establish data sovereignty and feel assured that they’re working with a company they can trust.

If the next stage of digital transformation depends on a company's abilities to develop and operate custom-built, data-rich apps, OVHcloud is working to support them, through DBaaS platforms that are agile, scalable and based on managed infrastructure. That gives enterprises the freedom to grow and move rapidly, and build the products, tools and services their customers’ demand.

Learn more about OVHcloud’s database-as-a-service offering

Databricks has announced several new contributions to popular data and AI open source projects, including Delta Lake, ML flow, and Apache Spark.

At its Data + AI Summit, the data and AI specialist said it will contribute all features and enhancements it has made to Delta Lake to the Linux Foundation, as well as make all Delta Lake APIs open source as part of its Delta Lake 2.0 release.

That means the open source community will benefit from the full functionality and enhanced performance of the Delta Lake 2.0 ecosystem, enabling the building of high-performance data lakehouses on open standards. The Delta Lake 2.0 Release Candidate is now available, with a full release expected later this year.

The firm also announced the next iteration of open source machine learning project MLflow 2.0, which introduces MLflow pipelines to the platform. The addition aims to substantially decrease time to production and improve execution at scale through standardisation.

MLflow Pipelines offers data scientists pre-defined, production-ready templates based on the model type they’re building, allowing them to bootstrap and accelerate model development without requiring intervention from production engineers.

Additionally, Databricks revealed its new Spark Connect, which will enable the use of the unified data analytics engine Spark on virtually any device, as well as Project Lightspeed, a next-gen Spark Structured Streaming engine for data streaming on the lakehouse platform.

“From the beginning, Databricks has been committed to open standards and the open source community,” commented Ali Ghodsi, Co-Founder and CEO of Databricks. “We have created, contributed to, fostered the growth of, and donated some of the most impactful innovations in modern open source technology.

“Open data lakehouses are quickly becoming the standard for how the most innovative companies handle their data and AI. Delta Lake, MLflow and Spark are all core to this architectural transformation, and we’re proud to do our part in accelerating their innovation and adoption.”

New Lakehouse innovations

Databricks also unveiled several innovations for its Lakehouse Platform at the Data + AI Summit. New capabilities include data warehousing performance and functionality, expanded data governance, and new data sharing innovations which include Databricks Marketplace and Data Cleanrooms for secure data collaboration.

There’s also automatic cost optimization for ETL operations, as well as machine learning lifecycle improvements to “radically simplify” MLOps at production scale.

“Today’s announcements are a significant step forward in advancing our Lakehouse vision, as we are making it faster and easier than ever to maximise the value of data, both within and across companies,” Ghodsi said.

Boeing has struck a deal with space-based automatic dependent surveillance-broadcast (ADS-B) service provider Aireon to better predict and prevent aviation hazards.

Per the accord, Aireon will offer its flight data stream to Boeing, expanding the aerospace firm’s data analytics capabilities. Select Boeing airplane programs will have access to historical aircraft data, including near real-time aircraft event data, via AireonINSIGHTS.

“The global space-based ADS-B data from AireonINSIGHTS can help customers gain insights to key performance indicators on flight safety,” stated Aireon.

Boeing will notably integrate Aireon-provided ADS-B data into enterprise Safety Management System (SMS), in a bid to proactively monitor risks and ‌changing‌ ‌safety trends.

Oceanic, polar, and outlying areas can be monitored by space-based ADS-B, which adds to conventional ground-based surveillance limited to terrestrial airspace.

"We are investing in a data stream that can be transformed into safety intelligence," commented Vishwa Uddanwadiker, vice president of aerospace safety analytics at Boeing.

"We are adding this to our data analytics ecosystem to help predict and prevent safety risks, while identifying other opportunities to strengthen our Safety Management System."

Aireon CEO Don Thoma said, "The power of the Aireon data unlocks a cache of information for Boeing regarding the operations of its aircraft in the global airspace. With this integration, Boeing will have data to provide a full operational view of its fleet, and we are excited to partner with them.”

Datadog has launched a new service to help organizations aggregate, filter, and route ‌large volumes‌ ‌of observability data.

Dubbed Observability Pipelines, the vendor-agnostic tool offers customers with a unified view of infrastructure and application metrics, logs, and traces.

The granular outlook helps with lowering costs, reducing technology lock-in, standardizing data quality, improving compliance, and scaling observability practices, according to Datadog.

Most notably, businesses can gather, enrich, transform, sanitize, and route observability data from any source to any destination at petabyte scale. Rule-based throttles and reactive routing strategies ‌can also be deployed to protect against data spikes.

If need be, organizations can also choose to redact sensitive data from a data stream ‌before‌ ‌it‌ ‌leaves‌ ‌the company‌ ‌network.

"As the amount of telemetry continues to grow at an organization, teams are often completely overwhelmed—if they're not blind to—the runaway costs, reliability and compliance risks that come from a lack of visibility into infrastructure and application data," said Zach Sherman, senior product manager at Datadog.

“We built Datadog Observability Pipelines to give organizations a powerful way to take back control of their data, without compromising visibility for engineering, security and SRE teams," added Sherman.

Datadog’s Observability Pipelines also comes with an extensive list of integrations or built-in connectors that give organizations the ability to collect and route data quickly and easily to any existing tools.

KPMG has signed a 'strategic alliance' with Californian tech firm Alteryx that will aim to boost the accountancy giant's data analytics services.

With Alteryx, KPMG will seek to push forward plans to automate its own systems, while simultaneously ramping up its data handling platforms.

The two firms will combine their expertise in developing data strategies and digital transformation programs to help companies deploy "automated analytics" into everyday business processes so they can make "smarter and faster decisions". All of this will be powered by Alteryx's unified end-to-end analytics automation platform, which is rolling out across KPMG's systems over the coming months.

The two companies will be able to deliver actionable insights and also reduce operational costs, according to KPMG, with shared clients able to accelerate digital transformation plans with data-driven insights powering automation.

"As we adapt to the digital age, we are seeing a cultural shift across the firm to one founded in technology and data," said Emily Watkins, KPMG's head of deal analytics. "But trapping our capabilities in our technology teams is simply unsustainable. Working with Alteryx will put the power of data science and automated analytics in the hands of our people and our clients - empowering the next generation of data workers to capitalise on what their data is telling them."

Alteryx, which was founded in 1997 by the software entrepreneur Dean Stoecker, was originally known as 'SRC LLC'. The company provided demographic mapping systems to the US government for use in its yearly census.

The term 'Alteryx' was a type of data mapping software the company released in 2006, which it adopted as its name four years later. The company went public in 2017, floating on the New York stock exchange, and is headquartered in Irvine, California, with a development centre in Colorado.

Auditing is an independent consulting activity, intended to help organisations improve their operations, as it evaluates ways to improve risk management. However within the technology industry, auditors are being replaced by automation, and with CFOs looking to reduce spending, the function of audit and compliance could become obsolete.

The audit process needs to change for the better, and with emerging technology, there’s great potential for auditing to adapt and drive real value in relation to data analytics and risk management monitoring.

Download this whitepaper to learn where audit technology is headed, and what it means for your organisation.

Provided by

The global automotive data management market size is projected to reach $4,807 million by 2030, according to a recently published report by Acumen Research and Consulting (ARC).

The market research firm estimates a compound annual growth rate (CAGR) exceeding 20.5% over the forecast period beginning 2022 through 2030.

“Previously limited to marketing and sales, data analytics is now being used in product development, industrial processes, logistics, and aftermarket solutions.”

“However, the increasing use of connected vehicles is increasing the demand for automotive data management. The growing usage of embedded sensors in self-driving automobiles is driving the automotive data management market forward,” explained ARC.

The firm also affirmed, “automotive sector confronts strong competition, and fresh developments are required to stay competitive.”

Acerta Analytics Solutions, Caruso GmbH, AMO Foundation, Amodo D.o.o, AWS, and IBM are among the key vendors influencing the automotive data management industry.

By region, North America is anticipated to hold the majority of the global automotive data management market share during the forecast period, due to a large number of automakers in this region, along with the widespread implementation of lean analytics and deep learning.

In the next few years, analysts expect Europe to be the second-largest market for automotive data management. The region's share of the market is predicted to grow exponentially as a result of increased car production, advancements in vehicle electronics layout, connected automobiles, V2V & V2I innovations, and the expansion of key OEMs.

“The increased demand for built-in informatics and telematics in automotive has resulted in advancements in technology aimed at improving the entire in-vehicle experience for both passengers and drivers. With the connected vehicle, ubiquitous inbuilt cellular connections provide new opportunities to educate and engage drivers, as well as service the vehicles throughout their existence,” stated ARC in its report.

Search-powered solutions provider Elastic has announced an expansion of its collaboration with Amazon Web Services (AWS).

The pair will work together to build, market, and deliver seamless access to Elastic Cloud on AWS, while leveraging the cloud giant’s global footprint and wide range of services.

Elastic said the collaboration will enable easier migration to Elastic Cloud through expanded competencies, as well as simplified onboarding and new joint go-to-market initiatives.

“Through our collaboration, customers can take advantage of simplified data collection from AWS services, and cloud-optimised features that optimise their Enterprise Search, Observability, and Security solutions with Elastic Cloud on AWS," explained Steve Kearns, vice president of Elastic Platform at Elastic.

"Elastic is committed to delivering a proven subscription experience for our Elasticsearch service through the AWS Marketplace, and we look forward to continuing to work with AWS to help organisations get the most value out of their deployment and focus on what is critical to their business.”

In terms of expanded competencies, Elastic has been certified as an AWS ISV Workload Migration Program (WMP) Partner to support the migration of customers to Elastic Cloud on AWS. The firm has also earned multiple new AWS validated technical qualifications.

Elastic said it is also aiming to simplify onboarding to Elastic Cloud on AWS and will focus on offering enhanced integration for account creation and setup – reducing the initial setup time “from hours to minutes".

In fact, the pair have already completed more than 20 integrations to streamline data ingestion into Elastic Cloud on AWS, including integrations with AWS FireLens, Amazon S3 Storage Lens, AWS WAF, and AWS Network Firewall, along with the Elastic serverless forwarder.

Additionally, the two companies have committed to investing in joint go-to-market initiatives, which include a free seven-day trial via the AWS Marketplace.

Stephen Orban, vice president of AWS Marketplace, Partner Engineering, and ISVs at AWS, said the companies’ relationship is “rooted in listening to customers".

“Enabling frictionless migration and operation of Elastic Cloud on AWS and a streamlined subscription experience is just the beginning,” he said. “Through our expanded efforts, we are ensuring our joint customers have easier access to Elastic Cloud on AWS so they can focus on solving their most complex challenges.”

With the wide-ranging capabilities that Netsuite offers, Netsuite integration projects can involve many applications. This guide can support you in simplifying this process with a checklist of things to consider in your planning.

From the people involved, to the data flow processes, the applications, and the data, this guide takes into account how to identify each stage, what you need to determine - including specifications and recommended methods - as well as common issues.

Download this checklist now to ensure your Netsuite integration is straightforward, and gives you a competitive advantage.

Provided by

This asset shares advice and tips on the technical requirements you might need, as well as key considerations for your Salesforce integration project.

From the people involved, to the data flow processes, and the applications, this guide takes into account how to identify each stage, what you need to determine, as well as common issues and customisation considerations.

Download this checklist now to future proof your Salesforce integration.

Provided by

It’s clear that embedded analytics as a core part of most enterprise software is a useful tool. However dashboards and data reporting have often been bolted on to existing software in the past, rather than being a core component of the application.

This can mean limitations when analysis is reviewed separately, causing disruption in workflows, lack of context, and friction in the user experience.

Download this whitepaper to better understand:

• What contextual analytics is and how it can enhance your business software
• How the maturity model can help speed up modernisation
• Why contextual analytics should be part of your next innovation

Provided by

While there is a broad understanding that successful organisations need to be data-driven, what does that mean in practice and how can your business achieve this?

This whitepaper lays out the key steps a company needs to take to become a truly data-driven institution that bases strategic business decisions on concrete evidence, and how agility, digital transformation and continuous innovation tie into this project.

Download now to learn how other companies are making the best use of their data and what that can teach your organisation.

Provided by

IT infrastructure services provider Kyndryl has expanded its global alliance with Dell Technologies in the form of a new cyber resilience solution to protect critical business data.

The partnership is focused on providing cyber incident recovery, offering cyber resilience through Dell’s high-performance storage, servers, and data protection systems.

Kyndryl says the new jointly developed solution has been designed to help secure critical data sets and provide a verified process to recover data back into a business.

It uses an “air-gapped” data vault from Dell, which means one copy of the data is offline and cannot be accessed to help ensure availability in the event of a cyber incident.

“We’re excited to expand our relationship with Dell and look forward to delivering solutions, services and support that customers need to store and protect their data and recover from cyber threats,” commented Kris Lovejoy, Global Security and Resiliency practice leader at Kyndryl.

“Through this important alliance, we will work together to help companies improve performance and availability of critical data, predictive maintenance, non-disruptive upgrades and refreshes.”

Dell recently elevated Kyndryl to Titanium Black Global Partner status in the Dell Technologies Partner Program – a status that recognises partners it deems best equipped to support customers through their digital transformation.

Building on 20 years of engineering and go-to-market collaboration, the firm says its new status and expanded alliance will enable customers to enhance their digital innovation and business transformation.

Ultimately, Kyndryl’s new Cyber Incident Recovery with Dell Technologies offering aims to enhance and complement existing backup and disaster recovery solutions, help minimise the impact of cyber incidents, as well as enable a quicker return to normal operations with a greater likelihood of recovering critical data.

“It’s more important than ever for companies to have confidence in the protection of their critical data to ensure cyber resiliency,” said Denise Millard, senior vice president, Global Alliances, Dell Technologies. “Dell and Kyndryl are well positioned to help our customers succeed in the multicloud and data era.”

The partnership follows a string of deals for Kyndryl, which completed its spin off from IBM in November 2021. These include a major cloud migration deal with Microsoft, two separate edge computing partnerships with Nokia and Lenovo, and a deal with AWS to create a cloud 'centre of excellence' for customers.

Google Cloud has introduced a preview of BigLake today, a data lake storage engine, it said would remove data limits by unifying data lakes and warehouses.

The tech giant said that managing data across disparate lakes and warehouses create silos and increases risk and cost, especially when data needs to be moved. Through BigLake, it hopes to allow companies to unify their data warehouses and lakes to analyse data without worrying about the underlying storage format or system. This, it said, eliminates the need to duplicate or move data from a source and reduces cost and inefficiencies.

Google Cloud customers will gain access controls through BigLake, with an API interface spanning Google Cloud and open file formats like Parquet, as well as open-source processing engines like Apache Spark. The company said these capabilities extend a decade’s worth of innovations with BigQuery to data lakes on Google Cloud Storage to enable a flexible and cost-effective open lake house architecture.

BigLake is set to be the centre of the company’s overall strategy from a storage perspective, revealed Sudhir Hasbe, Google Cloud’s data analytics product manager.

“We are going to make sure all of our tools and capabilities work seamlessly with BigLake,” he said. “Similarly, all of our analytics engines, whether it's BigQuery, whether it's our Spark engine, or whether it's our data flow engine, all of these engines will seamlessly integrate out of the box with BigLake.”

The company also announced the formation of the Data Cloud Alliance, a new initiative to ensure global businesses have more seamless access and insights into the data required for digital transformation.

The alliance comprises Google Cloud, Accenture, Confluent, Databricks, Dataiku, Deloitte, Elastic, Fivetran, MongoDB, Neo4j, Redis, and Starburst. The companies said they are committed to making data more portable and accessible across disparate business systems, platforms, and environments.

Google Cloud said that the proliferation of data means businesses increasingly need common digital data standards and a commitment to open data, to effectively use data to digitally transform.

The Data Cloud Alliance is committed to accelerating the adoption of data analytics, artificial intelligence, and machine learning best practices across industries through common industry data models, open standards, and integrated processes.

The alliance’s members will work together to help reduce customer challenges and complexity with data governance, data privacy, data loss prevention, and global compliance. They will provide infrastructure, APIs, and integration support to ensure data portability and accessibility between multiple platforms and products across multiple environments.

Each alliance member is also set to collaborate on new, common industry data models, processes, and platform integrations to increase data portability and reduce the complexity associated with data governance and global compliance.

HPE’s Spaceborne Computer-2 (SBC-2) has successfully completed 24 research experiments on the International Space Station (ISS). The company claims this has accelerated time-to-insight from months and days to minutes.

The SBC-2 is the first in-space commercial edge computing and AI-enabled system to run on the ISS, according to HPE, and was installed in May 2021. The experiments involved real-time data processing and testing of new applications to prove reliability in space to increase autonomy for astronauts. The experiments were focused on healthcare, image processing, natural disaster recovery, 3D printing, 5G, and AI.

SBC-2 consists of HPE’s edge computing hardware, including the HPE Edgeline Converged EL4000 Edge system, which is designed to perform in harsher edge environments, such as space. It’s also made up of the HPE ProLiant DL360 server, a server for additional high-performing capabilities to target a range of workloads, including edge, HPC and AI.

The supercomputer is part of a mission to advance computing and reduce dependence on communications as humans travel further into space to the moon, Mars, and beyond. It aims to demonstrate ways astronauts can increase self-sufficiency when processing data directly on the space station, bypassing longer latency and wait times that occur when relying on sending raw data to Earth to be processed and analysed, and sent back to space.

For example, HPE outlined that previously, 1.8 GB of raw DNA sequence data took an average time of 12.2 hours just to download to Earth for initial processing. With SBC-2, researchers on the ISS could process the same data in just six minutes to gather meaningful insights, compress it to 92 KB, and send it to Earth in just two seconds, representing a 20,000X speed-up.

“By introducing edge computing and AI capabilities to the International Space Station with Spaceborne Computer-2, we have helped foster a growing, collaborative research community that shares a common goal to make scientific and engineering breakthroughs that benefit humankind, on space and here on Earth,” said Mark Fernandez, principal investigator of Spaceborne Computer-2 at HPE. “We are proud of this ongoing work, which has already resulted in 24 completed experiments, from various organisations, demonstrating new possibilities for space exploration and milestones for humanity.”

What experiments have the Spaceborne Computer-2 been involved in?

The SBC-2 has run experiments for researchers developing capabilities for space exploration, like Axiom Space, Cornell University, Cumucore, Microsoft, NASA, and Titan Space Technologies.

Using AI-enabled damage detection in astronaut gloves

Astronauts on the ISS are often on spacewalks where they repair equipment, install new instruments and upgrade features and functions on the space station. They wear essential gloves that can experience natural erosion, along with even rips and cuts, that can present potential safety concerns.

In an experiment led by NASA, HPE and Microsoft, photos and recorded video taken in space of recently worn gloves by astronauts were processed using SBC-2’s AI capabilities. The glove analyzer model was then used to rapidly look for signs of damage on-orbit, in space. If damage is detected, an AI-annotated photo is generated in space and immediately sent to Earth, highlighting areas for further review by NASA engineers.

Automatic interpretation of satellite imagery after a disaster

NASA Jet Propulsion Laboratory (JPL) observes Earth from space to study science and climate, as well as support disaster response. Using Spaceborne Computer-2 along with embedded processors, NASA JPL tested several deep learning inference networks to automatically interpret remote sensed images from land and structures after a disaster.

Two of these techniques use radar data to determine flood extent, such as from a hurricane, and to determine urban building damage, such as from an earthquake. HPE said these techniques could be used onboard future spacecraft to rapidly deliver actionable products to relevant authorities to assist in disaster recovery.

3D printing in space through validated software

As humans look to future deep space travel, ordering supplies to repair or build new equipment from Earth will not be timely and practical, explained HPE. To increase self-sufficiency by enabling additive manufacturing for humans travelling beyond low Earth orbit (LEO), Cornell Fracture Group, part of Cornell University, developed a modelling software that can simulate 3D printing of metal parts and even predict failure and deformation that may result when printing in space. HPE said the software was successfully tested on SBC-2, showing that it can be used in space to digitally simulate a part and understand how it will perform in reality.

Expanding network capability on the ISS with a 5G core prototype

Cumucore, a private mobile network, tested its 5G core network, along with RAN emulators and other features, on SBC-2, to emulate current capabilities on the base station and end-user devices. HPE underlined that the experiment showed the potential to install state-of-the-art 5G capabilities on selected satellites and spacecraft to access a new level of communications in space.

Developing software codes to calculate fuel requirements based on space travel distance

HPE opened up SBC-2 to students in India involved in Codewars, an educational community and competition for computer programming. The projects largely focused on developing code using C++, Python and Fortran, with one involving C++ Pythagoras theorem code to calculate how much fuel is needed for a given space travel distance to directly travel to without requiring refuelling.

Telefónica Tech has announced the acquisition of UK-based Microsoft partner Incremental for up to £175 million, including potential contingent payments linked to future performance.

The move strengthens Telefónica Tech’s presence in the UK and Ireland, as well as its scale and competencies in Microsoft technologies, the firm said.

Launched in 2016, Incremental Group has become one of the leading Microsoft Dynamics Partners in the UK and is also a major player in the data and analytics market.

The company has been a Microsoft Business Applications Inner Circle member for the past three years and is one of just two partners in EMEA on the Redmond giant’s Global Advisory Board for data analytics.

The acquisition means Telefónica Tech now becomes one of Microsoft’s largest partners in the UK, with 16 Gold Competencies and 5 Advanced Specialisations.

Telefónica said the addition of Incremental’s business applications, power apps, data analytics and management capabilities to its cloud services portfolio will enable it to enhance its end-to-end cloud offering for mid-market, large enterprise, and government customers.

“We warmly welcome Incremental Group, and are delighted that this acquisition enables us to strengthen our position as a leader in the UK market for IT services,” commented José Cerdán, CEO of Telefónica Tech.

“We are now able to provide end-to-end Microsoft services, including digital transformation, managed services and data analytics, and achieve attractive cross-selling synergies with Telefónica Tech UK&I, complementing and positioning us as a leading Microsoft-focused company in the UK.”

Glasgow-headquartered Incremental has experienced significant organic and inorganic growth in recent years. Early last year, the firm acquired Redspire in a bid to expand its capabilities in the financial services market and become one of the UK’s largest Microsoft Dynamics partners.

It then snapped up data analytics company Adatis, which enabled the business to offer end-to-end digital transformation to customers.

Today, Incremental has more than 350 employees on its books, which will take Telefónica Tech UK&I’s headcount to around 1,000 staff post-transaction.

Neil Logan, CEO at Incremental Group, said the focus was on securing the “right support for the long term” as the business looked to the next stage of its development.

“Becoming part of Telefonica Tech propels us into the next phase of our journey and enables us to further scale the business, creating exciting opportunities for our people and our customers,” he said.

If you want to win in the channel, B2B e-commerce is a tool you can't afford to ignore, and, according to McKinsey, it's now the leading edge of successful B2B sales.

Indeed, larger channel players like Westcoast, Tech Data and SCC have been working on their e-commerce propositions for years, anticipating further potential as data and analytics are further integrated into the technology landscape. That's despite persistent myths surrounding B2B e-commerce – including notions that customers aren't ready; that it's only for simple repeat business; and that customers always prefer face-to-face interactions.

Ian Briffett, director of sales process optimisation at Tech Data, confirms the distie continues to invest in digital platforms as part of its own digital transformation strategy, spurred on by the last two years of disruption. “If nothing else, that made people sit up and look for where we need to make a difference,” he tells IT Pro.

Briffett, however, “broadly agrees” with McKinsey that B2B e-commerce strategies can suffer from misperceptions, relying on outdated notions of online as being all about an advertising shopfront.

“If you go back about two years, many people did look at B2B e-commerce as being either marketing or purely transactional; it was a very different era,” he explains. “What's changed is it's more about how we enable a broad channel and provide additional capabilities and value to help customers grow their businesses.”

Fully integrated platforms will drive success

From working out how to increase software licensing and recurring revenues to providing the right levels of customer service, platforms that can drive processes will enable success.

Most B2B companies (65%) now offer e-commerce, according to McKinsey, meaning they can fully transact sales online – up from 53% in early 2021. Would the channel not be further ahead than non-tech sectors? Tech Data reportedly earned 50% of its revenues via e-commerce five years ago; back in 2017.

Canalys chief analyst Alastair Edwards says their research suggests 40% offer “some form of e-commerce online”, with another 20% planning online sales channels.

“It’s complex. What is a channel digital marketplace, as opposed to an e-commerce site?” asks Edwards. “We see large distributors and bigger partners, however, building digital marketplaces as a necessity as more software as a service (SaaS), cyber security and cloud-based services are transacted digitally.”

Data, and its growing role in digital transformation, poses questions to businesses that can be answered with e-commerce solutions, creating data that drives further insights, prompting further questions, and so on. From addressing immediate problems and pain points, e-commerce moves into driving new business incrementally.

“Software vendors, too, are pivoting away from selling traditional perpetual licences now to a subscription model,” Briffett adds. “Once you've sold something to a customer, you have to retain them, get them to renew.”

By helping vendors or distributors put the right information in front of the right people – at the right time – they should be better positioned to go and drive the “right cadence” of activities, he says.

“The other piece is to migrate these customers now to this subscription licence,” he continues. “We, therefore, need to be offering alternate quotes and help customers make informed decisions in an appropriate amount of time. That’s good for us, good for customers and good for vendors.”

For maximum benefit, information can be provided in a number of ways, including through portals, social media and other channels. If a channel partner needs a quotation to go from A to B, Tech Data publishes it automatically, for instance. Information can also be targeted to specific customers.

“We know that customers that got red boxes can move to blue boxes – or if they got one product from a vendor that they can move to another product in that portfolio,” Briffett says.

Will Omnichannel become omnipresent?

For each stage of the sales process B2B customers now employ a roughly even mix of traditional, remote, and self-service sales using e-commerce and digital portals, McKinsey suggests. Now, 94% of decision-makers say omnichannel sales are “as effective or better” than pre-pandemic models.

Many partner e-commerce forays are mostly for marketing, addressing different points in the customer journey. SCC, for instance, just partnered with IBM on a series of “virtual engagement” centres.

Tech Data, meanwhile, offers specific platforms and services for different tasks – such as for cloud services set-up with “appropriate” automated billing. Traditional account management, however, will remain important, not only for in-person calls or meetings but support and guidance. E-commerce platforms should facilitate, rather than take away, the entire burden.

“Partners, however, aren't there because we're beating them with a stick,” says Briffett. Canalys' Edwards warns, too, that face-to-face interactions can be preferred for complex projects or systems. Sophisticated e-commerce engines, meanwhile, perhaps embedded with AI analytics, mainly suits the scale of the Bechtles, Dustins, Ateas or Insights of the world. Without sufficient digital presence, however, partners can miss customer opportunities.

“Closed intranet-type procurement models are one potential differentiator, where pricing and discounts are pre-agreed,” Edwards says. “Capturing customer usage data from digital platforms can drive targeting that supports customer lifecycle strategies.”

Knowledge is power, or so they say. It’s an axiom that the business world has taken to heart. For the past several years, organisations have been making efforts to gather as much data as possible, using new digital tools and services to fuel data ingestion on an unprecedented scale.

This includes operational data like sales figures as well as technical metrics around application performance and uptime, but the area that businesses have arguably been putting most of their data collection energies into is customer profiles. In some ways, this represents a natural extension of their pre-digital behaviour, evolving from the analogue record-keeping efforts which drove mailing lists and the like. Today’s companies are putting ever-larger investment into building digital data warehouses filled with every scrap of information about their customers that they can conceivably muster.

Making the most of the data

However, while they’ve been focused on growing those data warehouses, organisations haven’t necessarily been putting the data itself to use in ways that benefit their customers. As Adobe’s head of audience and data technologies Steve Allison points out, changing regulations are inspiring many businesses to re-evaluate their strategies. Recent regulatory frameworks like GDPR and CCPA have prompted, and been prompted by, a growing awareness among customers of just how valuable their data is.

“The whole thing around consent has been really important,” Allison says. “It focuses [companies] on what they actually need in order to make personalisation work on their site. And it's interesting because often that leads to the question of what personalisation is. The ‘Hi Steve, welcome back’ stuff is fine, but it's not a great, sophisticated use of personalisation, and I think now we're discovering new ways of using data to make it meaningful for the actual person giving you the data.”

One of the most significant shifts Adobe is seeing in this regard is a move away from large-scale, upfront data collection at the commencement of a customer relationship, with more emphasis on “drip-collection of data”. In this model, users expect organisations to demonstrate the value exchange of handing over any information before they do so, and will only share additional details if they feel they’re getting an appropriate benefit.

To do this effectively, Allison says, organisations must focus on making common customer journeys easier. This could be something as simple as pre-populating the fields in an order form with information that a user has previously provided, but he also notes that businesses need to ensure that they’re looking at how data and personalisation can be applied to interactions beyond an initial sale, over the course of an entire customer lifecycle.

The tools for a robust strategy

Utilising customer data can bring many benefits beyond just marketing and sales. Recognising that data-driven personalisation can be leveraged throughout an organisation is a key aspect of building an effective data strategy. However, building that strategy around a robust technical foundation such as Adobe’s Real-time Customer Data Platform (CDP) is also crucial for ensuring that these benefits can be effectively realised.

Adobe’s Real-time CDP is a customer data aggregation and management platform which allows enterprises to collate all the first-party data they have pertaining to customers – including identity information, on-site actions and behaviour, comms preferences and more – into a single, governable profile. The aim is to provide a complete picture of customers that dynamically adapts to changing data and events, and allows organisations to make business decisions in real time.

“It's about making sure that marketing, servicing, post-sales, etc all have visibility to the data at the right time, at the right stage of the journey that user’s in. It actually makes the enterprise more aware of that customer, and that's the beauty of having a customer profile that is manageable and open to all elements of the business,” Allison explains, “rather than the more siloed approach that we often had before.”

In-stream segmentation is one use-case for this CDP technology. Rather than having to ingest and process data on user behaviour before you could re-segment users for the purposes of more effective content targeting, this re-categorisation can now be done on the fly, based on real-time actions that those users are taking, which minimises page loads and ensures more accurate targeting.

“Having that data accessible also means that you will start to do more intelligent processing of it,” Allison continues. “So you can automatically be calculating things like propensity scores, or churn scores, or machine learning-based fraud detection, out there on the edge, doing that analysis in real time as the data is happening.”

In addition to the marketing and sales teams, implementing a CDP also brings marked benefits for CIOs and IT departments, particularly where data management and portability is concerned. The standardisation of data model and schema definition can be helpful not just in terms of importing and sharing data, Allison says, but also for helping IT departments to properly understand the data itself. A CDP can also help free up time and resource, allowing IT departments to focus on their other duties and bigger-picture strategy.

Because CDPs pull data from various pre-existing locations like data warehouses, data lakes, CRM systems and point-of-sale units as well as incoming data streams, they can be used to create a central cache of all of that information without having to substantially re-engineer backend systems. CDPs also allow CIOs to reconcile a potentially large number of front-end tags for various data collection tools, and replace them with a single set of universal tags, cutting down on workload and minimising potential conflicts.

Deeper consideration

Of course, this kind of data management strategy isn’t without challenges. According to Allison, the primary thing that organisations will need to be mindful of is how they approach data governance. While modern data platforms like a CDP allow businesses to easily layer and map consent information onto their data using a scalable model, thought must still be given to what kinds of data are collected, how long it’s stored for and what purposes it’s put towards – all with the goal of providing genuine value to customers in exchange.

“A lot of the people I talk to at the moment are going through that process of rethinking exactly what kind of interactions they want to have with customers,” says Allison. “And a lot of that navel gazing is changing the ways that they want to interact and therefore the data that they need in order to actually make that interaction take place.”

Organisations are taking an increasingly strategic view of their data assets and how they can best be used to benefit the business, but a strong data strategy is only as good as the technology used to actualise it. A unified data management toolset like Adobe’s Real-time CDP can accelerate innovation throughout the enterprise, giving multiple departments the ability to contribute to and manipulate customer profiles in a non-destructive manner, creating a single source of truth from known and unknown customer data that delivers actionable insights in real time.

“The one piece of advice is start simple: start small,” says Allison. “Start with stuff that you know is going to make a difference, because the beautiful thing about a CDP is that you can add on use cases all the time… you can expand it as much as you want.”

Sign up for sessions at Adobe Summit (15-16 March) here and read more about Adobe’s Real-time Customer Data Platform here

To achieve cost-effective performance at scale and leverage multiple public clouds at the same time, Dell EMC PowerScale for Multi-Cloud has proved itself to be one of the most easy-to-use cloud file services today.

Enabling high availability for business continuity, data resiliency and flexible scalability, this technical review, as part of the Multi-Cloud Data Services for Dell EMC PowerScale, demonstrates how your organisation can solve your data challenges in the public cloud effectively.

Download this whitepaper to learn how Dell EMC PowerScale for Multi-Cloud can enable:

• Compute cost savings
• Storage cost savings
• And analytics efficiencies

Provided by

With the world changing faster than ever, governments and local authorities are under pressure to respond to a growing list of problems, but this sector is notoriously plagued with IT struggles that limit what they’re able to do with the wealth of data these public bodies hold.

Having a better understanding of public sector transformation and the factors holding this back can support governments to empower non-technical staff and gain valuable insights from their data.

Download this report now to learn how to maximise existing data assets, make your digital services even smarter and deliver cost savings.

Provided by

More than half of public sector leaders believe that their organisation’s data is stored on infrastructure that is not fit for purpose.

A survey of 200 UK public sector decision makers across central and local government, health and social care, emergency services, education, and defence organisations, commissioned by NetApp, found that cloud growth will accelerate and investment in skills will increase over the next 12 months.

While 56% of leaders think their data infrastructure isn’t fit for purpose, 47% of respondents believe that building a platform to implement more ambitious data strategies and deliver digital services will require additional investment.

However, 75% of public sector leaders believe that the National Data Strategy, which aims to consolidate the country’s position as a global digital leader through data, is imperative to the UK’s future economic growth. 73% of respondents said that it will help the UK cement its position as a global digital leader.

79% of public sector leaders also said that their organisation has a policy for how it uses data that is aligned to the data strategy. 87% also said that they will maintain or grow their expenditure in the cloud, with 53% expecting to increase usage in the next 12 months.

NetApp said the study shows that cloud and data strategies are maturing across the public sector. Although 79% of organisations currently store data in the cloud, only 30% consciously adopt a cloud-first approach. When modernising their infrastructure, leadership remain pragmatic in their selection criteria to ensure their choices deliver the best performance at the lowest cost, said NetApp.

Public sector organisations also acknowledged the need to adapt culturally and to upskill employees to deliver on the National Data Strategy’s aims. 69% said their organisation must change its culture to transform digitally, while 63% think further employee training and support from senior management is required to meet digital objectives in 2022.

What's more, 65% of leaders also believe digitally upskilling existing employees will help their organisation overcome shortages in technical skills across the UK.

“Public sector organisations experienced tremendous digital acceleration in response to Covid-19, and continuing the progress made by increasing their capabilities to use and share data will be fundamental to shaping the UK’s recovery over the next 12 months,” said Tim Skinner, UKI head of public sector at NetApp.

“Modernising infrastructure, continuing cloud migration, and equipping employees with sufficient digital skills will enable this, but the National Data Strategy is an essential piece of the puzzle, providing clear direction for creating a truly digital society.”

The government’s existing data strategy was insufficient to maintain the momentum built during the COVID-19 pandemic with regards to how local authorities use data to transform public services, warned the Centre for Data Ethics and Innovation (CDEI) a year ago. Despite that the pandemic had led to substantial positive developments when it came to the way local government uses data, it risked being undermined by a narrowly-focused set of measures that won’t guarantee any long-term positive changes as we move beyond the pandemic.

UltimateSuite offers four stages to its task mining, from data capture to pre-processing and enrichment, through to data analysis output, and they stand out from competitors with automation of much of the manual analysis that the more typical tools offer.

Download this whitepaper to learn more about UtlimateSuite and what this task mining tool can offer your business, as well as how easy it is to set up.

Provided by

Password manager NordPass has inked a new deal with cyber insurance provider Cowbell Cyber, which will provide its policy holders with discounted cyber security tools.

Cowbell Cyber, which provides custom-designed policies based on risk profiles, operates a continuous risk assessment platform that alerts policyholders to security vulnerabilities, improving risk posture and preventing potential incidents.

NordPass will now be incorporated into Cowbell Rx, Cowbell's referral platform for cyber security solutions and risk management, as part of the deal. Policy holders will also be offered a 15% discount on NordPass Business.

“With ever-increasing cybercrime, we highly encourage all of our customers to think about cyber insurance. Implementing and using a password manager, having Multi-Factor Authentication (MFA) in place, or having an import detection response tool - all of these things reduce your risk, and that ultimately may increase your chances of securing a cyber-insurance policy,” commented ​​Gerald Kasulis, head of business and channel operations at NordPass.

NordPass has also announced an upcoming free webinar on cyber insurance and ways to assess cyber hygiene. Participants will be able to hear about the biggest concerns they have regarding cyber-risks, as well as how they can improve their cyber security.

The webinar is being led by Arch Insurance Group’s VP and cyber risk product leader Shiraz Saeed, AmTrust Financial Services’ VP and head of cyber claims Andrew Lipton, and Cowbell Cyber’s head of claims and risk engineering Theresa Le.

“Together with our partners NordPass, Cowbell brings streamlined access to top cybersecurity solutions to current and future policyholders to maximize their ability to be secure,” said Theresa Le.

“We connect directly with trusted partners to improve the cyber risk profile of our policyholders.”

The database is the backbone of the enterprise. Data is the currency of modern business, and organisations need to ensure consistent, optimal database performance to keep their business running smoothly.

The consequences of compromised or even suboptimal database performance can be significant; including delays in responses to customer enquiries, sales processes, and transaction processing.

This report looks at how ensuring efficient and effective operations across multiple databases is critical to maintaining business continuity and performance.

Provided by

Google has expanded its open-source differential privacy (DP) platform to support the Python programming language, widening availability to millions more developers and data analysts.

The announcement makes Python the fourth language supported by the project after initially launching in 2019 with support for C++, Java, and Google-created language Go, sometimes referred to as Golang.

It comes after Google reported a significant number of developers contacting the company expressing their interest in using the open-source library for their Python projects. Google worked for more than a year with OpenMined on Python support and said numerous projects have already used its DP library, including Australian developers who have accelerated scientific discoveries through analysing medical data in a private way.

DP is a system used by data analysts to preserve the privacy of the individuals whose data is used in an analysed data set. Work to develop strong DP dates back decades, but only in recent years have tech giants such as Google and Apple embraced the system.

One of the key areas of DP development for Google in the past year has been on providing a tool for developers within the library to fine-tune the 'epsilon' - a mathematical measure of privacy. Finding the optimum epsilon requires a great deal of trial and error to perfect and having a tool within the library that allows developers to make adjustments to yield a lower epsilon, which indicates a more private release, means individual projects are able to be tuned as privately as possible.

Google said now Python is supported, the DP library is now available to nearly half of all developers worldwide which means more developers and researchers will be able to analyse data and make new discoveries while preserving the privacy of users to whom the data belongs.

Python is among the most popular programming languages currently in use and won 'Language of the Year 2021' from the TIOBE index, which ranks programming languages based on their popularity. Python is useful for a wide range of programming activities but is especially well-known for its capabilities in data analysis, making it a natural progression for Google's DP library.

As part of the launch, Google has released a new web-based product, pipleinedp.io, which allows any Python developer to analyse their dataset with differential privacy. Google also said it has seen organisations experimenting with new use cases such as showing a website's most visited web pages by country, in an anonymised fashion.

The library is compatible with leading large data processing engines, the Spark and Beam frameworks, and Google will be launching an additional tool to help users "visualise and better tune the parameters used to produce differentially private information".

"We encourage developers around the world to take this opportunity to experiment with differential privacy use cases like statistical analysis and machine learning, but most importantly, provide us with feedback," said Google announcing the news. "We are excited to learn more about the applications you all can develop and the features we can provide to help along the way.

"We will continue investing in democratising access to critical privacy-enhancing technologies and hope developers join us in this journey to improve usability and coverage. As we’ve said before, we believe that every Internet user in the world deserves world-class privacy, and we’ll continue partnering with organisations to further that goal."

What is differential privacy?

Differential privacy is a tool that has gained acclaim in recent years as data and identity protection have become focal points for researchers, businesses, and regulators alike.

Some argue it is fundamentally necessary in data analytics to preserve the privacy and hide the identity of people whose data is being analysed. For technology companies especially, it has been at the forefront of how their users expect them to handle the data they hold on others.

DP works by adding 'controlled noise' to datasets so that people cannot be individually identified by the data they provide to the dataset. For example, if residents of a neighbourhood supplied data for analysis involving their salaries which were then represented as an average, and one resident left the neighbourhood, their salary information could be tied to their identity by looking at the difference in the data pre- and post-move.

Similarly, if two databases were analysed, one with a single data point on 50 people and one with a single data point on 51 people, the analysis results for both would have to be indistinguishable from each other to avoid identifying that 51st person, in order to qualify as differentially private.

Adding controlled noise to a dataset would remove the possibility of identifying an individual by skewing the statistics just enough to remove the element of identification, without significantly compromising the accuracy of the results.

All major Big Tech firms have embraced DP in different ways. Microsoft's AI Lab works with Harvard University on projects to facilitate DP-enabled research. Apple has used DP on its products since macOS Sierra and iOS 10, and Facebook and Amazon also have experience working with the system too.

DataOps is the key to leveraging data for competitive advantage, but success depends on the establishment of a strong functional infrastructure.

The foundation for DataOps requires an integrated storage and data management platform to unify the capture, storage, movement, classification, ingestion, and protection of data to make it available for high-value data engineering and data leverage purposes.

Download this resource to understand how to tackle the challenges presented by data growth and how to overcome them for greater agility.

Provided by