HashiCorp has announced that it has changed its source code license to the Business Source License (BSL), restricting commercial use and sparking a degree of upset in the open source community.

The move follows similar changes by companies such as MongoDB and Elastic as companies founded on open source ideals try to overcome commercial difficulties.

In this instance, HashiCorp, which had formerly worked under a free open source license, has opted to move to a model that allows for copying, modification, and redistribution in non-commercial projects. Use by commercial vendors will only be allowed under specific conditions.

The reason for the switch, HashiCorp said, was due to too many vendors profiting from the work of open source software (OSS) projects - projects that often don't receive a cut of the profits their work allows others to generate.

“There are other vendors who take advantage of pure OSS models and the community work on OSS projects, for their own commercial goals, without providing material contributions back,” it said.

“We don’t believe this is in the spirit of open source.”

The decision is, however, also grounded in the realities of business. HashiCorp’s finances depend on its products and services and these have taken a hit in recent months.

In June 2023, the company suffered a substantial drop in its stock price following the release of its Q1 financial results for fiscal year 2024. While revenues increased, the workforce had been trimmed by 8% and the revenue outlook lowered to between $564 - $570 million for the full fiscal year 2024.

Making it possible for competitors to build their own software based on the company’s work is not a reality that will delight many investors. Couple this with its less-than-ideal financials from June and it’s clear to see that HashiCorp’s decision to restrict access to its source code makes complete business sense.

The rationale behind the move was echoed by other companies that have also decided to restrict their source code in recent times. MongoDB, for example, switched to the Server Side Public License (SSPL) several years ago also with a view to restricting the use of its code by other vendors.

Since making the decision, MongoDB’s business has fared well with its stock price rising more than 450% over the last five years, and its most recent financials indicating a 47% YoY increase in revenue.

Granted, the licensing switch won’t have been the sole influence over that large increase, but the company’s success will undoubtedly have been helped by its move to claim more money from its work in the DBaaS space.

HashiCorp has said it does not wish to move to a closed-source model, so has moved from the Mozilla Public License v2.0 (MPL 2.0) to BSL 1.1. 

While end users will still have access to the code, vendors providing competitive services built on HashiCorp’s community products will be prohibited from incorporating future changes.

Avi Press, founder and CEO at Scarf, praised HashiCorp’s approach to a problematic subject, describing the move as: “A difficult business decision carefully communicated”. 

“They are a well-meaning group of people who have shown they do care about OSS, whether or not you like their decision,” he added.

Open source outcry

HashiCorp’s move may make a great deal of sense from a business perspective, but the decision to adopt the BSL has upset some high-profile members of the open source community. 

Amanda Brock, CEO at OpenUK, for example, described HashiCorp’s statements on the BSL as “open washing”.

She went on the say: “The pressure of enabling their competitors with their innovations - an inevitability of open source - did not align with the need to generate shareholder value”.

Peter Zaitsev, co-founder and former CEO at Percona, was equally critical and went so far as to brand the switch to the BSL as “hostile”.

“HashiCorp is, of course, within their right to do these changes, but it is hostile towards the community who supported the company along the way,” he said.

It’s a comment that the company isn’t likely to take kindly. In its view, its source code remains available - keeping the spirit of open source alive - just restricted in a way that allows it to make some extra money from its work.

However, both Brock and Zaitsev raise valid points, since HashiCorp is joining a growing list of open source companies trying to find the fine balance between satisfying shareholders and venture capitalists, while also staying true to their open source roots.

Given the very nature of open source, that balance is almost always never perfectly found - often tipping in one way or the other. The polarization taking place within the open source community has been highlighted, and comparisons drawn between the likes of HashiCorp and MongoDB, with foundation-based projects like Kubernetes and PostegreSQL.

The latter two continue to use approved open source licenses, while others - commercial open source companies, both venture-funded and publicly listed - “continue to show their ‘commitment’ to open source as being no more than a marketing ploy to efficiently attract users and customers at early stages”, Zaitsev said.

And this is where the notion of becoming “hostile” towards the community that supported the likes of HashiCorp rings true. Where a foundation has sponsors or donors providing financial or technical support, companies such as HashiCorp are dependent on their products and services for revenues. 

Where rivals are making commercial use of the products, making a difficult business decision such as a change of license is inevitable.

Joe Duffy, CEO at Pulumi, also had harsh words for HashiCorp, saying: “This relicense is very disruptive across the entire market, and ultimately it is the practitioners who pay the price. 

“This new license is insidious and we do not consider it to be true open source given the extreme restrictions preventing an open marketplace”.

Adam Jacob, co-founder and CEO at System Initiative, was blunter still adding that “if Hashicorp had developed their open source community into a diverse and broad one, they would have been the lingua franca of the cloud”. 

“But since they failed to do that, the only rational move is to extract as much money as possible from what remains.”

Disappointment from the open source community aside, license changes such as those made by HashiCorp have a ring of inevitability about them, especially when uneasy investors are involved.

However, the messaging and license wording around the change requires standardizing. 

As Press noted, there are myriad ways for companies to restrict access by commercial competitors, and a standard path to a ‘you can do what you like with our code except commercially compete with us’ license has yet to be defined.

Experts have warned of a sophisticated peer-to-peer (P2P) worm, written in Rust, targeting instances of the open-source database software Redis.

The worm leverages a critical vulnerability to establish itself within a Redis instance, which is then assimilated into a wider P2P network used to spread the worm further.

After the worm infects a Redis instance, it creates a P2P connection via port 60100 to a large command and control (C2) botnet. This is used to download further malicious samples, after which the worm scans for other exposed Redis instances to infect.

Cloud researchers at Unit 42, Palo Alto Networks discovered the worm and named it ‘P2PInfect’ after a term found within leaked symbols in the worm’s code.

At the heart of P2PInfect’s infection chain is the exploitation of CVE-2022-0543, a sandbox escape vulnerability in the Lua Library.

The vulnerability’s maximum severity score of 10.0 on the CVSSv3 severity scale highlights the possibilities open to attackers if it’s exploited.

Flaws with the maximum severity score are rare, and they are often designated the maximum mark due to the high potential impact of an attack resulting from their exploitation, which can extend beyond just the vulnerable component.

Leveraging this specific vulnerability allows the worm to operate in cloud container environments, an unachievable feat for other worms aimed at Redis such as the cryptojacking malware operated by Adept Libra.

Once present within a Redis instance, the worm runs a Powershell script, which changes local firewall settings to prevent the infected Redis instance from being accessed by its owners, while opening a port to give the worm operators free access.

This is one of several sophisticated methods that P2PInfect uses to establish persistence on infected systems. 

Another is a process titled ‘Monitor’, saved to the Temp folder within a user’s AppData directory, which when run downloads multiple randomly named P2PInfect executables alongside an encrypted configuration file.

Although the samples downloaded from the threat actor’s C2 include two labeled ‘miner’ and ‘winminer’, Unit 42 was unable to establish any evidence of the worm performing cryptomining using infected instances.

Researchers posited that at this stage, P2PInfect may be laying the groundwork for a more active future campaign in which mining activity could be initiated using the botnet. As Redis runs on both Windows and Linux, it has a wide potential victim base.

This was supported by unexplained features in the C2 that allow the worm to update, and open the possibility of P2PInfect being granted new behaviors and features in the future.

Unit 42 discovered the worm on 11 July using its HoneyCloud platform, a geographically varied set of honeypots used to attract and analyze public cloud threats, and collected evidence to suggest that it is quickly spreading.

Of 307,000 publicly-communicating Redis instances observed by researchers, 934 were identified as vulnerable though the exact number of malicious nodes in the botnet has not yet been established.

“As the world’s most popular in-memory database, it’s no surprise that Redis installations are frequently the target of threat actors, and we are glad to see cybersecurity researchers actively working to find these bad actors," a Redis spokesperson told ITPro.

"We’ve previously seen other malware created to take advantage of CVE-2022-0543, a vulnerability created by how certain versions of Debian Linux package the Lua engine for open source Redis. Redis Enterprise software bundles a hardened version of the Lua module which is not susceptible to this vulnerability.

"As such, customers running Redis Enterprise licensed software are not at risk from CVE-2022-0543 and P2PInfect. Users of open source Redis are encouraged to use official distributions available directly from redis.io.”

Many ransomware groups have moved to Rust in the past year, for benefits such as faster encryption and more effective evasion of common detection methods.

But outside of malware, Rust offers a number of benefits over other programming languages. 

The NSA has recommended it for its ‘memory safe’ qualities in comparison to languages such as C++ which can contain exploitable flaws based on memory, while Microsoft has released kernel features written in Rust and advocated for widespread uptake of the language across the industry.

Huawei has claimed its new Cyber Engine database security system is "the best in the world" with a 99.9% detection rate for ransomware.

The system, which was launched on the opening day of the 2023 Mobile World Congress event in Barcelona, is described as a "last line of defence" by the Chinese conglomerate.

"The mechanism we have built for Cyber Engine, together with data storage, is the last line of defence, but it also should be the most effective way," Huawei's president of IT product line, Dr Peter Zhou, told journalists after his presentation.

"Our mechanism and also the algorithm we have developed within Cyber Engine is AI-trained to detect the fingerprint of the ransomware virus and whether or not there has been an attack on the system," he explained.

Once it has detected an attack, the system then informs the database to begin shutting down affected systems or systems that can be implicated. It does this with a technique called 'air gap' which is about isolating data sets.

"We honestly believe its ability to detect ransomware is the best in the world," Zhuo said.

Ransomware detection systems, whether AI-powered or otherwise, will tend to have a very high success rate, according to Jake Moore, cyber security expert at ESET, however, they have to work every time to remain secure.

"An intruder only has to infect once for the disruption to begin," he told IT Pro. "SMBs must always prepare for the worst-case scenario by simulating an attack to fully understand the localised risks rather than relying on any one tool.

"Threat detection and response are absolutely vital but cannot be relied upon completely. However small the percentage 'get out clause' is, it remains every information security controller's nightmare and therefore preparation is key to survival."

What's behind Huawei's growth in cloud computing?

Cyber Engine was just one of several new cloud computing capabilities Huawei revealed at MWC.

The Chinese firm also launched Blu-ray storage. Related to the storage of cold data, which requires low-cost archiving, Huawei said it has a longer service life and reduces the total cost of ownership for organisations.

The company also unveiled a multi-cloud storage service that deals with cross-cloud data transfers. This service is simply called 'Tiering' and categorises data to create a prioritised order for transfers to and from other cloud environments. Its aim is to reduce energy consumption as well as lower operational costs.

Another multi-cloud-based offering was the Cloud on Cloud solution, which will enable carriers to quickly build and operate their own cloud services with the capabilities and resources of Huawei.

One of the driving factors behind some of its cloud ventures, particularly its expansive database portfolio, is an expectation of increased data generation.

Dr Zhou spoke of yottabytes of data being created by 2030 - a storage volume equivalent to a quadrillion gigabytes. The company is keen to begin working on systems that handle vast amounts of data, with its AI-based security system acting as a first step.

A Dutch hacker has been arrested after reportedly stealing data belonging to 9 million Austrian citizens through a misconfigured cloud database.

The attack was initially discovered in May 2020 and concerned the Fees Info Service (GIS) - the organisation responsible for collecting TV and radio licence fees in the country.

It revealed at the time that it had suffered a data breach, with data previously stored by the GIS, belonging to Austrians, was discovered on a dark net marketplace.

The hacker, whose identity has yet to be revealed, was arrested in the Netherlands in November 2022, Austria's Federal Criminal Police Office (Bundeskriminalamt/BK) revealed on 25 January, as reported by Die Presse.

The BK said the GIS had hired an unnamed IT company based in Vienna to restructure its internal databases. The databases contained information on citizen locations to help it track anyone attempting to avoid paying a broadcast fee.

An employee belonging to the company reportedly used the GIS data during a test and left a database online without securing it. Investigators said the hacker found the data through a search engine 'that wasn't Google'.

The data is thought to have affected nearly all Austrian citizens, as it has a population of around 9.1 million. The information included names, dates of birth, and registration addresses, said Klaus Mits, department head for the Cybercriminal Police Office in the BK.

The police were alerted by New Zealand authorities that an individual was trying to sell the data on notorious online hacker haven RaidForums using the name “DataBox”. Investigators then secretly bought the data for an amount of money they said was in four digits.

The culprit’s identity was then confirmed after a German server used by the hacker to store the downloaded data was seized and analysed. Investigators also found that the money for the data was exchanged in a cryptocurrency which the police said was easy to recognise.

Austrian police then contacted the Dutch authorities, and together they determined that the hacker had downloaded other information, in addition to the 9 million Austrian records, taken from around 130,000 databases.

The data wasn’t only from Austria - it included records on individuals from the Netherlands, the UK, China, Colombia, and Thailand. The hacker was also selling health data belonging to patients located in these other countries.

"The rapidly growing cyber crime will continue to be fought with all vehemence and new methods in the future," said Gerhard Karner, interior minister at the Austrian government.

"This case shows how important and necessary investigations in cyber space are. Our investigators have the know-how and no perpetrator should be sure of being able to disappear into the anonymity of the internet.”

IT Pro has contacted NCSC for comment on the UK data involved.

What does this mean for Austrian citizens?

“This could happen to any other nation. They all depend on third parties, they all have insiders that may be unhappy, and they all have access to such data elements,” Andreas Wuchner, a former global CISO and current cyber security advisor, said to IT Pro.

“Similarly, they are all facing shortages of resources and budgets alongside a rapid shift to the cloud and self-service functionalities. It’s a perfect storm, so this breach shouldn’t really be a surprise to anyone.

“That’s not to minimise the concern that citizens in Austria and across Europe should be facing. This registration data could make it very easy to impersonate someone, register for digital services and so on,” he added.

“Paired with some criminal energy and knowledge of how to obtain letters being sent before they reach someone’s letterbox, it opens up lots of seriously concerning opportunities for malicious actors to use the data to make money, buy goods, or access sensitive information - like digital health records - that could open individuals up to blackmail.”

Rebecca Harper, head of cyber security analysis at ISMS.online, said that citizens should be aware that there is an increase in the risk around identity theft, fraud, and financial losses for individuals.

"It also raises concerns about government and private organisations' security and protection of personal information," said Harper. "Citizens must be vigilant and monitor their financial accounts and credit reports to detect suspicious activity.”

“This incident does highlight the vulnerability of personal data in the hands of government agencies and the potential for it to be accessed and misused by hackers, which is a meaningful conversation to have ongoing," Harper said.

"It's vital for governments to have robust cyber security measures in place to protect citizens' personal information and to ensure that any breaches are quickly identified and addressed.”

Data has become a competitive advantage, helping businesses anticipate and react to change. But with data in multiple formats and locations, costs quickly add up and expertise becomes a limiting factor, forcing businesses to fall back on spreadsheets and disconnected databases.

The cloud has emerged as a critical tool for businesses of all sizes to more easily access and synthesise data to gain greater insight and make better decisions--predicting trends, improving operations, and identifying new customers and business opportunities.

Download this eBook from AWS to learn:

• How the cloud can help you turn your data into one of your most important assets
• Why businesses of all sizes can benefit from better use of their data to gain insights
• How the cloud can help overcome common data challenges and accelerate transformation
• How to evaluate if your business would benefit from a cloud-based analytics strategy
• and What to look for in a cloud service provider

Provided by  AWS

The IRS accidentally republished 112,000 taxpayer data records in November after they were initially published as a result of a technical error earlier this year.

Blame for the incident has reportedly fallen on an outside contractor working on behalf of the IRS and tasked with managing a database for the government department.

The incident relates to the upload of 990-T forms which contain private information used by tax-exempt entities, including government entities and retirement accounts, to pay income tax on income that comes from specific investments or that which is unrelated to their exempt purpose, according to a letter sent to congressional leaders, Bloomberg Tax reported.

In September 2022, the IRS said that some Form 990-T data had been made available for bulk download in its Tax Exempt Organization Search (TEOS), which shouldn't have been made public. At the time, it removed the files and was set to replace them with updated files in the future.

However, this time a contractor reuploaded older files to the database with the original data, instead of new ones which ensured the forms were set to be kept private. The IRS had given the corrected data to the contractor on 23 November, but the contractor had not removed the old files from their system.

The IRS was only made aware of the files being available again on their website when a third-party researcher reached out. The IRS then told the contractor to remove them immediately.

Around 104,000 of 106,000 forms previously shared in September were reuploaded to the site. Some forms contained names or business contact information, and the IRS is currently contacting people affected by the data breach. When more than 100,000 forms suffer a disclosure, the IRS is forced by the law to inform Congress.

The revenue service is also reportedly reconsidering its relationship with contractor Accenture on this project. IT Pro has contacted the company for comment.

“The IRS is continuing to review the situation to identify opportunities to establish additional controls and strengthen existing controls to protect taxpayer information,” said US Treasury spokesperson John Rizzo.

KuppingerCole's Leadership Compass provides an overview of the market for database and big data security solutions along with guidance and recommendations for finding the sensitive data protection and governance products that best meet your requirements.

KuppingerCole named IBM Security Guardium an overall business leader in Database and Big Data Security Solutions. IBM was also again ranked as a leader in all three sections: product, innovation, and market.

With this in mind, download this report to look at how KuppingerCole measures today’s solutions and why good data security is so important.

Provided by

Datadog has announced its Database Monitoring service now supports Microsoft SQL Server and Microsoft Azure database platforms.

The expanded support enables engineers and database administrators to swiftly identify and resolve database performance issues including slow queries, incorrect indexes in SQL Server and/or Azure databases, and application bottlenecks.

"Microsoft Azure SQL Database and SQL Managed Instance are fully managed database services that feature built-in security controls, automated maintenance and are always up to date," said Ramnik Gulati, senior director, product marketing, data & AI at Microsoft.

"The expanded support of Datadog's Database Monitoring product further strengthens this collaboration by providing Microsoft customers with deep insights into their managed and self-hosted SQL Server, PostgreSQL and MySQL databases, enabling them to build and scale workloads with confidence," added Gulati.

Notable Datadog Database Monitoring features for Microsoft SQL Server and Azure database platforms include valuable query metrics, explain plan analysis, and centralized query, database and infrastructure metrics.

"We launched Database Monitoring last year because we wanted to help our customers reduce database costs, troubleshoot performance inefficiencies and increase collaboration between engineers and database administrators," said Omri Sass, group product manager of application performance monitoring at Datadog.

"By adding support for SQL Server and Azure database services, Microsoft users are better able to accomplish these goals and discover and implement the right database improvements while saving time communicating and reconciling information."

Oracle and Microsoft have teamed up to deliver Oracle Database Service for Microsoft Azure, designed to provide Azure users with direct and streamlined access to Oracle databases on Oracle Cloud Infrastructure (OCI).

The deal allows users to provision, access, and monitor enterprise-grade Oracle Database services in OCI in a way that is familiar, Microsoft said in an announcement.

Users can migrate or build new applications on Azure and then connect to Oracle’s high-performance, high-availability database services.

“Microsoft and Oracle have a long history of working together to support the needs of our joint customers, and this partnership is an example of how we offer customer choice and flexibility as they digitally transform with cloud technology,” explained Corey Sanders, corporate vice president, Microsoft Cloud for Industry and Global Expansion.

“Oracle’s decision to select Microsoft as its preferred partner deepens the relationship between our two companies and provides customers with the assurance of working with two industry leaders.”

The solution builds on Oracle Interconnect’s core capabilities to enable customers to easily integrate workloads on the cloud platform with Oracle Database Services on OCI.

In a matter of a few clicks, users can connect their Azure subscriptions to their OCI tenancy, with the service then configuring everything required to link the two cloud environments and incorporate Azure Active Directory identities.

There are no charges for using Oracle Database Service, Oracle Interconnect, or data egress or ingress when moving data between OCI and Azure, either. Instead, customers will only pay for the other Azure or Oracle services they consume – such as Azure Synapse or Oracle Autonomous Database.

Clay Magouyrk, executive vice president of Oracle Cloud Infrastructure, said the solution gives customers the ability to “easily test and demonstrate the value of combining Oracle databases with Azure applications".

“There is no need for deep skills on both of our platforms or complex configurations – anyone can use the Azure Portal to get the power of our two clouds together,” he said.

Business data/insights reside in databases, making them integral to business operations. As a result, IDC estimates that at least 75% of business applications interact with databases. With cloud-based application modernisation becoming the core strategy, databases also need to become cloud-ready and be flexible/cost effective, agile, and scalable to create and manage database operations.

Cloud is the infrastructure engine of digital business. But cloud adoption is moving beyond vanilla IaaS strategies.

The combined Public Cloud IaaS and PaaS market is forecast to have revenues of $400 billion in 2025 with a 28.8% CAGR for 2021-2025. Unstructured data analytics/data management and media streaming are forecast to be the fastest growing segments with CAGRs of 41.9% and 41.2%, respectively.

Deploying and scaling databases at the speed of applications and ensuring continuous security, patching, cloning, and data reduction capabilities is complementary to application modernisation on PaaS.

Provided by

Bespoke, data-rich and highly available applications are at the heart of the next wave of digital transformation. These applications will be key in giving enterprises a deeper understanding of their customers and their needs and helping them to build new digital experiences to engage those customers with.

They could provide real-time visibility and analytics across their business. Yet development of these applications is being hampered by issues relating to the provisioning and management of data architecture – in particular, the traditional data architecture being run from on-premises servers. Legacy database technology is holding innovation back.

Why is this? The fundamental reason is that traditional, SQL-based databases and the architecture they run on were designed to support the needs of very different applications, built for industries and markets that didn’t involve the same quantities of data or operate at today’s speeds. Traditional on-premises data architecture is slow to provision and slow to scale, which is a problem when you need to be able to scale upwards rapidly to meet peaks of demand or scale down to keep costs and complexity under control.

Enterprises are already handling larger volumes of data, and those volumes threaten to get larger still. IDC’s StorageSphere Forecast 2021-2025 predicted that worldwide data storage capacity would grow from 6.7 zettabytes in 2020 to 16.1 in 2025, meaning an increase of 240% in the infrastructure and services required to store and manage it. On the level of the individual enterprise, that means more servers and infrastructure to manage, as well as a wider range of different architectures to cover different applications and requirements. Server sprawl is becoming a serious issue.

This in turn demands even more of database administrators (DBAs), which is an issue when there’s a growing skills shortage in the field. A 2021 report by EFG Research puts DBAs within the top 10 most common IT roles facing shortages. The UK government’s May 2021 paper, Quantifying the UK Data Skills Gap, puts database management as the seventh most in-need skill for UK businesses. The growing complexity of the role is only exacerbating the shortages. As a recent article for Global Banking and Finance Review puts it, “It is common that database professionals have been tasked with developing a working knowledge of multiple database systems, but they might not know each of them well enough to be considered an expert, and that’s what is needed.”

DBAs are also having to contend with the growing challenges of database security in environments where instant, high-speed availability is crucial, and where more advanced applications in AI, analytics and machine learning need to handle complex datasets in real time. Ransomware threats are on the rise, while many organisations are struggling to build a robust data retention strategy. Overall, traditional architectures are proving difficult to manage and secure, while actively slowing down enterprises’ path to insight. Is it any wonder that many enterprises are looking for a smarter way to work?

As with so many IT challenges, the answer lies in the cloud. Back in 2019, Gartner predicted that, by 2022, 75% of all databases will be deployed or migrated to a cloud platform, and that only 5% of these would ever be considered for repatriation to on-premises hardware. Flexera’s 2022 State of the Cloud report shows data warehousing, relational database-as-a-service (DBaaS) and NoSQL DBaaS making up three of the four most used platform-as-a-service (PaaS) cloud services. What’s more, open source technologies are taking off. Gartner has also predicted that 70% of new in-house applications will be developed on an open source database management system (OSBDMS) or OSDBMS-based database-platform-as-a-service (dbPaaS).

Next gen apps need a modern architecture

These open source technologies and cloud-based database platforms are a better fit for this new generation of applications. MongoDB is built for high-speed analytics and innovation. You can spin up new applications and services without wasting precious time configuring and tuning the architecture. MySQL is fast, scalable, highly customisable and cost-effective. As an object-relational database system, PostgreSQL has powerful features that increase its flexibility and make it easier to organise and model data with a higher level of consistency. Meanwhile, Redis and Apache Kafka are ideal for caching and streaming data securely, at high speeds, for real-time applications, while OpenSearch provides fast access, analytics and search with integrated visualisation tools.

What these platforms all have in common is that they’re highly scalable, both up and down, and that they enable innovation, with rapid deployment, high security and high availability. What’s more, as they’re open source, there’s no vendor lock-in. While switching from one platform to another is never trivial, the architectural features and available tools actively support it.

Yet simply moving to a modern, cloud-based database architecture is no silver bullet; DBAs and IT teams still face challenges in managing and securing them, particularly with multiple systems in place, working with existing, traditional architecture within a hybrid estate. That’s why it makes sense to work with off-the-shelf and custom DBaaS solutions from a provider such as OVHcloud. OVHcloud Managed DBaaS solutions give you instant provisioning across an infrastructure designed for modern data architectures, with scalability, security and high availability built-in. The platforms are true open-source, with no vendor lock-in, and offer a market-leading, highly cost-efficient price/performance ratio, with transparent, predictable pricing.

Perhaps most importantly, there’s no need for any time-consuming infrastructure management, meaning DBAs and IT teams can spend less time and manpower just maintaining back-end operations, and more time delivering solutions that add value. OVHcloud manages all logs and metrics, backups, database management system (DBMS) installations and updates, not to mention operating system installations and updates. OVHcloud handles all the hardware and server maintenance, and is responsible for delivering availability, power and network access according to the terms of the SLA. Everything is in place to spin up new nodes – or spin them down – within minutes rather than days. OVHcloud has been one of Europe’s leading cloud and infrastructure providers since 1999, enterprises can be confident that they won’t run into CLOUD Act issues, establish data sovereignty and feel assured that they’re working with a company they can trust.

If the next stage of digital transformation depends on a company's abilities to develop and operate custom-built, data-rich apps, OVHcloud is working to support them, through DBaaS platforms that are agile, scalable and based on managed infrastructure. That gives enterprises the freedom to grow and move rapidly, and build the products, tools and services their customers’ demand.

Learn more about OVHcloud’s database-as-a-service offering

This report from O’Reilly Media and Amazon Web Services provides a concise overview of database options in the cloud, the changed role of database administrators (DBAs), and what it takes to deploy databases in the cloud.

The eBook explores the types of databases available, the differences between cloud databases, and key considerations when planning the migration of on-premises databases to the cloud.

Download now to learn:

• The pros and cons of database options in the cloud, including self-managed, and managed (traditional, cloud natives)
• How the cloud transforms a DBA’s role, including the new tasks a DBA needs to learn when cloud vendors assume traditional administrative tasks
• Steps required for migrating your databases to the cloud, such as planning, data transfer, and optimisation

Provided by

Search-powered solutions provider Elastic has announced an expansion of its collaboration with Amazon Web Services (AWS).

The pair will work together to build, market, and deliver seamless access to Elastic Cloud on AWS, while leveraging the cloud giant’s global footprint and wide range of services.

Elastic said the collaboration will enable easier migration to Elastic Cloud through expanded competencies, as well as simplified onboarding and new joint go-to-market initiatives.

“Through our collaboration, customers can take advantage of simplified data collection from AWS services, and cloud-optimised features that optimise their Enterprise Search, Observability, and Security solutions with Elastic Cloud on AWS," explained Steve Kearns, vice president of Elastic Platform at Elastic.

"Elastic is committed to delivering a proven subscription experience for our Elasticsearch service through the AWS Marketplace, and we look forward to continuing to work with AWS to help organisations get the most value out of their deployment and focus on what is critical to their business.”

In terms of expanded competencies, Elastic has been certified as an AWS ISV Workload Migration Program (WMP) Partner to support the migration of customers to Elastic Cloud on AWS. The firm has also earned multiple new AWS validated technical qualifications.

Elastic said it is also aiming to simplify onboarding to Elastic Cloud on AWS and will focus on offering enhanced integration for account creation and setup – reducing the initial setup time “from hours to minutes".

In fact, the pair have already completed more than 20 integrations to streamline data ingestion into Elastic Cloud on AWS, including integrations with AWS FireLens, Amazon S3 Storage Lens, AWS WAF, and AWS Network Firewall, along with the Elastic serverless forwarder.

Additionally, the two companies have committed to investing in joint go-to-market initiatives, which include a free seven-day trial via the AWS Marketplace.

Stephen Orban, vice president of AWS Marketplace, Partner Engineering, and ISVs at AWS, said the companies’ relationship is “rooted in listening to customers".

“Enabling frictionless migration and operation of Elastic Cloud on AWS and a streamlined subscription experience is just the beginning,” he said. “Through our expanded efforts, we are ensuring our joint customers have easier access to Elastic Cloud on AWS so they can focus on solving their most complex challenges.”

Microsoft Defender for Cloud users can access an early preview of Defender for Azure Cosmos DB, Microsoft announced on Tuesday.

Microsoft Defender for Cloud helps organizations identify and address weak points in their cloud infrastructure through cloud workload protection (CWP) and cloud security posture management (CSPM), the company said.

Defender for Azure Cosmos DB support has now been added to Microsoft Defender for Cloud, allowing for centralized management of database security across multiple cloud environments.

“Databases are constantly evolving to handle new use cases, incorporate more intelligence, and store more data, giving developers and organizations a wide range of database types to meet their varying needs,” commented Inbal Argov, senior product manager of Microsoft Defender for Cloud.

“Because aspects including architecture, capabilities, configuration options, and authentication methods are unique to each database type, so are the security threats—requiring custom security measures and protection capabilities to address the most common threats across databases,” added Argov.

The new Defender for Azure Cosmos DB proactively monitors Azure Cosmos DB accounts and protects them from a range of attack vectors, including attacks made from the application layer, SQL injections, unusual access patterns, compromised identities, and malicious insiders.

Threat detections are made using Microsoft Threat Intelligence, the Microsoft Defender SQL query analysis engine, and the Microsoft Defender behavioral models.

A fully managed NoSQL database, Azure Cosmos DB also helps facilitate non-relational data models with multiple SDKs and APIs, ultra-fast response times, and instant scalability.

The service can be enabled at either the subscription level or the resource level.

The database is the backbone of the enterprise. Data is the currency of modern business, and organisations need to ensure consistent, optimal database performance to keep their business running smoothly.

The consequences of compromised or even suboptimal database performance can be significant; including delays in responses to customer enquiries, sales processes, and transaction processing.

This report looks at how ensuring efficient and effective operations across multiple databases is critical to maintaining business continuity and performance.

Provided by

The cloud does more than just enable businesses to save costs. It also helps you ask the question, “what should we do differently and better?” By automating the modern data warehouse, you can spend less time managing data and more time enabling data-informed decisions, and ultimately transform your business.

Read this report to explore how automation can help you better understand your data and improve decision making—boosting innovation, productivity, and efficiency while reducing complexity.

Get more value from your data with data warehouse automation.

Provided by

Why require an already overburdened IT staff to manage disparate data services when a converged database can deliver the management simplicity and efficiency desired by everyone throughout your IT and data teams? Discover the benefits of having all your data reside in a common database, including enhanced performance, better security, simpler management, and faster deployment.

Read this ESG Showcase report to explore Oracle’s autonomous, integrated, and secure data-centric strategy.

Explore Oracle’s modern data platform strategy.

Provided by

A successful data management strategy isn’t about how much data you can store, or how fast you can move it, but about having it all integrated, available, accessible, and useable, wherever it sits, and whoever uses it. Discover an approach to data management that enables you to reduce costs, enhance performance, improve security, simplify management, and speed up deployment.

Read this ESG Showcase report to explore Oracle’s autonomous, integrated, and secure data-centric strategy.

Build a data-driven future.

Provided by

According to an IDC Perspectives report, 75% of IT management budgets are spent on manual database management. And the pressure on IT is only growing as they struggle to secure, monitor, and manage your data while making it accessible in real time for the business.

Discover how an autonomous database can help you transform your database operations into a modern cloud model that simplifies processes, secures all your data, lowers operating expenses, eliminates costly downtime—and ultimately enables your business to innovate more while using fewer resources.

Boost innovation and efficiency with an autonomous database.

Provided by

Data drives almost every business decision today. So, the more data your business collects, the more critical the need to analyse and interpret it—this is vital for making the right decisions, fast.

Read this guide to learn how augmented analytics—which includes embedded AI and the ability to present data in easy-to-understand visuals—can improve every area of your business. Whether it’s enabling proactive sales and marketing campaigns, improving financial reporting effectiveness, or increasing supply chain efficiency, this eBook will show you how to extract maximum value from your data.

Analyse and interpret your data with augmented analytics.

Provided by

Amazon Web Services (AWS) has announced three new database capabilities for business applications as part of this week's Re:Invent 2021 event.

Customers now benefit from a managed database service that customizes existing databases and operating systems, an optimized table class for Amazon DynamoDB for lowering storage costs, and a machine learning-powered service that diagnoses and remediates database performance issues.

According to AWS, the aforementioned database features will help businesses manage data cost-efficiently and at scale.

Amazon RDS Custom caters to business applications requiring customizable underlying databases and operating systems. The managed service automates administrative tasks such as capacity provisioning capacity, scaling, and backup, and is compatible with commercial databases including Oracle and MS SQL Server.

Furthermore, the Amazon DynamoDB Standard-Infrequent Access (Standard-IA) table class offers DynamoDB users a 60% reduction in costs for infrequently accessed table data. DynamoDB Standard table class, by contrast, offers up to 20% lower throughput costs than Standard-IA, making it the most cost-effective option for tables with high throughput.

Finally, Amazon DevOps Guru for RDS, a new machine learning-powered feature built into Amazon DevOps Guru, identifies and diagnoses performance bottlenecks and operational issues in a database, providing detailed recommendations on how to fix them.

NetApp SVP and GM of Cloud Volumes, Ronen Schwartz, said: “NetApp offers cloud services to enable organizations to easily run highly efficient, cost-effective relational database migration and operation programs from on premises to the cloud. However, some organizations running applications that require customization to the database environment and operating system have been unable to move to a fully managed database service in the cloud due to the customizations these applications require”.

“With Amazon RDS Custom, these organizations now have a managed database service for applications that require operating system and database customization. Organizations can run Amazon RDS Custom on NetApp ONTAP to benefit from advanced data protection, autonomous efficiencies, and continuous optimizations.”

Provided by

Organisations need the ability to back up their legacy data, and with the choice of open source and commercial databases available, you can choose the best option for your business.

Join Database & Solutions Specialists’, Domagoj Kovac and Baris Furtinalar, as they discuss options for running managed services for your SQL servers, which offer full control, automated backups and options for scale.

Download this webinar for expert guidance on best performance and migration in order to make your tech life easier.

Microsoft has warned thousands of its Azure cloud customers that their main databases have been compromised.

The impacted customers included some of the world's largest companies, according to cyber security researcher Wiz.

The vulnerability is in Microsoft's Azure Cosmos database and allows intruders to read, change and even delete customer information, according to Wiz. The researchers were able to find keys that control access to databases held by "thousands" of companies.

The chief technology officer of Wiz, Ami Luttwak, is former CTO of Microsoft's Cloud Security Group. Her team found the exploit, dubbed 'ChaosDB', on 9 August and notified Microsoft on 12 August.

"This is the worst cloud vulnerability you can imagine. It is a long-lasting secret," Luttwak told Reuters. "This is the central database of Azure, and we were able to get access to any customer database that we wanted."

IT Pro has approached Microsoft for comment, but it seems that it cannot change the access keys by itself, according to emails sent by the company to Wiz. The tech giant has reportedly agreed to pay the security researchers $40,000 for finding the flaw and reporting it.

In the email to customers, Microsoft said it has fixed the vulnerability, adding that there was no evidence the flaw had been exploited: "We have no indication that external entities outside the researcher (Wiz) had access to the primary read-write key," it said.

This latest disclosure comes just a few months after the SolarWinds hack, where actors suspected to be working for the Russian government stole Microsoft's source code and caused breaches and issues around the world.

Exchange email flaws were still cropping up last week, with the US government sending out a warning that customers needed to instal patches that were issued months ago because ransomware gangs were now exploiting them.

In our last tutorial, we explored the Postgres database, looking at how to set one up and use it to store some records. We used the Python programming language to access it and enter some records. A database isn't that useful on its own when you want to retrieve records, though, unless you're happy to repeatedly type Structured Query Language (SQL) commands into a terminal.

In this tutorial, we'll build a basic web app to retrieve those records using Flask. This is a micro-framework built using Python designed for building web applications; it handles the basic tasks of a web app, working out what pages to display and managing interactions with visitors. It's a basic platform to which we can add more functionality using third-party libraries, which makes it relatively simple to work with.

Begin by installing Flask using pip3 install flask. You can check the installation by dropping into a Python shell using python3 and then importing the package using from flask import Flask.

As a micro-framework, Flask doesn't come with built-in database interactivity, so we'll also need another tool to do that. In our last tutorial we used a database driver for Python called psycopg2 that allowed us to submit SQL commands to Postgres. However, it still required us to think more like a database.

Relational databases store data in tables and rows, whereas Python is an object-oriented language that treats things as objects. SQLAlchemy is an object-relational manager (ORM) that bridges the two, translating between Python's object-based model and the SQL syntax that psycopg2 uses to interact with Postgres. Install it using pip install SQLAlchemy.

We're not quite done with our setup yet, however. Because we're using the Flask framework, we must install yet another package called Flask-SQLAlchemy, which is an extension that enables Flask to work with SQLAlchemy. After installing it with pip install flask-sqlalchemy you can once again test it in the Python shell: from flask-sqlalchemy import SQLAlchemy.

Building a basic Flask app

Now it's time to begin working on our app, assuming that you've already created a database using our Postgres tutorial. Create a directory for your project, and within it create a file called retronerds.py. In this file, we begin with an import section for importing the packages we need, starting with the Flask package:

Next comes the configuration section where we define some parameters for our app. We start by simply declaring the app:

__name__ gives it the same name as our file, retronerds.

Next, we define the section of the app that describes our routes. These are functions based on which part of the web app we're accessing. To program a route, we use a concept known as a decorator, which is a function that wraps another function inside it, usually modifying its behaviour. This lets us define multiple functions to do different things inside a route.

For now, though, when a visitor hits the root directory of the site (the index page), the function inside the @app.route('/') decorator will display a message. Here is our route code:

At the end of our program is a section of code that runs the app in Flask:

Now we can tell Flask about our app directly in the terminal by entering FLASK_APP=retronerds.py. We will also put Flask into development mode so that it gives us debugging information and updates the web app on the fly when we save our file by entering export FLASK_ENV=development. Now we run it using python3 retronerds.py. If all is well, the server tells us:

That URL is on your own machine. Go visit it to see your welcome message.

Now we can connect our application to our database. To do this, we must configure the database connection for the app by adding this code to the end of the configuration section:

This sets a configuration variable that tells our Flask app about the Postgres database that we created in the last tutorial. It says that we're connecting to Postgres and then states our username and password, followed by the host that the database is on, the standard Postgres connection port, and then the database name. The second line defines db as an instance of SQLAlchemy that we can use to interact with our app. It becomes an object representing the underlying database.

We'll add the following code underneath our configuration section to create a database table and populate it with columns. If you already created the database table in the last tutorial this won't interfere with it, but we still need to define our class:

This code uses a SQLAlchemy class called Model, which maps directly to a table in Postgres. We use this to create a Python class called Game. This maps directly to a table in the retronerds database that we call games. We then use another object in SQLAlchemy called Column to create individual properties of the Game class that map directly to columns in the games table that it represents. We do all this via our db database instance.

Having described the table using our class, we then create it using SQLAlchemy's create_all() database method. If you run the app and use psql to check your database, you should see that it has now created that table.

If you followed the steps in our Postgres tutorial then your table already has some records. Regardless, we will add another record to show you how you can enter data directly using the SQLAlchemy interface rather than programming SQL commands.

To do this, be sure that you are in the same directory where you keep your retronerds.py file, because we need to access it in our code. Then we will drop into the Python shell using python3 and enter the following code, pressing enter after each line to run it:

We begin by importing the db and game objects that we defined in our retronerds app. Then we create an object representing a new record that we want to add to our table (remember that the Game class represents records in the games table). That object includes properties that will populate each field in the record.

Now we have to get that object into the database, which is a two-part process that uses SQLAlchemy's session method. We first ask it to add the record to the database, which stages it for processing. At this point you can still roll back the queued transaction because it has not yet been written to the database; The record is only written to the database when you call the commit command.

We can check to see that our new Game object made it into the database by checking Postgres directly via psql. We'll then adapt our Flask app to show us that last record. Adapt your routes section with this code:

For completeness, here is the full code listing for retronerds.py: The key line here is games = Game.query.all(). This queries the database and retrieves all of the records as a list. We then set up a loop that iterates over all the games in the list and prints out their details.

This brings us one step closer to a bona fide web app. We have the 'read' part of the create/read/update/delete (CRUD) functionality that makes a basic web app - but we still had to create the records outside of the web interface. In order to make it truly useful, however, we'll need to add the ability to create and modify entries in the database without leaving the app itself - in the next tutorial, we'll show you how to do just that.

Technology skill sets continue to be greatly sought after, with the highest-paying tech jobs in 2024 closely linked to skills shortages in key areas including cyber security skills, cloud skills, and AI skills.

According to the results of the surveys that informed Hays’ US and UK 2024 Salary Guides, this has led to a 3-5 % average salary increase across these technology sectors over the last year. 

Unsurprisingly, roles with the biggest demand from employers have seen greater salary rises. Some of these roles require formal qualifications but role-specific certifications can also help with job prospects – for example, prospective workers could weigh up a cyber security certification versus a degree. 

We’ve compiled a list of some of the highest-paying full-time roles from the most in-demand disciplines, with data based on figures from Hays’ 2024 UK Salary Guide and 2024 US Salary Guide. 

1. Cyber security architect 

Average salary US: $208,000/year

Average salary UK: £115,500/year

Cyber security architects are in charge of producing and maintaining an organization’s IT security systems and ensuring they evaluate vulnerabilities and risks. It’s important for them to keep on top of current trends in the cyber security landscape and keep staff in the loop on new procedures and standards. 

The responsibilities of a cyber security architect include running penetration tests, creating cyber security budgets, staying abreast of cyber security compliance obligations, and evaluating new technologies such as the impact of AI on security. 

They are expected to maintain good communication with their organization’s chief information security officer (CISO), to allow the executive to perform their demanding role with all the necessary security insights.

2. Solutions architect 

Average salary US:  $203,500/year

Average salary UK: £93,000/year

Solution architects develop and implement solutions that meet business needs, which involves working with stakeholders to understand business requirements and cultivate the right tools. 

Strong analytical and problem-solving skills are essential, and when it comes to technical expertise, employers look for experience with design and software development.

3. Enterprise architect

Average salary US: $171,500/year

Average salary UK: £95,000/year

While solutions architects evaluate business requirements and develop products or services accordingly, enterprise architects identify business needs in relation to external and internal factors including industry competition and the business’ own IT landscape.

Enterprise architects analyze current trends in technology architecture, perform gap analysis, educate IT teams about new frameworks like agile development. With an overarching view of the organization, they’re also often in charge of developing its IT roadmap. 

4. Cloud engineer AWS/Azure (UK) / Cloud solutions engineer (US)

Average salary US:  $152,000/year

Average salary UK: £67,500/year

Cloud engineers manage the maintenance of cloud-based infrastructure, allowing businesses to run their cloud applications and services. Cloud computing has become increasingly important for organizations of all sizes as digital transformation efforts have matured and the cloud engineer has become an all-important role at many firms.

Programming skills are crucial for this role, as well as a strong grasp of DevOps, Kubernetes and containerization, and database management.

5. Infrastructure specialist/architect 

Average salary US: $142,000/year

Average salary UK: £78,000/year

Infrastructure specialists/architects are involved in the development, configuration, integration, and maintenance of their firm’s IT hardware and programs. They are expected to regularly assess IT infrastructure to ensure it meets business needs and look for optimization opportunities such as eliminating legacy infrastructure.

They are well-versed in common operating systems, possessing critical technical problem-solving and analytical skills. Employers also look for strong SQL skills and CCIE or CCNP certifications.

6. Data scientist

Average salary US: $123,000/year

Average salary UK: £65,5000/year

A data scientist's main focus is to help the business assess the questions it needs to ask about data and to collect it in a manner conducive to this, in comparison with data analysts who answer these questions with data-driven insights.

They do this through the use of data mining, the creation of mathematical algorithms, and predictive analytics models for collecting, sorting, and analyzing information. Data scientists also need a firm grasp of programming languages such as Python, R, and SQL.

To become a data scientist, candidates will need at least a computer science degree but typically data scientists will hold a Master's degree or PhD. Data science bootcamps can also be a helpful path to reskilling within the sector.

7. Data analyst

Average salary US: $81,000/year

Average salary UK: £43,000/year

Data analysts are responsible for taking the data that organizations collect and looking for trends that will allow them to provide actionable insights. In today's sector, data analysts will need to be well-acquainted with machine learning (ML) and AI, as well as SQL.

They also need a strong understanding of data management, to raise insights from unstructured data. Strong analytical skills are crucial but, just as importantly, they must be able to effectively communicate their insights to the rest of the business.

The most in-demand tech skills

As mentioned above, the highest-paid tech jobs are primarily those connected to the skills that can help address gaps or an anticipated demand within the sector. For the UK specifically, this includes cyber (a 4% rise), architecture (4.1%) and cloud and data (both 4.3%) per Hays.

“With organizations collecting increasingly large amounts of data and opting for off-site storage solutions, cloud and data professionals have seen a significant uplift in demand,” says Amanda Whicher, director at Hays specializing in technology. 

“Cyber experts are equally sought after – especially in the current geopolitical climate – with employers seeking specialists who can review their security frameworks and protect against heightened cyber threats.” 

Are tech salaries stagnating?

Salary increases do look to be levelling out however, as the UK average was down from seven percent on the previous year. 

But they’re far from stagnant, with Hays UK reporting that over the last twelve months more than three quarters (78%) of UK technology employers increased salaries, with over a third (35%) of these by five percent or more.

In the US, Hays’ survey found that 81% of IT employers plan to increase salaries in 2024, with 39.4% set to raise incomes by 3-5%, and 12% planning to increase salaries by more than 10%.

“US salaries shot up a couple of years ago, but over the last year they have remained somewhat the same – which is high,” says Gene Smith, senior director, Delivery – Cyber, Projects, and Non-Core, Hays US.

“They were increasing at a very fast rate, which resulted in ‘the great resignation’. This caused companies to overpay many tech professionals and ultimately led to mass layoffs in the US tech business. Companies are now in cost-saving mode while levelling things out and seem to be focused on the best value.”

How tech unionization has affected salaries 

Worker are undergoing a shift, as unionization within the tech sector is on the rise. Conventional wisdom had been that, as well-paid workers, there was no need for collective bargaining. Increasingly however, employees are realizing compensation alone doesn’t guarantee fair treatment or a voice on the issues that matter. 

“Workers want more of a say in things like ethical product development, harassment policies, diversity and inclusion efforts – the list goes on,” says Doug Stevenson, technology consultant and founder of website BlinqBlinq. 

In 2023 the sector marked a rise in unionization efforts due to a number of factors including, but not limited to, the changing nature of work (i.e. remote and flexible working), widespread layoffs and employee feelings of being undervalued and overworked.

Levels of organization differ greatly between the US and UK with the former often facing roadblocks from major firms who remain staunchly anti-union and varying state laws on unionization. But we are seeing changes, as evidenced by Microsoft’s first recognition of a union in early 2023.

Forrester research manager Leonardo Bulgarelli Freitas adds that awareness is also an issue across the board. 

“Many tech workers are not aware that unionization is even a possibility. But there are focused groups trying to organize themselves, and if awareness grows, we could see more consolidation over the next few years.”

Postgres is one of the most popular online databases in existence. It powers countless web applications around the world. In this tutorial, we're going to cover how to set up and manipulate this relational database, creating a foundation for a future web application based around Python's Flask framework.

To start, let's install Postgres. The PostgreSQL project provides ready-to-use packages or installers for Linux, macOS, Windows, Solaris, and BSD, but on our Debian-based Linux system we'll just use the package manager by typing: apt-get install postgresql.

Installing the system should start Postgres running as a server daemon. Check this with systemctl status postgresql. If it isn't running, you can start it yourself:

If you want to stop it, then:

But don't do that just now, because we want to play with it. To start, we're going to become the postgres user. In Linux, you do that with a simple sudo -u postgres -i.

From here, we can access psql, which is a terminal-based interface to Postgres that lets us enter queries and see the responses. To do this, just type psql.

The default user doesn't ship with a password, but leaving it that way will trip us up later when we use Python to access the database. Let's use psql to create a password for Postgres. For demonstration purposes, we'll use one you'd never use in production:

If the system returns ALTER ROLE, you're all set.

Psql uses the \list or \l command to show the current databases.

postgres is the default database that the database is created with. The other two databases are templates. Postgres uses template1 when you create a database yourself, enabling you to define a standard schema by editing this template. template0 is a standard schema that you can use to replace template1 with an out-of-the-box template in case it becomes corrupted.

Rather than using the standard Postgres database, we'd like to create our own. We're going to document old video games, so let's call it retronerds.

\list your databases again and there it is:

Our video games database needs a table to contain its data. We'd like a title, the game's description, its year of release, a publisher, and a genre.

We'll start by connecting to our database. From within psql, type \c retronerds. Then:

id gives us an ID unique to each record, called a PRIMARY KEY. The VARCHAR instructions tell us the maximum characters allowed in that field. NOT NULL means that we can't leave that field blank.

Now, check that the table is there with a \dt (for database tables):

You can also examine the structure of the table - its schema - with \d games:

Note that because we used the SERIAL descriptor for id when creating our table, the schema tells us that it will automatically increase id every time we add a record. It has also set up the database to index the records using id.

So far, so good, but there's nothing in our table yet. Querying the table in psql with SELECT * FROM games; gives us:

Let's change that. We could use psql to manually add records at this point, but eventually we want to use a Flask app to control our database, which means getting a Python program to do it for us. To make that work, we'll use a python library called psycopg2.

Start a new terminal to keep your psql session alive in the original window. In our new terminal, with Python 3 installed, we'll make a virtual environment to support psycopg2. We don't technically need to do this, but it's a way to keep our system tidy and reliable. Think of it as a room where Python can keep all the libraries it needs for one project without cluttering things up for other projects. Python 3 uses the venv command for this, which we had to install manually on Ubuntu:

Then we'll create the virtual environment, calling it flask-app.

Then, 'enter' that room by activating the virtual environment:

Your terminal prompt will change to reflect the virtual environment that you're using. Don't forget to deactivate the virtual environment when you want to install Python libraries for projects other than this one.

For a production system, you'd install the library from source. For our purposes, we can use the Python package manager to do it for us.

Now it's time to connect to the database. Here's a Python program that will import psycopg2 and create a database connection before creating a nested list with three video game records. For the purposes of the exercise, these are currently hard-coded into the program, although future tutorials will look at cleaner ways to automate the process of adding more entries.

The program then loops through each of those records, building an SQL query for each one by using string formatting to insert the necessary values from each list into the query. The cursor.execute command queues each of these queries, and the conn.commit command writes them to Postgres:

Save this as insert-game.py and then run it with python3 insert-game.py. Now, pop back into Postgres and connect to the retronerds database using the steps outlined above. A quick SELECT * FROM games; reveals that our titles are all there.

Notice how our Python program didn't specify an ID for any of these records. The database schema knew to insert one each for them and increment it automatically.

We've set up Postgres and shown how to manipulate it using both the native psql tool and Python, via a special library. It's the basis for building a create/read/update/delete (CRUD)-style system that you could use to keep records of any kind.

Other things that you could explore in the future include adding more tables to create a more complex record system with more entities. For example, you might pull the publisher field into its own table, so that you could store more information about each publisher. You could then use a foreign key to link a game to one or more publishers.

However, a system like this also needs a front end so that people who aren't psql ninjas can use it, and in a future tutorial, we'll look at the basics of creating a Python application that will interact with the system in more meaningful ways, including being able to add extra records without directly coding them into a Python program.

Google Cloud is launching three database services across its analytical portfolio that aim to unify its data services.

Dataplex, Analytics Hub and Datastream are all 'previews' that aim to reduce silos and securely predict business outcomes in 'dynamic' digital environments.

Datastream is a serverless offering and replication service that enables customers to replicated data streams from Oracle and MySQL into Google Cloud products like BigQuery or Cloud Spanner.

The Analytics Hubs is a service where companies can create, curate and manage analytics tools. This includes sharing data insights, dynamic dashboards and machine learning models both internally and externally. Google said it builds on BigQuery's existing sharing capabilities, which have already proven popular with organisations.

The third new service is called 'Dataplex' and is described as an '"intelligent data fabric". It meshes the best of Google Cloud with open source technology to enables secure and rapid curation, integration, and analysation of data at scale.

"Data must be thought of as an ability that integrates all aspects of working with it," said Gerrit Kazmaier, VP and GM of Databases, Data Analytics and Looker, at Google Cloud. "Every industry is accelerating their shift of being digital-first as they recognise data is the essential ingredient for value creation and the key to advancing their digital transformation."

"At Google Cloud, we're committed to helping customers build the most powerful data cloud solution to unlock value and actionable, real-time insights, needed to future-proof their business."

Equifax is an early user of the new previews, working with Google to incorporate Dataplex into the company's core analytics platform. The firm is hoping to simplify its workloads and build a single interface for policy management and governance across all its analytics data offerings.

"Google Cloud has been a critical part of the Equifax journey, helping us protect our customers' sensitive and proprietary data," said Bryson Koehler, Equifax CTO. "Google Cloud allows us to create a rich, unified and trusted data ecosystem between business units and partnerships - one in which everyone gains immediate value."

Urgent reforms are needed after thousands of criminal records were accidentally deleted from an IT system that was first developed in the 1970s, an independent report has said.

In January, an engineer accidentally deleted 209,550 offence records, including fingerprint and DNA data, from the Police National Computer (PNC).

While all the records have been fully recovered, an independent inquiry, chaired by former Metropolitan Police chief Lord Hogan-Howe, has attributed the "critical incident" to both "human error" and failures at the management level.

However, the system in question, a mainframe computer used for background checks on suspects, was first built in 1974 and the report has also called for it to be fully updated or replaced. There is a planned replacement for the 47-year-old system, but that is not expected to be ready for another two years.

The PNC has had several updates since its launch but was slammed as "inflexible" in the report as it relies on a "diminishing skills base" of software engineers who know how it works. The deletion of the records was down to a single error in a piece of code, made while the engineers were trying to update the system, which was a legal requirement.

"The team who operate it have worked together over a long period of time," the report noted. "The expertise and closeness of the teams involved in running the PNC increased the risk that their work would be accepted rather than checked by a leadership that was in a poor position to challenge their decision-making."

The report has stated that management was too slow to act when the deletion of the data was first discovered and have also given the Home Office eight weeks to come up with a plan to either replace the PNC or to invest in substantial improvements.

The PNC is based in Hendon, North London, and holds personal records on around 13 million UK citizens, including data for arrests, convictions and even vehicle owners.

While low-code app creation and management may be Appian’s bread and butter, data and automation were the name of the game on day one of Appian World 2021.

During the opening day keynote of the annual conference, once again held virtually rather than in the US, CEO Matt Calkins unveiled the latest iteration of Appian Low-code Automation Platform. While it still has the standard low-code app creation and management features, the new platform also includes low-code data.

According to the company, low-code data overcomes a major hurdle for businesses when it comes to creating apps and processes that have really high functionality, namely integrating data. Until now, it’s claimed, this would have required developers to have a host of database skills, however low-code data enables them to access data in the same simple, visual way that they use to develop low-code apps.

“We've seen year after year that the hardest thing about building an application is the data," Calkins explained, speaking to IT Pro ahead of the conference. "It’s hard to access the data, it's hard to know where it is, it’s hard to find the thing that you need specifically, it's hard to get permissions, it's hard to do multi-database joins... it's slow. In the end, most people give up.”

This, he said, leads to significant underuse of data in most businesses, which sits in silos that have little or no visibility to each other and, because of the difficulty with integration, aren’t fully used by apps either.

With low-code data, Appian is making all of an enterprise’s data drag-and-droppable objects that can be integrated into the workflow in the same way anything else would be. This, the company says, allows its customers to make full use of their data across all apps, rather than having to take a narrow view of data one silo at a time.

“Traditionally, it has been really hard to do joins across multiple data sources because they don't have a foreign key to each other and so there's no index – You're just sort of mashing them together and maybe they don't really line up or you have different spellings or something,” Calkins said.

“We've solved it in a way that once you identify all the objects in your enterprise, and then Appian goes ahead and queries those objects and pre-caches an index to help you do the join. And then we update that cache periodically, to be sure that you're always getting a fresh join… and we're doing it all behind the scenes so that whoever makes the application does not have to be a data expert, or know what's joining where; they just need to say what they want,” he explained.

As well as low-code data, the new version of Appian Low-code Automation Platform also introduces robotic process automation (RPA) functionality. During his keynote, Calkin described automation as fundamental to business, saying it was “about time someone brought [it and low-code development] together”.

He explained that automation and application development are about the same thing – workflows – so it makes sense to be able to deliver them in one package.

“Low-code and automation belong together,” he told IT Pro. “The simplicity, the ease of having it all be a single source application is superior and advantageous.”

Just because RPA is now included in Appian’s platform doesn’t mean customers are obliged to use it, though.

“We're still very open, and we're happy to work with any RPA or AI leader,” he said. “But I think many customers are going to prefer to use our RPA, because there's no marginal cost for additional bots – it's very economical. And some customers will prefer to use our AI, because you don't have to send your data outside of the enterprise to get AI analysis, if it's packaged in Appian.”

The new version of Appian Low-code Automation Platform will be generally available in June.

The Walt Disney Company has extended its partnership with Amazon Web Services (AWS) as it continues the global rollout of its Disney+ streaming service.

Since it was revealed in 2017, AWS has been heavily involved in the development of Disney+, providing a range of services from data analytics to storage.

The extension will see Disney tapping into more than 50 different AWS services, such as machine learning, content delivery, and serverless computing, as it continues to launch in new regions.

The service has been built on a flexible, secured cloud video infrastructure powered by AWS with the two firms collaborating "closely" to balance loads and handle usage spikes with viewing peaking during premium content like The Mandalorian and the WandaVision. When it launched in November 2019, there were more than 10 million new sign-ups within the first 24-hours.

"Disney+ has completely reinvented what's possible in content delivery by challenging convention and using cloud technology to build a streaming product from scratch that had never been launched and marketed before on such a global scale," said Joe Inzerillo, the executive vice president and CTO, direct-to-consumer, of the Walt Disney Company.

"AWS has been our preferred cloud provider for years, and its proven global infrastructure and expansive suite of services has contributed meaningfully to the incredible success of Disney+."

Amazon Kinesis, which is a data analytics platform, and Amazon DynamoDB, a database that helps manage metadata and enables content to be watched on different devices, have both been extensively used by Disney to create and deploy Disney+. The company also uses Amazon Timestream to monitor the streaming platform and ensure that users continue to have access to the highest quality video content.

With the assistance of AWS, the Walt Disney Company has been able to rapidly expand Disney+ to 59 countries across North America, Europe, Asia and South America and there are plans to continue pushing its service around the world.

Open source database company Redis Labs has announced a late-stage $110 million financing round, bringing its valuation to over $2bn.

This is a Series G round for the company and brings the total funding it’s raised to $347 million. This doesn't include a secondary market transaction also announced this week, in which investors bought equity from each other.

In this funding round, the company snagged two new investors, Tiger Global and SoftBank's Vision Fund. Tiger Global invested in new Redis Labs equity, while also participating in the secondary market transaction along with SoftBank and existing investor TCV.

This is the company's largest round to date. Its Series F last August featured six investors and raised $100 million, while Series E financing in February 2019 raised $60 million, and Series D funding in August 2017 raised $44 million. Funding rounds before that, which extended back to its inception in 2011, raised funds in the low-double digits. Past investors have included Bain Capital Ventures, Francisco Partners, Goldman Sachs Growth, Viola Ventures, TCV, and Dell Technologies Capital.

Redis Labs sponsors the open source Redis database project that uses NoSQL technology instead of traditional relational database structures. One of its biggest selling points is multi-model operation that supports an array of data structures natively in the database, including lists, bitmaps, and sorted sets. It also features module-based expansions for use cases, including search and graphing.

The company has also been aggressively targeting the cloud database market. It offers database options for Azure, Google Cloud, and Amazon Web Services (AWS) and has designed the database so instances in different clouds can communicate with each other and on-premises versions. Gartner named it as a challenger in its Magic Quadrant for Cloud Database Management Systems in November.

Running a fair comparison between two databases that follow different approaches (relational vs document) is extremely difficult.

This benchmark compares PostgreSQL and MongoDB Community Serve to help developers choose between the two. Using three types of benchmarks, this paper analyses the capabilities and performance of PostgreSQL and MongoDB in the same data design, with different active dataset sizes, and tests the performance of transactions.

Download the paper to see which you should choose and a detailed breakdown of the reasons why.

Fill out the form below to access the free resource.

It’s important to have a layered security model that addresses physical security, network security, host access control, database access management, and data security, but this report will specifically drill down into the capabilities of PostgreSQL for securing data.

Download this whitepaper to get:

• A framework and a series of recommendations to secure and protect a PostgreSQL database
• Outlines for adding PostgreSQL security features to the AAA (Authentication, Authorisation, and Auditing) framework
• Tips for preventing an SQL injection attack

Fill out the form below to access the free resource.

There are many reasons to implement PostgreSQL. It’s lightweight, easy to install, scalable, and its feature set is continually growing.

Ranked the fourth most popular database management system in April 2019 according to DB-Engines, PostgreSQL has seen a sharp rise in popularity with developers over the past few years.

Download this tech guide to learn what other features and capabilities this open source database system has to offer, and how it compares to enterprise databases.

Fill out the form below to access the free resource.

Spare a thought for the humble database – as the demands of the modern web user have grown over the years, databases have been asked to shoulder an ever-increasing burden by providing near-instant access to applications and services with little to no room for error. These high demands have given rise to techniques such as database clustering, which allows for increased failover, more efficient load balancing and generally better performance.

At the same time, these trends have also driven the growth of non-relational database technologies like NoSQL. These databases are extremely easy to horizontally scale, are designed to process high volumes of unstructured data and are ideally suited for managing real-time transaction data across a wide network. Public cloud networks and private cloud deployments alike have both proven excellent hosts for NoSQL databases and layer on top of the benefits offered by database clusters to deliver even greater performance and scalability.

Using clustered databases within your data centre can have a number of benefits, but if you’re restricting the boundaries of a cluster to one specific site, you may be missing out on a whole host of potential advantages, including faster performance, greater stability and improved resilience. Thanks to modern database technology and cloud-computing platforms, database clusters can now span not just multiple locations within the same geography, but even multiple regions.

Challenges

This strategy brings with it some unique challenges, however. Availability is arguably the most important; according to a recent survey from VMware, more than half of consumers would abandon a company if they felt it wasn’t offering an acceptable digital experience, so service outages are to be avoided at all costs.

Outages can happen for a variety of reasons: a buggy software update can knock out a server node, a rack could suffer power failure, or in extreme cases, an entire data centre may go dark thanks to fire, flood or loss of network connection. Some issues will be quicker to resolve than others, but in an ideal world, your users should never even know there’s an issue.

Multi-site databases are inherently more resilient than single-site instances, as they have the ability to fail over to one of the other sites in the event of an emergency, but they’re not entirely immune to failure. A multi-site cluster where databases in two regions are fed by a data centre in a third location offers some failover benefits – if one of the regional databases goes down for any reason, then users can simply be routed over to the other. However, if the central location suffers an outage, neither of the two regional databases will be able to function.

Benefits

Using an active-active configuration when designing multi-site databases can help mitigate the impact of any localised outages, with all necessary data and services replicated across all regions and continuously synchronised. This adds more redundancy between racks and data centres, and helps keep availability high since as long as one site is still online, database requests can be processed locally. It also has the added benefit of aiding in disaster recovery, as the data from one of the other sites can simply be replicated once a failed node or cluster segment is back up and running.

At the same time, replication between multi-site databases can sometimes cause its own problems. If two database instances attempt to save different updates to the same field at the same time, it can cause a write conflict which brings the whole database to a grinding halt. This can be due to a number of factors, from misconfigured concurrency settings to high-latency connections between cluster sites.

The immediate impact of an unresolved database conflict is that no changes can be made until it’s been fixed, potentially locking users or linked services out of accessing it. However, it can also result in your database containing inaccurate data or losing data altogether. None of these scenarios are good, but they can be especially damaging in highly regulated industries where maintaining accurate data records is a compliance requirement, and ensuring your database is built to support strong consistency is key to negotiating these requirements.

On the subject of compliance, data protection rules will almost certainly need to be taken into account when architecting a multi-region database. Recent rules like GDPR now make it more difficult for organisations to transfer individuals’ personal data across borders, while certain public sector bodies and private companies will contractually obligate partners to keep data within specific countries.

How multi-site databases shine

There are, however, ways in which multi-region databases can be configured to meet compliance needs, automatically routing data to the relevant territories and ensuring that it doesn’t leave the specified region for any reason. The more granular you can make these rules, the easier it will be to meet compliance requirements, and some architectures are better suited to this than others. In some cases, this can even make your database faster, as transactions between two databases in the same region will be lower latency than transfers across different continents.

This brings us to the issue of performance. As previously discussed, modern users expect almost instantaneous access to services, which means databases need to be able to process operations and return results in less than a second. This demand is part of the reason why multi-site databases have become so common – the closer a customer is to the server their request is being processed by, the quicker the results will be delivered. Therefore, maintaining more database sites allows organisations to reach a higher number of users with sub-second latency.

However, this strategy can sometimes result in issues of load balancing. For example, routing users to the nearest available node in a cluster may not always be the optimum solution; regional spikes in traffic can result in one data centre in a cluster being placed under immense strain while the rest of the cluster remains under-utilised. In order to prevent this scenario, intelligent load balancing can be deployed to make sure that traffic is routed in such a way that no single node becomes overwhelmed, while still maintaining acceptable latency and performance.

These challenges, among others, make running a multi-site database a taxing task, but multi-region architectures are essential for any business that wants to operationalise data on a global, real-time scale. By utilising a best-of-breed technology stack, however, IT administrators can remove much of this headache, relying on proven tools to manage elements such as backup, failover, performance optimisation, load balancing and much more.

The key to unlocking value from your database stack is basing it on a solid foundation. Aerospike’s multi-site clustering architecture allows database managers to construct their toolchain in a way that works for them, integrating with industry-leading tools to offer maximum flexibility and sophistication. What’s more, it’s combined with an architecture which provides strong data consistency and high availability, with a footprint requirement that allows IT leaders to massively reduce their TCO.

To find out more about how Aerospike can help your business manage cross-region data at scale, click here

By now, the fact that data is the most important commodity for a modern business is old news. The value of data and the need for a solid data strategy is well established, and the benefits speak for themselves. However, one aspect that often gets overlooked in conversations about data-driven business is that this information needs to pass through a database in order to start generating value – and it’s here that many organisations run into trouble.

Running a database can be tricky, especially once you start scaling to multiple regions or high volumes of transactions being processed at the edge. There are countless ways in which an inexpertly configured and maintained database can start developing problems, most of which can have a significant impact on performance and availability. Here are some of the top problems that can creep into databases – as well as what to look out for and how to solve them.

Data ingestion

This may sound obvious, but in order for a database to be valuable, it needs to be able to ingest data – and this process can present its own problems for unwary organisations. For example, you need to think about what sources your data is coming from, and any integrations that may be required in order to get it from those sources into your database. These connections may require additional resources to build and operate, and their health needs to be monitored in order to prevent bottlenecks.

Another concern is at what speed data needs to be ingested – real-time data processing sounds attractive, and can be hugely advantageous for certain applications, but for databases which aren’t accessed regularly or where the contents aren’t time sensitive, real-time ingestion may be introducing unnecessary complexity and slowdown when batch processing would have been just as effective.

These issues can be circumvented by planning out and documenting the details of your database architecture as early as possible. As part of this, DBAs should also be thinking about what kinds of data they’re ingesting into a given database; for example, if you’re building a customer database to serve a mailing list, you can probably stick to basic details like name and email address, without worrying about age, physical address, height and shoe size. The smaller your database is, the faster it will run.

Database size

As we mentioned above, a lean database is a fast database – and by the same token, a larger database will perform increasingly poorly over time. This can often happen without being noticed, which is why it’s important to conduct regular audits and health checks on your database, both to monitor the size and to make sure that it doesn’t contain any unnecessary data.

Of course, what counts as ‘unnecessary data’ will differ from one organisation to another, and organisations should have specific policies governing what data to collect, which databases to house it in and how long to keep it for. Not only does this help keep the size of databases down by allowing for old data to be cleared through rules-based automation, it also helps organisations maintain a good compliance posture when dealing with personal information.

Query optimisation

One of the most pervasive problems that affects databases is poorly-constructed queries. This is sometimes the result of gradual tweaking as a database evolves over time, other times the queries were badly written from the word go. Whatever the reason, an inefficient query can have major impacts on performance and stability.

There are a few common culprits to look for when troubleshooting underperforming queries. For example, it may be using too many join operations, or be structured in such a way that it’s performing full index scans for a relatively simple task. While you can manually comb through your database operations looking for inefficiencies, this can be a long and time-consuming task for DBAs.

It’s often much simpler to use utilities like Quest SQL Optimizer, which can automate much of the process. Tools such as this allow you to scan your database environment for potentially inefficient SQL statements, test alternatives to determine relative performance and restructure indexes to improve overall database speed.

Database security

The database is the crown jewel of any organisation’s IT. It’s the place where your customer records, financial information, access credentials and any amount of other valuable information is kept – which makes it a tempting target for hackers. Unfortunately, databases are also one of the elements of a company’s infrastructure which is most prone to developing security holes as it grows. Therefore, it should be among the top priorities for security teams to protect.

We’ve already covered the importance of regular auditing from a database health perspective, but it can also have a big impact on security. With Quest’s Change Auditor, admins can track data-level events such as changes to rows, triggers and tables, as well as alterations to access permissions - another area that must be closely monitored in order to ensure good security.

Resource consumption

According to Gartner, three-quarters of all databases will be deployed in the cloud by next year, but while cloud can be great for simplifying the operation and maintenance of your databases, that doesn’t mean that you can forget about hardware capacity and configuration altogether. If you’ve over-estimated the compute and storage needs of your cloud database in the initial planning stages, it’s easy for running costs to rapidly spiral. Similarly, if the needs of your database change over its lifetime, you may find that capacity has to be expanded as your database grows.

This is why it’s important to keep a close eye on the performance and resource consumption of cloud databases, optimising your storage, compute and networking configuration periodically to ensure that costs and performance are matching up to expectations. Tools like Quest Foglight Evolve can help organisations to set more accurate estimates of necessary cloud service tier requirements prior to migration of a virtual machine or container on which the database resides, and costs of that workload can be easily tracked on an ongoing basis once a migration has been completed, all within Foglight.

The government’s existing data strategy isn’t enough to maintain the momentum built during the COVID-19 pandemic with regards to how local authorities use data to transform public services.

The pandemic has led to “substantial positive developments” in the way local government uses data, according to the Centre for Data Ethics and Innovation (CDEI), which has witnessed data-driven innovations in public health and service delivery.

Hackney council, for example, combined internal and external datasets for the first time to help them identify residents particularly vulnerable to COVID-19 as an illness. The VIPER tool was used in Essex, meanwhile, to enable emergency services to share data in real-time.

Much of this innovation has been driven exclusively by COVID-19, which has forced the hand of the public sector to innovate and experiment. This reflects a pattern in both the public and private sector, with the pandemic forcing organisations to adapt in order to continue delivering services that are essential to their end-users.

“Almost every aspect of local government has required at least temporary reform during the pandemic,” said board member for the CDEI, Edwina Dunn.

“Data and data-driven technologies have played an important part in enabling local councils to respond to the COVID-19 crisis, helping to inform public health measures, protect the most vulnerable in local communities, and keep public services running.

“With the right support, councils can retain and build on efforts to utilise data effectively, in a way that is in keeping with the expectations of their residents, to provide local services communities can rely on.”

Dramatic improvements

The CDEI hosted a forum with several public sector groups, including local authorities, to establish how data was being used at a local level, and how this could be improved in future.

The report also found that local authorities have had more success in changing how they deploy existing datasets than in acquiring or sharing data with others while concluding the scale of change was more vast with authorities with dedicated data teams.

The sharing of health data has also improved dramatically across the board. The government, for example, has granted councils access to the shielding patients database, which has allowed them to deliver support such as food parcels and pharmacy deliveries.

All progress that local authorities have made in 2020, however, is being undermined by a “narrowly-focused” set of measures that won’t guarantee any long-term positive change as we move beyond the pandemic.

The Local Government Association (LGA) recently published a guide on using predictive analytics for local councils, for instance, while the Ministry Ministry of Housing, Communities and Local Government (MHCLG) is running a COVID-19 Challenge Fund. As part of this latter scheme, 11 projects have been awarded a share of £800,000.

'Fragmented and rudderless'

Although these are positive steps, on their own, the CDEI concluded, these are “unlikely to move the needle in how local authorities use data”.

One widely-shared anxiety among local government leaders is that the emergency powers granted during COVID-19 will be rescinded once the pandemic has subsided.

There’s particular concern that local authorities will lose their appetite for experimentation once the emergency subsides and with it their willingness to experiment with untried data-driven interventions.

There is, however, some hope in the way that these schemes, in combination with the government’s new National Data Strategy, shows a desire to make more support available for fostering innovation at a local level.

Launched in September 2020, the strategy signals a growing conviction within government to innovate using data, the CDEI claims, promising to strengthen skills, improve standards and bring clarity to regulations.

This strategy, however, was described as "fragmented and rudderless" shortly after its launch by the influential Public Accounts Committee (PAC) of cross-party MPs.

SAP’s new long‐term ERP platform is moving to HANA, and existing SAP ERP customers should start planning today for their migration to SAP S/4HANA from their current R/3 or ECC solutions. IBM has a long history of supporting HANA on IBM Power Systems with high performance, scalability, and reliability.

In this report, read more about migration to SAP S/4HANA and why IBM Power Systems is an excellent choice.

Hospitals around the country will soon be able to use artificial intelligence (AI) to help diagnose COVID-19 patients and reduce the time needed to create treatment plans.

The 'COVID-19 Chest Imaging Database' (NCCID) has been pulled together by NHSX, the unit tasked with the organisation's digital transformation. The database includes over 40,000 CT scans, MRIs and X-rays taken during the course of the pandemic.

It is hoped that the database can speed up coronavirus diagnosis and help assess whether a patient will be likely to end up in a critical condition. Currently, the rapid rise in cases has put a strain on most of the country's hospitals, with reports that a person with COVID-19 symptoms is admitted every 30 seconds.

The health secretary, Matt Hancock, said that the NCCID was "a testament to how technology can help save lives in the UK".

An algorithm based on the NCCID images is currently being developed by clinicians at Addenbrooke's Hospital in Cambridge, which will help to inform more accurate patient diagnoses when they present to a hospital without a confirmed test.

Chest scans with visual signatures of the virus will be compared to patterns in previous scans held in the NCCID. With an understanding of the earlier stages of the disease, clinicians can determine appropriate medical intervention and reduce the potential for complications at a later stage.

"We are applying the power of artificial intelligence to quickly detect disease patterns and develop new treatments for patients," said Dominic Cushnan, the head of AI imaging at NHSX. "There is huge potential for patient care, whether through quicker analysis of chest images or better identification of abnormalities."

The NCCID is just one avenue of AI that NHSX is looking into. The NHS AI Lab has also launched a £140 million AI award for innovations that can bring benefits across the organisation.

Oracle is set to move its company headquarters from Redwood City, California to Austin, Texas, according to a report from Bloomberg.

The news comes after another erstwhile Silicon Valley stalwart, Hewlett Packard Enterprise (HPE) announced that it too would be building and moving to a huge campus in Houston. That company already has locations in Austin and Plano, near Dallas.

Oracle said in a regulatory filing that the move from Redwood City to Austin “means that many of our employees can choose their office location as well as continue to work from home part-time or all of the time”. The company will, however, maintain its former headquarters.

"We believe these moves best position Oracle for growth and provide our personnel with more flexibility about where and how they work," the company said in the filing.

The coronavirus pandemic has prompted Oracle to offer employees flexible working arrangements and remote working.

The Bloomberg report said that the company has been in the process of moving away resources from California since 2018. That year saw the opening of its Austin, Texas campus that featured employee apartments.

Texas governor Greg Abbott welcomed the company in a tweet. “Texas is truly the land of business, jobs, and opportunity. We will continue to attract the very best,” he said.

Oracle is not the only company to be moving to the Lone Star state. Tesla founder Elon Musk is also moving to the state to concentrate on major projects his company is involved with, such as SpaceX. Texas also has a lower cost of living and no state income tax.

Amazon Web Services (AWS) has announced the general availability of its new visual data preparation tool that lets users clean and normalise data without having to write code.

Built as part of its AWS Glue service, the new DataBrew tool aims to make visual data preparation more accessible for a greater number of users.

According to AWS, DataBrew facilitates data exploration and experimentation directly from AWS data lakes, data warehouses, and databases. Its users will be able to choose from over 250 built-in functions to combine, pivot, and transpose the data, with the tool also providing transformations that use advanced machine learning techniques such as natural language processing.

DataBrew is serverless and fully-managed, the claim being that users will never need to configure, provision, or manage any compute resources directly.

“AWS customers are using data for analytics and machine learning at an unprecedented pace”, commented Raju Gulabani, AWS vice president of Database and Analytics. “However, these customers regularly tell us that their teams spend too much time on the undifferentiated, repetitive, and mundane tasks associated with data preparation. Customers love the scalability and flexibility of code-based data preparation services like AWS Glue, but they could also benefit from allowing business users, data analysts, and data scientists to visually explore and experiment with data independently, without writing code.

"AWS Glue DataBrew features an easy-to-use visual interface that helps data analysts and data scientists of all technical levels understand, combine, clean, and transform data,” he added.

AWS Glue DataBrew is generally available starting today in Ireland and Frankfurt, Germany, as well as select parts of the United States, including Ohio and Oregon, and the Asia Pacific Region. AWS said that it will announce the availability in additional regions “soon” but has yet to confirm when the tool will arrive in the UK.

When it comes to pricing, AWS said that the DataBrew users will not be faced with any “upfront commitments or costs” to use the tool, but will be expected to pay for the ability to create and run transformations on datasets. AWS did not immediately respond to IT Pro’s query regarding specific pricing.

Cockroach Labs just released CockroachDB 20.2, the latest version of its cloud-native, distributed SQL database. It includes new capabilities for spatial data, a new CockroachDB for Kubernetes option, and enterprise backup and restore functions for free users.

The open-source and freely available CockroachDB is the first cloud-native distributed database to include spatial data types and associated libraries. It’ll support net new workloads for Internet of Things (IoT), transportation and environmental applications, and includes a geodistributed database with geospatial capabilities. The syntax is compatible with PostGIS, an open-source software that adds support for geographic objects to the PostgreSQL object-relational database.

CockroachDB on Kubernetes combines the cloud-native database with a custom operator optimized for orchestrated deployments. Its distributed, share-nothing architecture enables users to attach storage however they want, as CockroachDB manages the scale, resilience and distribution of data. The included Kubernetes operator simplifies deployment configuration and ensures no-downtime rolling upgrades in production.

The CockroachDB 20.1 update brings basic distributed backup and restore features that were previously available only in the enterprise version to CockroachDB Core, the free, open-source version of the database.This update ensures companies’ disaster-recovery plans are in place and they can freely build high-performing production applications.

The latest version of CockroachDB extends its TPC-C benchmark performance to 140,000 warehouses, improving its throughput to 1.7 million transactions per minute (tpmC), a 40% improvement versus 2019. The TPC-C benchmark serves to simulate e-commerce or retail environments.

CockroachDB 20.2 other updates include:

• The introduction of Pebble, an open-source storage engine that provides performance and stability improvements, in place of the old storage engine.
• The database’s new monitoring UI displays key troubleshooting metrics for developers to query performance, including an SQL transactions page and a database sessions page.
• It adds new SQL functionality, such as partial indexes, materialized views, enumerated types, and user-defined schemas.
• It gains enterprise-grade security with logging, Role Based Access Control (RBAC) and certificate management.
• Adds support for Java tools Hibernate, MyBatis, Spring Data JPA, and Spring Data JDBC; and the Ruby tool Active Record.

"We're committed to making CockroachDB the database of choice for developers everywhere," said Spencer Kimball, CEO and co-founder, Cockroach Labs. "We've listened to our users, and we've made improvements to all areas of the database, including a new package, CockroachDB on Kubernetes, to ease deployments in cloud-native environments. No matter what the use case, we want to equip every developer with the database they need to build and create their business's next billion-dollar idea."

Comcast, DoorDash, eBay, Nubank, JPMorganChase, SpaceX, and hundreds of other companies use CockroachDB for transactional cloud-based applications. CockroachDB builds on the traditional relational databases PostgreSQL and provides elastic scale, indestructible resilience and low-latency distributed performance.

"LaunchDarkly empowers development and business teams to control their software. Our customers depend on our services to be available, fast, and correct from anywhere on the planet," said Justin Caballero, senior distributed systems engineer at LaunchDarkly. "We chose CockroachDB for our next gen database because of its consistency and flexibility, and the control it affords over our data."

As businesses adopt SAP HANA to run their core applications, the server environment needs to scale on demand with high reliability. IBM Power Systems scale large HANA environments with flexibility to easily accommodate changes in business needs while increasing application availability and systems admin productivity.

Customers told Forrester that moving to IBM Power Systems reduced the cost to purchase, manage, and maintain systems to deploy SAP HANA, so Forrester Consulting conducted a Total Economic Impact™ (TEI) study and examined the potential return on investment (ROI) enterprises may realize by deploying Power Systems for SAP HANA.

The purpose of this study is to provide readers with a framework to evaluate the potential financial impact of the IBM Power Systems for SAP HANA on their organizations. To better understand the benefits, costs, and risks associated with this investment, Forrester interviewed several customers with experience using IBM Power Systems for SAP HANA.

As a tech person, there’s nothing quite as infuriating as seeing people dispensing opinions on technology that have quite clearly been pulled directly out of their posteriors. We saw it with US and UK governments’ Huawei bans, and it’s now rearing its head again with the news that England’s much-vaunted track and trace system neglected to count around 16,000 cases due to a technical error.

This technical error is currently believed to be the result of using an Excel spreadsheet to store case records, which hit the maximum number of columns and then stopped updating. These reports are unconfirmed at time of writing, but this hasn’t stopped political commentators from flocking to Twitter to take potshots at the government for ‘wasting billions’ on a track and trace system that uses ‘a consumer app’ to store records. Excel, so they say, isn’t a database system, and the government is wrong and stupid for using it as such.

While I’m no fan of the current government or it’s handling of the coronavirus crisis, this grinds my gears somewhat. As any IT person worth their salt will be able to tell you, Excel is actually a phenomenally powerful database tool when used in the right way. With the right formulas and macros, it can do just about anything, and it even includes support for highly advanced tools like SQL queries.

Many databases run quite happily as Excel spreadsheets, and suggesting that Excel is somehow incompatible with these use-cases is simply incorrect. What’s more, it does a great disservice to the extreme flexibility and extensibility of Microsoft’s industry-standard spreadsheet package. Excel is, in many ways, the swiss army knife of enterprise IT, able to turn its hand to a truly staggering array of applications.

However, some situations require more than a swiss army knife – and it’s here where the jibes have hit upon a grain of truth. If the track and trace programme has been using Excel as its main platform for storing records of cases (although remember that this is still yet to be confirmed) then the problem isn’t with using Excel as a database platform, it’s with using Excel as a platform for such a large and rapidly-changing database.

The fact is, flexible though it may be, Excel is only an effective tool for managing databases up to a certain size. Quite aside from the problem of column limitations, you start running into performance issues once the database grows too large. Not only that, but fully-fledged database management platforms like SQL Server offer far more fine-grained control and querying capabilities, as well as more integration and processing options.

The IT industry has spent years perfecting database technology in a dizzying array of permutations and combinations, including SQL, MySQL, PostgreSQL, NoSQL and many, many more. They’re the building blocks of enterprise software development, and there is a wealth of knowledge and support out there for those that wish to use it, not to mention legions of qualified database administrators to run them.

To my eyes, what this indicates is a system that has been hastily cobbled together, using the tools that were immediately available to hand; a bodge-job using components that were just about good enough to serve the intended function, rather than the ones that were best-suited to it. You can accuse the government of incompetently building a flawed system using poorly-chosen tools by all means, but laying the blame at the feet of Excel and dismissing it as a spreadsheet programme that shouldn’t run databases is similarly short-sighted and ignorant.

While databases are one of the foundational building blocks of enterprise IT, many organisations may find themselves blindsided by the difficulty of running them effectively. Poorly-optimized databases can become serious time sinks, incurring unnecessary costs and distracting IT teams from other important tasks.

Luckily for database administrators (DBAs), there are various tricks and techniques that can be employed to improve and speed up database management and create more efficient and cost-effective systems. Here are our top tips.

Automate management tasks

DBAs’ duties cover an enormous range of tasks, including everything from backup and recovery jobs to storage management and configuration – many of which are highly mundane and repetitive. This makes them ripe for automation. By automating these simple tasks, DBAs can free up their days for more rewarding, interesting and innovative activities that are a better use of their valuable time.

There are many options when it comes to automating database management tasks. Many DBAs will choose to write their own custom tools crafted to their specific needs, while others may rely on the automation tools that database management software vendors have built directly into their products. 

Whichever way you choose to go, automating elements of your database management can help you save both time and money, as well as reducing recovery time in the event of any failures.

Deploy containers and virtual machines

Virtualization and multi-tenancy are nothing new in the worlds of enterprise IT and software development, but they can have big advantages for database management. If you’re running your databases on single-tenant servers, then porting them to VMs could bring big savings. 

For starters, this allows you to host multiple databases on the same infrastructure, which can cut down on licensing and hardware investment costs. It also offers more flexibility in porting, replicating and modifying databases, although some may find vertical scaling becomes challenging.

If you want even more portability, you may wish to consider containerizing your databases. This works well as part of a DevOps environment and integrates nicely with both CI/CD pipelines and multi-cloud architectures. 

The lightweight nature of containers means you can run large amounts on a single server, and they also lend themselves to high levels of automation which (as we covered above) can have big benefits for cost and efficiency.

Optimise SQL queries

Not all database queries are created equal. In fact, inefficient or poorly structured SQL queries often take longer to run, and when run on live production databases they can introduce errors and cause outages for other database users. By putting in a little more legwork in fine-tuning their initial queries, database operators can speed up their operations to improve the long-term health and performance of their databases.

For example, when selecting fields as part of a query, it can be tempting to use SELECT* to quickly select all records, or using SELECT DISTINCT to identify unique results. However, this can lead to unnecessary processing if it’s being run on large databases. 

Instead, thinking about the results you’re looking for and structuring your query around those specific fields will cut down on the processing power required to run it. Similarly, using INNER JOINs instead of Cartesian Joins made using WHERE clauses can massively reduce the amount of work being done by the system.

Manage database growth

One of the biggest factors that affect both the performance and speed of a database is how large it is. The bigger it is, the longer it takes to search through it and deliver results from a query. Size also plays a factor in process and transaction costs for cloud-hosted databases, or in hardware upgrade cadences for on-premise systems.

The best way to ensure that your database doesn’t expand at an unsustainable rate is to set up your schema and validation rules so that it contains only data which is going to be necessary to its operation. Storing large amounts of infrequently or partially used records can increase the cost and the time it takes to run queries.

Retire old or unnecessary data

As we’ve just covered, keeping your database streamlined is a key part of making sure they remain efficient and cost-effective. However, even if you’re restricting database inputs to the bare minimum of fields that need to be there, the size of your database will inevitably swell over time. 

If you’re finding that your database is getting a bit unwieldy, it might be worth seeing if you can streamline it by removing old entries that no longer need to be in there.

The specifics of what records can safely be deleted and when will depend on the specific purpose of your database, but setting clear data deletion policies can help keep its size to a manageable level. Another added bonus is that this can ensure that your queries aren’t returning as many irrelevant junk results, and may even help with GDPR compliance in the case of databases containing personal information. 

Large tables can also be partitioned to help stop them from getting too big, and indexes should be monitored to ensure they're still being used; any that aren't can be removed to free up additional space.

Time your queries

Databases have many advantages, including that many of them can be used by a number of different applications and users for different things, and that many are multi-purpose. Despite this, it is necessary to think about the impact queries can have on performance for the rest of the business. If you’re running large, complex queries, for example, that need lots of processing power, it may cause other applications and queries to run slower.

Are you planning to run any particularly heavyweight queries? It may be worth considering how time-sensitive they are as if it isn’t urgent, it might be worth organizing them to run outside of peak business hours when there will probably be less activity on the database. This helps to reduce the impact on the database’s efficiency and keep disruption to a minimum.

Perform regular audits

At the end of the day, it’s important to remember that none of the tips detailed above will work well if you don’t perform regular audits on your databases. Just like humans need health checkups, databases also need to be thoroughly inspected from time to time in order to ensure that everything is running like clockwork. 

As your organization grows and evolves over time, so does the usage and purpose of its databases. Therefore, these regular audits should include monitoring elements such as input rules, data deletion policies, schema, and usage patterns, in order to ensure that your databases meet your business’ needs.

Although it might seem cumbersome and time consuming to review an entire database every few months, it does benefit the company in the long run. With regular audits, you can easily identify which areas should be optimized in order to save running costs, streamline tasks, and what actions to avoid in order to minimize mistakes as well as potential outside threats, such as data breaches or thefts.

Last but not least, make sure your database logs are compliant with data protection regulations such as the EU’s GDPR or the UK’s Data Protection Act 2018, especially if your organisation deals with personal data. 

Regular audits mean that you will be able to keep an eye on changing legislations and ensure that your organization’s databases follow the rules, while also letting it reap the benefits of data storage. After all, data is nowadays crucial to delivering the highest level of services, therefore it’s important that your databases are kept to the same standards.

A “technical issue” identified in the government’s flagship NHS Test and Trace contact-tracing scheme centres on serious limitations in the use of Microsoft Excel to log positive COVID-19 cases.

The government disclosed a technical issue, identified on 2 October, in the data load process that transfers positive COVID-19 lab results to reporting dashboards managed by Public Health England (PHE). This has led to a delay in reporting 15,841 positive COVID-19 cases, with some results stretching as far back as 25 September.

The IT blunder itself arose because the Excel spreadsheet being used to collect positive test results reached its maximum file size, which stopped new names being added in an automated process, according to the Press Association (PA).

To rectify this issue, the Excel files have been split into many smaller files so such a problem doesn’t arise in the future.

Further reports, circulating online, suggest a similar explanation, with the number of Excel columns instead reaching their peak of 16,384 rather than the spreadsheet file size reaching its maximum.

Reports also suggest the positive cases data is normally moved from Excel to Access before feeding into PHE’s reporting dashboards. It should be noted that PHE only reports the data it receives, which itself is collected by NHS Test and Trace.

However, neither explanation for the reporting blunder, nor the precise data transfer mechanism, has been confirmed by official sources. The health secretary Matt Hancock is expected to address the issue in a statement in the House of Commons this afternoon.

“A technical issue was identified overnight on Friday 2 October in the data load process that transfers COVID-19 positive lab results into reporting dashboards,” the government said in a statement published online. “After rapid investigation, we have identified that 15,841 cases between 25 September and 2 October were not included in the reported daily COVID-19 cases. The majority of these cases occurred in most recent days.”

“NHS Test and Trace and PHE have worked to quickly resolve the issue and transferred all outstanding cases immediately into the NHS Test and Trace contact tracing system and I would like to thank contact tracing and health protection colleagues for their additional efforts over the weekend.”

As a result of the backlog, there were an additional 12,871 cases reported on Saturday evening and 22,961 cases reported on Sunday evening, including thousands of out-of-date positive test results.

NHS Test and Trace had previously come under fire because its organisers launched the programme without conducting a Data Protection Impact Assessment (DPIA). Following legal action, the government conceded the scheme violated GDPR as a result of not being reinforced with a completed DPIA, although its organisers would argue that getting the system off the ground took priority.

Approximately 6.5TB of Microsoft Bing user data may have been left exposed online for two days after an Elasticsearch sever was left unsecured and open to the internet.

Data relating to search queries, device details, and GPS coordinates were found in a database by online security site WizCase, which was then able to trace the server back to Bing's mobile app.

The server was said to be growing by as much as 200GB per day while exposed, according to Ata Hakcil, a white hat hacker who also served as lead investigator for the website.

A link between the server and Bing was eventually found after researchers ran a search query for "WizCase" on the Bing mobile app and then paired the query data with that held on the server.

Exposed data included search terms in clear text, search times, locations, notification tokens, a partial list of URLs users visited from their search results, device models they used, their operating systems and three separate ID numbers (ADID, deviceID and devicehash) assigned to each user.

The server is believed to have been password protected until around the 10th September, with the server eventually being discovered by the WizCase team two days later. By the 16th September, Microsoft had been alerted and had encrypted the server once more.

The team believes that anyone who had used Bing's mobile app for internet searches between the 10th and 16th September may have had their data leaked online.

Researchers also believe that during this period the database was subject to what's known as a "meow" attack – an automated attack that targets unsecured Elasticsearch servers and destroys huge volumes of data without explanation or reason.

Cloud specialist WANdisco has partnered with Amazon Web Services (AWS) to launch a rapid self-service data migration tool, in a deal that's seen as a major win for the UK cloud industry.

The 'LiveData Migrator' lets companies move any size of on-premises data to AWS within minutes and without the need of engineers or specialists.

WANdisco, a publicly listed company with dual headquarters in Sheffield and California that specialises in distributed computing, is already an 'Advanced Technology Partner' in the AWS network and its LiveData Migrator services has also achieved AWS Migration Competency status – which requires a proven level of technical proficiency and customer success.

It is one of four AWS partners to collaborate on migration requirements for use cases including Hadoop, storage and database data migration, and mainframe data integration.

One of the first to use the LiveData Migrator is website hosting platform GoDaddy, which shifted 500 terabytes of Hadoop Distributed File System (HDFS) data to Amazon S3.

"We found that WANdisco's LiveData Migrator delivered the best time to value solution in the use case of a Hadoop to Amazon S3 data migration and replication," said GoDaddy chief data and analytics officer Wayne Peacock.

"Rather than running an internal time-consuming and costly manual migration project, using LiveData Migrator has helped us avoid disruption to our production processes and made 70 TB of data immediately available for Amazon S3 testing."

Migrating large data volumes with traditional approaches requires disrupting the operation of on-premises applications, but the LiveData Migrator works without any production system downtime or business disruption, according to WANdisco.

"Enterprises that want to move their data to the cloud but are concerned about the risks of doing so now have a powerful solution that's self-service and extremely easy to use," said WANdicso CEO David Richards.

"Regardless of company size or technical expertise, LiveData Migrator enables businesses to migrate their data risk-free to the cloud on a massive scale without any disruption to business operations. Data can be accommodated without any risk of data loss while modernising data and applications to stay competitive."

Saying that cloud platforms are popular with IT admins is like saying that water is wet. Over the past several years, the cloud has moved from an interesting emerging usage model to the de facto ruler of enterprise IT. While on-premise infrastructure is still in use (and will continue to be for the foreseeable future) there are few organisations that aren’t at least considering migrating parts of their infrastructure to the cloud.

Cost is a key motivator for this; received wisdom generally says that cloud infrastructure is cheaper and more cost-effective to run than on-premise equivalents, both in terms of the CapEx required for setup and in terms of ongoing costs. Scalability is also frequently cited as an advantage, allowing IT departments to scale the capacity of their infrastructure up and down as needed. The cloud is particularly attractive for database administrators, who are increasingly looking to migrate their large legacy database applications and take advantage of the benefits that cloud can offer.

This received wisdom, however, is not always accurate. In fact, many database admins have found to their dismay that a poorly-managed cloud migration actually results in lower efficiency and higher costs than they experienced with their on-premise systems. Part of the reason for this is a sub-optimal assessment of an organisation’s requirements prior to making the move, as not all databases are created equal.

Before moving your database to a cloud-based infrastructure, you’ll need to establish what service tier you require - this governs the level of CPU, storage, memory and network capacity that your database has access to. Most cloud providers offer right-sizing tools to predict your needs and model costs based on them, but because these are usually based on collecting a limited amount of data from your database over a fixed period, they’re not always accurate. This is important because, with cloud services, you pay for the resources you provision; not for the resources you actually use.

Inaccurately predicting your performance needs could lead to problems further down the road; either locking yourself into a needlessly expensive subscription that offers more baseline capacity than you require, or not having enough performance and headroom to scale when demand starts to spike. This undermines one of the core advantages of the cloud - while the technology is inherently scalable, the restrictions of subscription tiers may not be. These issues are magnified if you adopt a multi-cloud strategy since you’re now using different cloud providers with different rules.

It’s important, therefore, to have an accurate picture of your needs prior to beginning your cloud migration journey. Using tools like Quest Software’s Benchmark Factory to test and measure your database workloads pre-migration and compare performance with a similar cloud-hosted database, you can ensure you’re better equipped when you need to make the transition.

Similarly, it’s important to test whether your database will benefit from improved performance by migrating. Depending on the type of application and the frequency at which it is being accessed under load, your on-premise infrastructure may actually work better for some databases than a cloud-based equivalent. This, incidentally, is part of the reason why hybrid cloud solutions enjoy such a continuing level of popularity among businesses.

High workloads may also affect performance differently depending on where a database sits; although one infrastructure may outperform another when demand is moderate, at higher stresses, performance may start to slow down - an important consideration when planning the transfer of mission-critical systems that are prone to seasonal spikes. Quest’s Foglight for Databases monitoring solution helps counteract these issues by allowing database admins to run cross-platform performance analysis on both cloud and on-premise database workloads, allowing admins to drill down into which specific SQL queries are having the biggest impact on performance.

Without knowing how their databases actually perform in the cloud, admins run the risk of spending time and money on a migration, only to have to undo all that work and repatriate certain workloads that aren’t seeing the benefits they’d hoped for.

The hidden costs of cloud

Cost, of course, is arguably the biggest piece of the puzzle. All of the potential issues highlighted above can have an impact on the price you’ll pay to run your database in the cloud; by not right-sizing their level of cloud subscription, businesses can end up with unnecessarily large monthly bills, while performance bottlenecks can result in longer time-to-value from their applications. A poorly-handled migration can also result in downtime while the database is moved over, and for mission-critical applications, this could mean lost revenue and productivity.

Quest’s SharePlex data replication solution for Oracle ensures that you can safely migrate your Oracle data to the cloud, with minimum downtime and zero data loss, and continue to replicate data in real time if your cloud database is being used for offload reporting, as a failover for HA/DR (High Availability/Disaster Recovery) or other use case.

Perhaps the biggest hidden cost of the cloud, however, is in data transfer and access fees. With most public cloud platforms, you’ll incur a small fee every time your data is accessed, as well as every time you transfer data into, out of or across your cloud environment. While these charges generally amount to pennies per GB and are negligible in most cases, if you’re not closely monitoring your database utilisation, it’s very easy to get stung with high charges during busy periods.

Another area that can end up introducing unexpected costs for businesses is tooling requirements for managing their databases. Vendors such as Oracle only offer their most sophisticated database management tools exclusively for use by customers licenced on their expensive enterprise edition databases, which can be a problem for organisations who want to down-size their licences in order to reduce costs, but still need access to this database management functionality. One solution to this is to make use of third-party tools like Quest’s Toad for Oracle DBA Edition, which offers all the functionality of Oracle’s database management tools in a standalone package that operates independently of the subscription tier your database runs on, and works both for cloud and on-premises databases.

All of this may sound daunting and give the impression that the cloud is a minefield of hidden costs waiting to trip you up, but don’t be put off. There are still huge advantages to be had in migrating your databases, but the key is to go into a migration with your eyes open; it’s about shifting the right databases, in the right way, at the right time. Planning is an essential step in the process, and in order to inform your plans, you also need to be monitoring and measuring your database applications in the most effective way possible.

Thanks to tools like SharePlex, Foglight, Benchmark Factory and Toad, Quest Software’s enterprise-grade testing, monitoring and management products can help ensure that your cloud databases are right-sized, stable and as performant as you need them to be, without having to waste time and resources on trial-and-error troubleshooting. The journey to the cloud isn’t without its bumps - but with Quest, you can get there on time and on budget.

Find out how Quest can help migrate your on-premise Oracle databases to the cloud.

Automation is generally observed from one of two standpoints: those it affects positively and those it affects negatively. In the enterprise, it’s decision makers who mainly comprise the former group, as automation is able to lead to valuable cost savings through a reduction of staff, along with higher levels of efficiency.

Some employees on the ground, the lucky ones at least, also benefit. With their roles, at least to a degree, able to be completed by an automated machine, they themselves are liberated to pursue more valuable tasks that can enhance their personal skill sets and boost the enterprise as a whole.

But what about those that aren’t so lucky? Automation has been displacing jobs since the industrial revolution, with low-skilled workers laid off to give way to rows of chugging machines. Negative connotations attached to automation initiatives from that period continue to stream through society today.

Database administrators (DBAs) would be forgiven for harbouring this sentiment. At first glance, the rise of the autonomous database appears to threaten their livelihoods. But is there a way the two can coexist?

The need for an autonomous database

Databases are used to store information that’s critical to the efficient operation of the modern enterprise. Any firm with a digital element (in other words, any firm) will no doubt lean on a database, and most likely employ DBAs in order to maintain them.

DBAs however often find themselves overburdened with the laborious manual tasks related to database maintenance and management, to the extent that errors begin to creep into their work. And when it comes to errors and databases, no matter how inconsequential the mistake may seem, it can have a debilitating impact on uptime, performance, and security.

Take patching as an example. Failing to apply an update can create vulnerabilities, while applying the patch incorrectly undermines the security features it’s designed to bolster. Unsecured databases leave the enterprise at risk of breaches that damage both revenues and reputation. Further, the pure amount of data that the modern enterprise has to process is burying DBAs in a workload they can’t hope to meet successfully.

This considered, along with society's general trajectory towards automation, it was only a matter of time before databases were embedded with artificial intelligence and machine learning capabilities.

Shifting roles

Autonomous databases certainly aren’t the doom of the enterprise. With an autonomous database, errors are eradicated, ensuring maximum database uptime, performance and security, all while reducing costs and upping productivity, with automation able to complete menial tasks in a fraction of the time it takes a human predecessor.

A byproduct, yet certainly of no lesser importance, is the human impact, specifically regarding DBAs. When working with an autonomous database, DBAs find the tasks they used to complete instead undertaken by technology, freeing them from the shackles of the mundane task. However, the danger remains that a freed workforce is potentially a redundant workforce. Vendors would argue this isn’t the case. Instead, DBAs can refocus on tasks that add greater value to the enterprise.

Yet it sounds unrealistic for all DBAs to be deployed in this manner. And surely the immediate monetary boosts of reducing numbers of employed DBAs will prove too tempting for at least some organisations.

Despite these lines of thought, DBAs can take solace in a trend that is seeing their roles shift to a more expansive set of responsibilities, as opposed to seeing their position becoming entirely redundant. For this DBAs owe their thanks to a shortcoming of the autonomous database, manifesting in an inability to manage personal data securely.

The power of data

A prerequisite of the autonomous database is migration to a cloud environment which can leave data vulnerable. Though cloud environments are managed by service providers, the responsibility of actually securing the personal data that’s stored within doesn’t fall at their door, this instead falling to the enterprise using the database. The enterprise must identify and protect personal data to ensure compliance with data protection regulation, with a failure to achieve this risking hefty fines. For this they need a reinvented DBA, a journey that should result in prior DBAs becoming data administrators.

The buck doesn’t stop there. The smart DBA will be future-proofing themselves whether they are immediately threatened by an autonomous database or not, by embracing the automation of database administration tasks and positioning themselves for a more strategic role within the enterprise, allowing them to collaborate with other pivotal members of the IT department as well as the wider enterprise.

In the modern world, data is pervasive and DBAs find themselves in the fortunate position of being equipped with a transferable data-based skill set that can be utilised in other areas of the enterprise, because in business knowledge is data. Having the ability to interpret data can uncover answers key to responding to rapidly evolving marketplaces. A future-proofed DBA will be closely aligned with this line of business, and be able to provide information that the enterprise can use to differentiate themselves from their competitors.

In short, the role of the DBA is unlikely to be entirely erased given their existing skill-set and the prevalence of data in other sectors of the enterprise; however as autonomous databases become more commonplace, there will certainly be a significant shift in the responsibilities of their role.