Google and Oracle data centres hit by issues amidst hottest ever UK day

Both Google and Oracle cloud servers were impacted by yesterday’s intense heat, as data centres in South London were hit by a string of issues brought on by cooling failure.

In a service health update Google referred to IT Pro, the company identified 09:15 PDT as the point at which multiple services started to experience issues:

“Multiple Cloud products experiencing elevated error rates, latencies or service unavailability in europe-west2.”

“We are experiencing an issue with Google Cloud Storage, Google BigQuery, Google App Engine, Google Cloud Functions, Google Cloud Dataflow,” the company elaborated in a further description of the problems.

As of 19:06 PDT, Google was reporting that the affected cooling systems had been restored. Oracle faced similar cooling systems failures and explained that this was the cause for customer access delays with Oracle Cloud Infrastructure.

Any such customer with resources hosted in the UK South (London) region may have faced difficulties. Prompted for a statement, Oracle pointed IT Pro to the latest status report, in which it is acknowledged that the problem is “now resolved”.

“Following unseasonably high temperatures in the UK South (London) region, two cooler units in the data center experienced a failure when they were required to operate above their design limits,” read the summary.

“As a result, temperatures in the data center began to climb causing a subset of Compute infrastructure to go into protective shut down.”

Parts of London experienced temperatures exceeding 40 degrees celsius on Tuesday, marking the hottest UK day ever recorded. Normal business operations were put on hold for many workers, with some companies telling employees to work from home.

Meteorologists had warned for days in the lead-up to the weather event that critical infrastructure could be affected, as the UK was being forced to contend with unprecedented temperatures.

Luton airport reported that a small section of its runway had lifted due to heat, and train services were widely disrupted as tracks buckled and overhead lines melted. In some parts of the UK, firefighters were forced to tackle wildfires.

Both Google and Oracle identified portions of the affected systems that were still experiencing problems after their respective incidents had ended. Google specified that a “small number of HDD backed Persistent Disk volumes are still experiencing impact and will exhibit IO errors.”

Oracle identified those systems that had been subsequently fixed, and those that had not:

“Virtual Machine Instances which did not self-recover have been manually brought online by service engineers. However, a subset of Oracle Integration Cloud resources continue to experience impact. Engineers are actively working to mitigate those remaining service resources.”