Amazon Web Services (AWS) has confirmed a recent outage that impacted customers was caused by overheating at a North Virginia data center.

The disruption affected one of AWS' six Availability Zones, use1-az4 in the AWS US-EAST-1 region. This is one of the company's most heavily used regions globally.

Notably, the incident hit platforms including cryptocurrency exchange Coinbase, disrupting core exchange functions for more than five hours. Other reported victims include the CME Group trading platform and major gambling company FanDuel.

Coinbase last night warned that some users might experience delayed sends and receives on the Solana network and for ALEO, but said it was working on the issue.

The crypto trading platform has since resumed operations. In a statement, Coinbase said: "All markets have been re-enabled for trading on coinbase.com and in the Coinbase iOS and Android apps. Coinbase customers can log in to trade."

FanDuel, meanwhile, posted a statement on X, saying it was working to troubleshoot the issue. "Our team is aware and investigating the current technical difficulties prohibiting users from accessing our platform," it said.

Overheating behind AWS outage

In an update to customers, AWS attributed the cause of the outage to overheating. The hyperscaler is yet to confirm how the overheating occurred.

"We have experienced an increase in temperatures within a single data center, which in some cases has caused impairments for instances in the Availability Zone," AWS said in a status report.

"EC2 instances and EBS volumes hosted on impacted hardware are affected by the loss of power during the thermal event."

In its latest update, AWS said it had shifted traffic away from the impacted zone. The hyperscaler said it was still carrying out mitigation efforts.

These are taking longer than expected to bring additional cooling system capacity online and recover the remaining affected infrastructure safely and in a controlled manner.

AWS warned some customers will continue to see their affected EC2 instances and EBS volumes as impaired until it can achieve full recovery. It said it currently didn't have an ETA for this.

What services are back online?

A number of services are back online following the outage, according to the hyperscaler. This includes:

• AWS IoT Core
• AWS NAT Gateway
• Amazon Elastic Kubernetes Service
• Amazon Elastic Load Balancing
• Amazon Redshift

Some services are still impacted at time of writing, including:

• Amazon ElastiCache
• Amazon Managed Streaming for Apache Kafka
• Amazon OpenSearch Service
• Amazon SageMaker

Yet another AWS outage

It's not the first time that a major AWS outage has caused chaos. Last year, hundreds of apps and websites including Slack, Zoom, Coinbase, Snapchat, and Signal were taken down in a global outage.

Banking applications including Lloyds and Halifax also saw customers unable to access services. On that occasion, AWS attributed the outage to a DNS issue.

The incident highlights the extent to which major websites and apps are dependent on just a few tech giants.

In 2024, for example, issues with CrowdStrike saw hospitals, banks, and airports in Australia, New Zealand, India, Japan, the US, Germany, and the UK seriously affected.

FOLLOW US ON SOCIAL MEDIA

IT disruption is costing UK businesses a fortune, with nearly two-thirds losing more than £225,000 ($300,000) per hour.

The figure for the UK and Ireland is about 65% - slightly below the 68% reported globally, but still representing a critical financial risk; and one that executives are taking seriously.

According to PagerDuty's State of AI-First Operations report, top leaders are nearly unanimous (95%) that faster incident recovery and fewer major incidents create competitive advantage.

Notably, 53% identify brand and reputational damage as the biggest impact from major incidents, closely tracking the 52% reported globally.

Nearly half (48%) cite lost productivity as the next major impact when systems go down. Globally, organizations also report recovery and remediation costs (50%) and developer burnout (42%) as significant consequences of IT disruption.

“The extent of financial damage from IT disruption is a stark reminder that skipping a beat on resilience investment isn’t an option any more," said Eric Johnson, CIO at PagerDuty.

"There’s certainly a sliding scale in resourcing, based on what is feasible for any given business, but turning off the taps can spell disaster. About 65% of UK and Ireland companies face losses of £225,000 or more per hour during major incidents."

Enterprises are bolstering resilience

PagerDuty noted that 68% of respondents in the UK and Ireland have made efforts to increase operational resilience over the past year, close to the 71% global rate.

However, it's the UK and Ireland that have the widest investment gap globally: organizations here whose company revenue has increased in the past year are more likely to increase operational resilience budgets than those reporting flat or declining revenue.

This exceeds the 20-point global average by more than 50%.

“UK and Ireland organizations are making strong strides on operational resilience, but beneath that momentum lies a growing divide," said Johnson.

"When revenue is healthy, UKI leaders are injecting that cash back into IT departments. But those with flat revenue trends are choosing the opposite, creating a 31-point investment gap that is the widest we see globally."

AI brings potential and new risks

With enterprises across the UK and Ireland focusing heavily on AI investment, many are optimistic about the technology’s potential. Nearly half (41%) believe that AI-assisted operations automation could cut service disruptions by at least 20%.

This is at least in part due to faith in human oversight, with 59% having a three-year goal of maintaining an even mix of AI agents and human workers to reduce service disruptions and increase operational resilience, much the same as the 62% global rate.

"A blend of human workers and AI agents can help alleviate pressure on strapped teams, and balanced approaches here can surface issues and accelerate analysis effectively, all while preserving the judgment, context, and accountability," said Johnson.

FOLLOW US ON SOCIAL MEDIA

Most businesses would fall apart after three days of total downtime, according to Veeam, and executives are woefully unprepared for AI-driven risks.

According to its latest Data Resilience survey of more than 4,000 C-Suite and senior IT leaders, Veeam said data outages are now a bigger concern than the threat of a recession.

More than eight-in-ten organizations have experienced outages that weren't resolved immediately in the past five years. External cyber attacks were the most widespread cause, at 26%, closely followed by human error at 23% and system or hardware failure at 16%.

IT leaders are conscious of growing risks, the survey found, with 67% citing ransomware as their biggest threat looking ahead. AI-related risks, however, are starting to cause concerns.

Data leaks, algorithmic bias, and uncontrolled automation were cited by 29% of respondents as a key challenge for many moving forward.

The fallout from cyber incidents isn't just technical, either. More than half (57%) of leaders said employees had resigned, threatened to resign, or experienced burnout following major cyber incidents, with lost productivity and wellbeing among the top unexpected impacts.

Firms aren’t confident of data recovery

While nearly half (47%) of organizations expect they'll fall victim to a significant data breach or cyber attack, only 32% believe full recovery of critical data and business operations is very likely.

As a result, 62% of leaders see data outages as a bigger financial threat to their business than an economic recession.

“In today’s AI-powered world, trust in data is every organization’s most valuable asset. Backups are the last line of truth in a world where AI can fabricate, ransomware can encrypt, and a single misconfiguration can cascade across an entire infrastructure in minutes,” said Dave Russell, senior vice president and head of strategy at Veeam.

“In the AI era, it’s not just about recovering your data – it’s about keeping your business functioning and thriving with the trusted data it needs. Too many organizations are still managing risk reactively, when real innovation – and real trust – begin with a foundation of resilient, secure data.”

Nearly half (44%) of IT leaders said they weren't confident that their organizations could recover all critical data within 24 hours of a major cyber attack or data loss event, according to Veeam.

Despite this, though, 38% of boards or leadership teams surveyed have never formally discussed AI-driven or emerging attack types.

Only 31% of boards review resilience readiness, such as recovery KPIs or failover results, on a quarterly basis, and responsibility for resilience is often split across CIOs, CISOs, Heads of Risk, and COOs.

Meanwhile, fewer than half of organizations surveyed (49%) link executive KPIs to resilience outcomes, and only 24% of leaders regularly take part in crisis simulations for data loss or disruption.

FOLLOW US ON SOCIAL MEDIA

Cloudflare experienced a major outage on Friday that led to issues for a number of websites using its services.

According to Cloudflare, the issue affected customers using its Bring Your Own IP (BYOIP) service, who saw their routes to the internet withdrawn via Border Gateway Protocol (BGP).

"For some BYOIP customers, this resulted in their services and applications being unreachable from the Internet, causing timeouts and failures to connect across their Cloudflare deployments that used BYOIP," the company said in a blog post.

The impacted BYOIP customers first experienced a behavior called “BGP Path Hunting”, whereby end user connections traverse networks trying to find a route to the destination IP - and carry on until the connection that was opened times out and fails.

Meanwhile, visitors to the website for Cloudflare’s recursive DNS resolver (one.one.one.one) were met with HTTP 403 errors and an “Edge IP Restricted” error message.

DNS resolution over the 1.1.1.1 Public Resolver, including DNS over HTTPS, was not affected.

In total, the incident lasted six hours and seven minutes.

Cloudflare outage caused by configuration changes

Cloudflare confirmed the issue was caused by a change the company had made to the way its network manages IP addresses onboarded through the BYOIP pipeline. This change caused it to unintentionally withdraw customer prefixes.

"The specific piece of configuration that broke was a modification attempting to automate the customer action of removing prefixes from Cloudflare’s BYOIP service, a regular customer request that is done manually today. Removing this manual process was part of our Code Orange: Fail Small work to push all changes toward safe, automated, health-mediated deployment," the firm explained.

"Since the list of related objects of BYOIP prefixes can be large, this was implemented as part of a regularly running sub-task that checks for BYOIP prefixes that should be removed, and then removes them. Unfortunately, this regular cleanup sub-task queried the API with a bug."

While Cloudflare moved fast to reverse the change, around 1,100 prefixes were withdrawn first - around a quarter of the total number of BYOIP prefixes. Some customers were able to restore their own service by using the Cloudflare dashboard to re-advertise their IP addresses.

The outage affected a number of websites, including Uber, Workday, Minecraft, Wikipedia, and Microsoft Outlook.

Betting site Bet365 posted on X, "Hi, we’re aware of an issue with our Website/App, and our Technical Team are working to resolve it as soon as possible. We apologise for the inconvenience."

Cloudflare looks to prevent future issues

Cloudflare said it is already working on improving the Addressing API's configuration change support through staged test mediation and better correctness checks.

"Following our Code Orange: Fail Small promise to require controlled rollouts of any change into Production, our engineering teams have been reaching deep into all layers of our stack to identify and fix all problematic findings," it said.

"While this outage wasn't itself global, the blast radius and impact were unacceptably large, further reinforcing Code Orange: Fail Small as a priority until we have re-established confidence in all changes to our network being as gradual as possible."

FOLLOW US ON SOCIAL MEDIA

AWS has taken steps to boost resilience weeks after a major outage, rolling out a business continuity feature targeting DNS records.

Last month, an outage at the hyperscaler knocked an array of popular services offline – from Signal and Slack to Zoom and even Amazon itself – taking several hours to fully resolve.

The outage stemmed from DNS resolution issues and took place in what AWS calls US-EAST-1 Region, better known as northern Virginia, where plenty of massive data centers are located.

A month on, AWS has announced a new tool to help mitigate future incidents, helping to get systems back online within an hour after DNS issues, specifically in that often troublesome US-EAST-1 Region.

"Today, we’re announcing Amazon Route 53 Accelerated recovery for managing public DNS records, a new Domain Name Service (DNS) business continuity feature that is designed to provide a 60-minute recovery time objective (RTO) during service disruptions in the US East (N. Virginia) AWS Region," said Micah Walter, senior solutions architect, in a post on the AWS blog.

"This enhancement ensures that customers can continue making DNS changes and provisioning infrastructure even during regional outages, providing greater predictability and resilience for mission-critical applications."

AWS touts new tools

Walter explained that the tool targets DNS changes that customers can make within 60 minutes of a service disruption in that region.

"Customers running applications that require business continuity have told us they need additional DNS resilience capabilities to meet their business continuity requirements and regulatory compliance obligations," he said.

When services are disrupted, the new tool offers several benefits. "The feature maintains access to essential Route 53 API operations, ensuring that DNS management remains available when it’s needed most," Walter added.

"Organizations can continue to make critical DNS changes, provision new infrastructure, and redirect traffic flows without waiting for full service restoration."

However, the company also stressed that wasn't a sign of an issue at AWS, but pinned the need for the tool on demand from customers in sensitive industries.

"While AWS maintains exceptional availability across our global infrastructure, organizations in regulated industries like banking, FinTech, and SaaS want the confidence that they will be able to make DNS changes even during unexpected regional disruptions, allowing them to quickly provision standby cloud resources or redirect traffic when needed," Walter noted.

How the new tools work

The system works with existing setups and customers can continue using their existing Route 53 API endpoint without changes to apps or scripts.

"The implementation is designed for simplicity and reliability,” he said. This means customers won’t need to “learn new APIs or modify existing automation scripts”.

"The same Route 53 endpoints and API calls continue to work, providing a seamless experience during both normal operations and failover scenarios."

In the post, AWS noted that accelerated recovery isn't enabled by default, but switching it on for a new or existing hosted zone simply involves switching it on.

"You can enable this feature through the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS Software Development Kit (AWS SDKs), or infrastructure as code tools like AWS CloudFormation and AWS Cloud Development Kit (AWS CDK)," he said.

Accelerated recovery is available now for Amazon Route 53 public hosted zones and is free.

MORE FROM ITPRO

• The Microsoft Azure outage explained
• Inside a cloud outage
• The Cloudflare outage explained

Web users globally were met with error messages and site crashes yesterday after an outage at Cloudflare brought much of the web to a standstill.

The incident marks the latest in a string of major outages that have sent the web into meltdown this year.

Amazon Web Services (AWS) and Microsoft Azure, for example, both encountered technical faults that caused widespread disruption for consumers and enterprises alike.

While Cloudflare services are now back to normal and websites up and running again, like previous incidents, the outage will have had a serious impact on operations for a wide range of enterprises.

So what happened?

How the Cloudflare outage unfolded

Disruption began at around 11.20am UK time (7.20am EST) and was initially described as an “internal service degradation” which impacted limited services.

It’s around then that web users began reporting serious difficulties accessing a host of popular websites, services, and platforms.

OpenAI’s popular chatbot, ChatGPT, was among the services affected by the incident, along with social media site X and creative design platform Canva.

Online games such as League of Legends and Valorant were also impacted by the incident.

To add insult to injury, Downdetector, a site used to track web outages, was also temporarily taken down by the outage, preventing users from even checking the source of the disruption.

The cause of the outage

First and foremost, it’s important to note that the outage was not caused by a cyber attack of any kind.

In a blog post detailing the incident, Cloudflare CEO Matthew Prince said the company initially suspected that the source of the outage was due to a “hyper-scale” distributed denial of service (DDoS) attack. However, this was quickly dismissed.

Instead, Prince revealed the incident was “triggered by a change to one of our data systems’ permissions”.

While this appears rather vague, the root cause here lay with the company’s Bot Management system, which is used to protect against threats such as DDoS.

This is used by customers to “score” bot activity and control which bots are allowed to access an individual site – or block access.

As Prince noted, this system includes a “machine learning model that we use to generate bot scores for every request traversing our network”.

Thereafter, these scores are held in a “feature file” which refreshes every five minutes to keep pace with the behavior of bots attempting to access any given site.

These refresh cycles are generated by a query running on a ClickHouse database cluster, according to Prince. Changes to that query caused the database to “output multiple data entries”, which in turn meant the file doubled in size.

“The larger-than-expected feature file was then propagated to all the machines that make up our network,” Prince explained. “The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.”

Services are back up and running

Recovery efforts by Cloudflare meant that core traffic flows were back to normal by 2.30pm (UK time), with full recovery by 5.06pm.

Some users continued experiencing difficulties in the hours following, but this was largely due to a surge in network traffic as services came back online.

As with previous outages in recent months, some industry stakeholders have used these incidents to highlight the delicate infrastructural balancing act that keeps the modern economy running.

When one system or feature fails, it can have a devastating global impact, according to Brent Ellis, principal analyst at Forrester.

“In this case, the 3 hour, 20 minute outage could have direct and indirect losses of around $250 million to $300 million when you consider the cost of down-time and the downstream effects of services like Shopify or Etsy that host the stores for tens to hundreds of thousands of businesses.”

MORE FROM ITPRO

• UK and Irish businesses "severely underestimating" the cost of IT outages
• Website downtime costs businesses thousands a month
• Manufacturers report millions in losses as downtime wreaks havoc on operations

Web users are facing the latest in a string of web outages amidst technical issues at Cloudflare.

Users first began reporting issues at around 11.30am UK time (06.30am EST), and a host of popular websites have been affected by the incident.

Social media platform X, as well as Canva and OpenAI are among those caught up in the outage so far. Meanwhile, players on online games such as League of Legends and Valorant have also encountered difficulties.

Notably, Downdetector, which is used to monitor outages across the web, was also taken down temporarily. Users attempting to access the site were met with a message stating: "Please unblock challenges.cloudflare.com to proceed".

In a status update, Cloudflare said it’s aware of the issue and has launched an investigation to establish the root cause.

"Cloudflare is aware of, and investigating an issue which impacts multiple customers: Widespread 500 errors, Cloudflare Dashboard and API also failing,” the status update reads.

“We are working to understand the full impact and mitigate this problem. More updates to follow shortly.”

The incident at Cloudflare comes hot on the heels of major outages at both Amazon Web Services (AWS) and Microsoft Azure.

Both outages caused widespread disruption for consumers and enterprises, with thousands of websites impacted globally.

UPDATE: Cloudflare says it has identified the cause of the incident and has issued a fix.

Exact details on recovery rates are yet to be confirmed, however.

"We have made changes that have allowed Cloudflare Access and WARP to recover," Cloudflare noted in a status update. "Error levels for Access and WARP users have returned to pre-incident rates".

This is a developing story and will be updated as new details emerge.

UK and Irish organizations are outperforming the rest of EMEA in tackling IT outages – but are still facing massive costs.

According to research from New Relic, 26% of UK and Ireland businesses are hit by high-impact IT outages every week, compared with an average of 37% in EMEA.

UK and Ireland engineers said they were spending 25% of their time fire-fighting or addressing disruptions, taking focus away from developing new features or coding innovations.

And the costs are significant, with high-impact outages carrying a median cost of $2 million per hour, with the UK and Ireland reporting between $1 million and $3 million per hour.

For the UK and Ireland, the annual median cost of high-impact IT outages is $38 million per year. But, alarmingly, 34% of UK and Ireland respondents said they either don't know or aren't tracking the financial impact of outages. The EMEA average is 23%.

"Our latest findings are a wake-up call for UK and Irish organisations, which are severely underestimating the business cost of IT outages," said New Relic EMEA field CTO, Manesh Tailor.

For UK and Ireland organizations, the top three causes of high-impact outages are network failure, third-party or cloud provider services failure, and someone making a change to the environment.

Tool sprawl is a big problem, with 33% of UK and Ireland organizations citing too many monitoring tools and siloed data as a barrier to achieving full-stack observability, compared with the regional EMEA average of 27%.

As LLM-powered applications and agentic AI become more widely adopted, they introduce new visibility challenges. But, said the researchers, organizations are becoming aware of these challenges - indeed, AI is now the top driver of demand for observability.

The use of observability's AI monitoring capabilities in the UK and Ireland rose from 35% in 2024 to 45% in 2025, with only 5% of UK and Ireland businesses saying they weren't deploying or planning to deploy AI monitoring.

And UK and Ireland businesses are good at realizing productivity gains through observability, with 49% of practitioner respondents saying observability has helped free up time to work on other projects, significantly higher than the 34% EMEA average.

Just over four in ten said observability increases productivity by finding and resolving issues faster, up 12% on 2024. Strangely, though, perhaps, 20% aren't tracking the ROI of their observability investments, compared with 10% in EMEA.

"Organisations that aren't investing in observability not only risk huge revenue losses, the reputational damage from frequent outages is equally, if not more concerning," said New Relic EMEA Field CTO, Manesh Tailor.

"The data is clear: companies that embrace intelligent observability across their entire technology stack experience less downtime, fewer critical outages, a faster time to resolution, and increased operational benefits, enabling them to achieve their core business goals."

MORE FROM ITPRO

• Tool sprawl
• Too many organizations assume they're more resilient than they actually are in the UK

Google has apologised for a Cloud outage last week that knocked offline many of its own customers, saying the fault was down to a new feature that wasn't properly tested – and promising to do better at communicating such incidents in the future.

On Thursday last week, outages on status trackers like Downdetector began to suggest an issue with tens of thousands of reports of outages for Google Cloud, Spotify, Discord, and more. After initial media reports, Google's Cloud status page posted a short update about the outage.

The company has now released a full incident report, pinning the blame for the outage on a policy check system called System Control that was given a new feature last month to allow it to make extra quota policy checks for API requests.

"This code change and binary release went through our region by region rollout, but the code path that failed was never exercised during this rollout due to needing a policy change that would trigger the code," Google explained. "As a safety precaution, this code change came with a red-button to turn off that particular policy serving path."

"The issue with this change was that it did not have appropriate error handling nor was it feature-flag protected," it added. "Without the appropriate error handling, the null pointer caused the binary to crash. Feature flags are used to gradually enable the feature region by region per project, starting with internal projects, to enable us to catch issues. If this had been flag-protected, the issue would have been caught in staging."

The policy change was added to Service Control on the day of the incident. "This policy data contained unintended blank fields," Google said. "Service Control, then regionally exercised quota checks on policies in each regional datastore. This pulled in blank fields for this respective policy change and exercised the code path that hit the null pointer causing the binaries to go into a crash loop."

Wide impact

The outage hit a wide range of third-party companies, including OpenAI, Shopify, and MailChimp, as well as Cloudflare, which in turn impacted the customers of that web and cloud services provider.

Cloudflare said the Google troubles didn't touch its core services, but did cause outages in a large set of critical services including Workers KV, Gateway, Stream, and its Dashboard.

"We're deeply sorry for this outage: this was a failure on our part, and while the proximate cause (or trigger) for this outage was a third-party vendor failure, we are ultimately responsible for our chosen dependencies and how we choose to architect around them," Cloudflare said in a blog post.

"This was not the result of an attack or other security event," the post added. "No data was lost as a result of this incident. Cloudflare Magic Transit and Magic WAN, DNS, Cache, proxy, WAF and related services were not directly impacted by this incident."

Seven-hour outage explained

Google said the incident was spotted immediately with engineers on the case within two minutes, and the cause identified within ten minutes, and a fix rollout after 40 minutes. However, the knock-on effects of the incident lasted several hours, starting just before 11 am US Pacific time and ending after 6pm on Thursday.

As the systems restarted, it created a "herd effect on the underlying infrastructure it depends on" – and that overloaded the infrastructure, Google said. "Service Control did not have the appropriate randomized exponential backoff implemented to avoid this."

That took another two hours and 40 minutes to resolve – throttling the fix to avoid overloading the infrastructure and rerouting traffic elsewhere to reduce the load – but recovery time was longer for some Google and Cloud products depending on their architecture, the company said.

Going forward

Google admitted the incident would hurt its users and said it would take steps to avoid a similar outage.

"We deeply apologize for the impact this outage has had. Google Cloud customers and their users trust their businesses to Google, and we will do better," Google said in a statement. "We apologize for the impact this has had not only on our customers' businesses and their users but also on the trust of our systems. We are committed to making improvements to help avoid outages like this moving forward."

Those changes include rejigging the architecture of Service Control so it's modular, meaning any failures won't take out entire systems, as well as improving its communications to help customers react more quickly – which may be a response to the fact that customers spotted the outage and tweeted about it well before Google made any public announcement.

Travelers at London Stansted Airport, Essex, were left bound to terra firma on Sunday 11 May after an IT issue led to massive delays across several airlines.

In a statement to the PA news agency, the airport said only that there was an “IT outage” that occurred when the terminal was particularly busy on Sunday morning. The airport told ITPro that it was still investigating the root cause of the IT issue “but it looks like it may be related to internet connectivity”.

The airport confirmed to ITPro that while the issue affected "a host of airport systems ... the main ones were airline check-in, baggage and security.

The problem was first identified at 2.30am, according to The Standard and wasn’t resolved until for eight hours, meaning those turning up for early morning flights were caught in the middle of the outage. While Stansted Airport told ITPro that no flights were cancelled during the outage, by the time the systems had recovered thousands of people had nevertheless missed their flights.

While Jet2 and Tui chose to delay their flights and wait for passengers, according to Mail Online Ryanair “dispatched near-empty planes on time to avoid delays later in the day”. The website also reported that some people who were flying with Jet2 and Tui chose to leave Stansted and rebook for flights leaving from alternative airports, rather than wait in the crowds of people hoping to get help from their airlines..

This is far from the first IT incident to cause chaos at a major airport. In 2020, an IT problem at London Heathrow Airport – the busiest in the UK – meant digital departure and arrival boards stopped working, causing passengers to miss already delayed flights.

A 2019 IT outage caused major headaches for British Airways passengers departing Terminal 5 of the same airport, due to a supplier-side issue.

Meanwhile, in November 2015, planes were grounded at Paris Orly airport after a piece of software linking air traffic control with the French met office that ran only on Windows 3.1 malfunctioned.

The CEO of BCS, the Chartered Institute for IT, has formally thanked IT teams across the UK for their work dealing with the CrowdStrike crisis.

In an open letter, Rashik Parmar said that the work of IT staff is often taken for granted, but underpins every aspect of life.

"I would like to see more IT systems treated by government as a critical part of our national infrastructure – as important as energy suppliers, road or rail," he said.

Getting systems back up and running following the global IT outage has taken great dedication, he wrote.

"The crisis we faced last week tested everyone in our industry," he said. "I was proud that so many members of our professional community stepped up to be counted, whether advising CEOs, providing insight and updates across social media, or re-booting laptops."

While the fix is relatively straightforward, it involves manual intervention for large numbers of machines. Each device must be rebooted into safe mode manually, rather than using management tools and in some cases, this has meant carrying out several reboots of individual machines.

Over the weekend, CrowdStrike said that around 8.5 million Windows devices had been impacted, but that 'a significant number' were now back online and operational.

However, the BCS warned last week, it could take a long time to get everything back to normal.

"In some cases, the fix may be applied very quickly, but because it has to be applied to so many computers around the world, that may take longer than it sounds," said BCS fellow Adam Leon Smith.

"But if computers have reacted in a way that means they're getting into blue screens and endless loops it may be difficult to restore, and that could take days and weeks."

The BCS has called on organizations to make sure their IT teams are well supported as they deal with the crisis.

A call to professionalize IT

In his letter, Parmar also repeated the BCS's calls for IT professionals to be registered as Chartered and held to account.

"We need to ensure that senior technologists are on every board; that cyber security has a mandatory code of practice, and that government and industry expect technologists to be Chartered IT Professionals," he wrote.

"We should be proud to have our names on a professional register; we should be held accountable to the Chartered standard if we fall short, as with doctors or accountants."

Parmar added that the government should track, record and assess IT outage incidents to build up the UK's national resilience to events such as the CrowdStrike outage.

"None of these things in isolation will ensure software never fails – it is the quality and commitment of software experts and increasing education among non-technical leaders that will move us forwards," he wrote.

"For now, as technology professionals, you can be proud of your response and your vital role in our national life."

Microsoft customers in Europe have complained that both the Teams and Outlook platforms are down as users start their working day. 

Users of the sysadmin subreddit have queried whether both platforms are working optimally this morning, with some noting that portals are failing to load and reporting that Exchange Online is “extremely slow”. 

This issue appears to be affecting mainly European-based customers, with one user reporting that German admin teams are experiencing “no incoming or outgoing emails” for the Outlook service. 

Others reported that their teams called ‘dropped’ mid-call. 

Allestorungen, the European equivalent of Downdetector, shows a flurry of outage reports this morning for users. 

At present, UK-based users have not reported any widespread issues. However, Downdetector shows that some are reporting difficulties. 

Microsoft's own service update portal also shows that Teams and Outlook are down for selected users. 

Microsoft said the current issue is causing duplication of users’ contact lists, and is due to a syncing problem between Teams and Outlook. 

“The issue only impacts users who synced their personal account device contacts with Microsoft Teams,” the company said in its status update. 

“New syncs are no longer resulting in duplicate contacts since we’ve implemented the long-term fix.”

Microsoft revealed that a workaround is available for Outlook and Teams users, however. This can be done by following these steps:

• Visiting the Teams app on your mobile device (iOS or Android).   
• Clicking on the Profile Icon Home and navigate to Settings and People. 
• Turn off Sync using the “Sync my device contacts” toggle  
• Device contacts in Outlook will then be automatically deleted within a few days

The outage marks the second incident in as many weeks for Microsoft 365 customers. In early June, thousands of users across Europe were left without access to Microsoft Teams due to an outage that severely disrupted communications for a host of companies. 

In the latest of a series of tech failures for British Airways (BA), the company has apologised for yesterday's failure of its check-in services, which caused long queues and delays for passengers.

Because of the issue, customers at Heathrow, Gatwick and London City airports were forced to check in manually - a much slower process.

"We are sorry for the temporary check-in problems which causes some delays for our customers first thing this morning," a spokesman told Reuters. "This issue is now resolved and our staff are working flat out to help customers get away on their holidays."

This is not the first time this year that BA has experienced problems, with a massive IT outage in May ultimately blamed on human error. The incident saw hundreds of flights cancelled because an engineer had switched off the power supply to one of the company data centres by mistake.

In April another outage resulted in passengers experiencing long delays on the online check-in service, with customers unable to book, check-in online or access their accounts for around seven hours. Customers were asked instead to check-in at the airport on the day of their flight.

At the time, Mehdi Daoudi, co-founder and CEO of Catchpoint, said: "The most important takeaway from this incident is that we all have a duty to our customers to provide the best service possible, under any circumstance.

"Issues like this one are not unusual, but it's important that digital businesses take necessary steps to reset their preventative strategies to see and resolve performance issues."

A similar problem hit Manchester airport last month, when a technical failure left passengers stranded for several hours after the check-in system was found unable to register passengers - leading 12 Thomas Cook flights to be grounded while staff switched to a manual system.

07/06/2017: Human error was to blame for British Airways' massive IT outage that saw hundreds of flights cancelled late last month, the chief executive of its parent company has confirmed.

Willie Walsh, the chief executive of International Airlines Group of which BA is a part, said an engineer had mistakenly switched off the power supply to one of the company data centres which was then turned back on in an uncontrolled fashion, according to the Financial Times.

Speaking at a transport conference in Mexico, Walsh said: "It's very clear to me that you can make a mistake in disconnecting the power ... It's difficult for me to understand how to make a mistake in reconnecting the power."

The engineer is reported to be part of a team working at the Heathrow data centre hit by the power outage. The facility is managed by CBRE Works Solutions, a US property services company, which told IT Pro: "We are the manager of the facility for our client BA and fully support its investigation. No determination has been made yet regarding the causes of the incident on May 27."

A BA spokesperson told IT Pro that the company is undertaking an exhaustive investigation to find out the exact circumstances of the power failure.

The spokesperson said: "There was a loss of power to the UK data centre which was compounded by the uncontrolled return of power which caused a power surge taking out our IT systems. So we know what happened, we just need to find out why. It was not an IT failure and had nothing to do with outsourcing of IT, it was an electrical power supply which was interrupted."

This backs up the internal email sent by Bill Francis, head of group IT at IAG, which was leaked to the Press Association, that read: "This resulted in the total immediate loss of power to the facility, bypassing the backup generators and batteries . . . It was turned back on in an unplanned and uncontrolled fashion, which created physical damage to the system."

Approximately 75,000 passengers were affected by the crash when BA was forced to cancel flights from Heathrow and Gatwick airports, affecting families who were setting out on half-term holidays.

02/06/2017: British Airways' IT fiasco last week may have been because a contractor made a fatal error in shutting down the equipment rather than the equipment failing.

A BA source told The Times that it was rumoured that a contractor carrying out maintenance accidentally switched the supply off, but this report has not been confirmed.

However, an internal email from Bill Francis, head of group IT at IAG, was leaked to the Press Association which seemed to confirm this version of events. It said: "This resulted in the total immediate loss of power to the facility, bypassing the backup generators and batteries . . . It was turned back on in an unplanned and uncontrolled fashion, which created physical damage to the system."

A BA spokesperson told IT Pro: "We are conducting an urgent investigation and it would be premature to comment on details before its conclusion. As we've said before it was not an IT issue, it was a power issue. There was no data corruption or loss and IT outsourcing was not a factor".

Approximately 75,000 passengers were affected by the crash when BA was forced to cancel flights from Heathrow and Gatwick airports, affecting families who were setting out on half-term holidays.

BA still hasn't released details behind the nature of the power supply issue affecting the IT problems on one of the busiest weekends of the year or explained why backup system weren't in place to prevent such a major outage from happening.

30/05/2017: British Airways CEO blames IT outage on "power supply" issue

British Airways' CEO has blamed power supply issues for an IT outage causing mass flight cancellations and widespread disruption over the bank holiday weekend.

Thousands of travellers were affected by the crash when BA was forced to cancel flights from Heathrow and Gatwick airports - many customers being families embarking on half-term holidays.

However, CEO Alex Cruz said the majority of its flights are now back up and running and will be expediting refunds to customers affected to attempt to win back some of its reputation.

"We have experienced a major IT systems failure that is causing very severe disruption to our flight operations worldwide," Cruz said. He added that all the company's check-in and operational systems were affected and as a result, it was forced to cancel all of its flights from Heathrow and Gatwick.

"Our IT teams are working tirelessly to fix the problems. We believe the root cause was a power supply issue and there's no evidence of any cyber attack."

On Sunday, Cruz offered an update, saying the IT teams had made progress trying to fix the problems. "Many of our IT systems are back up today and my colleagues across the airline are working very hard to build back our flight programme and get as many of our customers as possible away on their travels," Cruz said on Sunday.

He told Sky News that the IT problems "have all been local issues around a local data centre", denying union GMB's claims that the problem could have been avoided had BA not outsourced IT jobs to India in cost-cutting measures.

However, BA has not detailed the nature of the power supply issue behind IT problems on one of the busiest weekends of the year, nor explained why backup systems weren't in place to prevent such a major outage happening.

A British Airways outage subjected passengers to long delays in online check-in yesterday, preventing access to most features on the site.

The outage started around midday and lasted until 7pm. Passengers were unable to book, check-in online, and access customer accounts.

In a statement, BA said: "Unfortunately, the IT system which manages the online check-in function has not been fully fixed and therefore we have taken the decision to switch it off for the time being. We recognise the frustration this is causing some customers and we are trying to fix it as an urgent priority."

"If you are flying with us on Tuesday evening or Wednesday, please check-in at the airport and arrive in good time for your flight as we expect some of the major airports on our network to be busier than usual at what is already a busy time of year with the Easter holidays."

The outage comes after British Airways made hundreds of IT workers redundant late last year.

The website is back up and running, and in a Twitter update yesterday evening, the airline said the problem was "related to an IT system upgrade to some databases we carried out overnight".

Robert Castley, senior performance engineer at Catchpoint, said he noticed that BA.com uses Akamai for serving all its content. "However, when the site went down, all the static contents of the page such as images that were served by Akamai continued to be available. For example, their logo was still accessible," he said.

Mehdi Daoudi, co-founder and CEO of Catchpoint, added that it would be interesting to find out if BA.com had any redundancy or failover plans to handle scenarios such as this, as it "looks like they were caught off-guard this time".

"It's unfortunate that it happened at a very busy time when many people will be travelling for Easter. The most important take away from this incident is that we all have a duty to our customers to provide the best service possible, under any circumstance. Issues like this one are not unusual, but it's important that digital businesses take necessary steps to reset their preventative strategies to see and resolve performance issues," he said.

An outage hit Microsoft's main cloud software products, including Outlook.com, Hotmail, Office 365 and OneDrive, last night with all geographies appearing to have been affected.

Between roughly 4pm and 7pm BST, users trying to access services that use the login.live.com authentication portal were unable to do so.

This would have been most apparent to those using the browser, where the login page would fail to load and clicking the 'login' link sometimes resulted in a file download. However, the Skype desktop client was also affected and users were unable to send or receive emails from their Outlook, Hotmail or Live accounts using clients on their phones or desktops either.

One notable exception, however, is that those using Outlook with Windows 10 were still able to receive emails throughout the disruption period.

All of the services listed and more use login.live.com as a central authentication portal. They also all run on Microsoft's Azure cloud infrastructure, although according to the service dashboard, there was no problem with Azure during the outage.

Down Detector recorded 7,995 outage incidents just before 7pm last night but shows far fewer now, and it seems most users can now login, as confirmed by downforeveryoneorjustme.com and independently by IT Pro. But Microsoft has indicated that some issues remain.

In a statement to IT Pro, a Redmond spokesperson said: "We are aware that some users are experiencing difficulties signing in to some services. Engineers are working to fully resolve this as soon as possible."

The root cause of the outage, the total number of customers affected and whether there was any risk to customers at any point is as yet unclear.

It seems some Americans are also looking to flee to New Zealand, as their immigration website was knocked offline due to a surge of traffic as the US election results came in.

This follows the crash of Canada's immigration site on Wednesday as US voters began to reel from a shock win for Donald Trump.

The website run by Immigration New Zealand (INZ) received 56,300 visits in 24 hours, an almost 2,500% increase over the daily average of 2,300 visits.

Americans expressed their dismay at the election results on Twitter, with many considering New Zealand as their best bet.

New Zealand Now, a partner website offering advice for living and working in the country, saw a similar surge in traffic over the same period.

"In the 24 hours up to 9am local time on Wednesday, NZ Now received 70,500 visits from the US compared to a usual daily average of 1,500 visits," said Greg Forsythe, marking manager at INZ, speaking to the BBC.

American visitors made almost 7,300 registrations of interest in 24 hours, according to Forsythe, up from the normal 3,000.

09/11/2016: US voters crash Canada's immigration website after Trump win

A surge in traffic following Donald Trump's victory in the US election crashed the official website for Canada immigration, officials have confirmed.

People initially linked the outage with Trump's surprise win, saying it showed Clinton supporters were looking into the possibility of leaving the US altogether.

A spokeswoman has since told IT Pro that the crash was indeed down to a "significant increase" in traffic experienced earlier today.

She said: "On November 8th, The Immigration, Refugees and Citizenship website (cic.gc.ca) became temporarily inaccessible to users as a result of a significant increase in the volume of traffic. Shared Services Canada worked through the night and continues to work to resolve the issue to ensure that the website is available for users as soon as possible.

"Please note that the Government of Canada is committed to secure and reliable IT infrastructure to ensure high quality services to all users."

Following Trump's victory, the website for Citizenship and Immigration Canada simply displayed a 'This site can't be reached' page, with some observers attributing this to plenty of Democrat-voting Americans trying to apply to live in Canada.

Parks & Recreation star Ben Schwartz tweeted: "I was gonna write this as a joke but found out it was real. The Canada Citizenship and Immigration site crashed around 8 PM"."

The site has since come back online, though the outage was one of several technical difficulties experienced at the close of the US election.

A Utah call centre providing a get-out-the-vote service for Hillary Clinton's campaign, called TCN, was a target of Republican-motivated hackers, according to The Verge.

A user posted on 4chan to say they would attack any pro-Clinton campaign tools "like a wild animal", using DDoS attacks based on the open source Mirai botnet to bring them down.

Only last month, a 4chan user managed to hack into Clinton campaign manager John Podesta's iCloud account and remotely wipe his iPhone, using a password published by WikiLeaks.

This story was originally published at 11.22am, and was updated at 2.15pm to include comment from a spokeswoman for the Immigration, Refugees and Citizenship website.

A cyber attack that has crippled HSBC's website has left customers locked out of online banking services today.

A DDoS (distributed-denial-of-service) attack knocked out HSBC's online functions at around 8.30am this morning, and customers have struggled to access their accounts online since, with HSBC warning at 4.45pm that the assault on its servers is ongoing, though a normal service is returning.

While no customer records or transactions have been compromised, the attack left customers unable to log into online banking facilities via web browsers or mobile apps.

A spokeswoman for HSBC initially told IT Pro that the bank "has successfully defended against the attack" at around 11.20am.

But a statement from COO John Hackett at 4.45pm read: ""We are continuing to experience attempted denial of service attacks and we are closely monitoring the situation with the authorities.

"HSBC's internet and mobile services have partially recovered, and we continue to work to restore a full service. We apologise for the disruption and inconvenience this may have caused."

Today's DDoS attack comes on the final Friday of January, when many people will still be getting paid, and follows two days of outages at the beginning of this month affecting 17 million customers.

Security firm ESET's security specialist, Mark James, said HSBC's reputation will suffer as a result.

"DDoS attacks, regardless of motive, are never good for any organisation," he said. "Its users may vote with their feet rather than be understanding and stay with them."

Tripwire security researcher, Craig Young, speculated that the DDoS assault could be the work of hacktivists - hackers with a moral axe to grind - or cybercriminals looking to force HSBC to cough up in order to put an end to the issue.

"This is a common type of attack used by so-called hacktivists looking to make a political statement as well as extortionists requesting a ransom in exchange for stopping the attack," he said. "Often times the flood of requests are coming from computers and routers which have been hacked and unwillingly enlisted for attack."

HSBC said it is working with police to identify the culprits.

A spokeswoman said: "HSBC is working closely with law enforcement authorities to pursue the criminals responsible for today's attack on our internet banking. We apologise for any inconvenience this incident may have caused."

In addition to the outage at the beginning of January, the bank also angered customers last August by stopping BACS payments from leaving their accounts. HSBC revealed this week in a letter to a committee of MPs that the error was caused by a mega-payment rejected by its system, according to the BBC.

Functionality began slowly returning to HSBC's beleaguered digital banking services last night, after two days of outages affecting the company's 17 million customers.

COO John Hackett lauded the "steady return of service" to personal and business customers' online accounts, but HSBC told IT Pro it cannot confirm a full return to service yet.

Hackett said in a statement released yesterday evening: "I'm pleased to say that we have seen a steady return of service to internet banking in the past few hours for our personal and business customers. We will be monitoring the service very closely, ready to respond should any new issues arise.

"It is encouraging that more and more customers have been able to log on this afternoon, and all customers should feel free to try."

HSBC also today declined to explain the technical fault behind the online banking failures that occurred at the beginning of the week.

The issues began at around 8am on Monday, prompting thousands of angry complaints on Twitter from frustrated users, though the bank promised that any penalties customers incurred as a result of the outage would be waived.

Yesterday it took steps to bolster its telephone banking and branch resources, expecting customers to turn to these options while online services were down.

This is not the first time HSBC has suffered such a failure - last August another issue stopped BACS payments from leaving customers' accounts, with the high street bank rectifying the problem a day later.

05/01/2016: HSBC: Cyber attack is not behind digital banking outage

HSBC has apologised for a digital banking outage that is now in its second day, but has denied it is the result of a cyber attack.

Saying it has identified an "internal technical issue", the high street bank added that its mobile app is back up and running, but admitted desktop users are still experiencing problems.

Its online banking services have now been down intermittently since 8.21am on Monday, and thousands of HSBC's 17 million customers have taken to Twitter to complain.

Customer Paula Munoz tweeted: "So annoying!!I can't still use @HSBC_UK internet banking. #Europe largest bank can't even have a reliable web site #HSBCdown."

And Lauren Smith wrote: "The whol[e] HSBC network is down!! Argh gimme my money!! #hsbcdown."

But HSBC's COO, John Hackett, said in a statement today that the bank is closer to solving the issues.

"There is a complex technical issue with our internet banking systems, and our IT team has been working non-stop since yesterday morning to find a solution," he said.

"This has involved many tests, diagnostics and trial runs. We are getting closer to solving the problem, but are not there yet. We can, however, confirm this is not a cyber-attack or any other malicious act."

He added: "We will ensure customers do not lose out as a result of this issue. Any fees customers incur as a result of this outage will be waived."

Instead of going online, banking customers are resorting to phoning HSBC or going into their local branches, and the bank said it has "mobilised all our available resources" to cope with the influx.

It remains unclear exactly what the cause of the fault is.

05/01/2016: HSBC apologises for mobile and online banking outage

HSBC has apologised for its mobile and online banking outage, saying it is investigating into what caused everything to stop working on the first day back to work after Christmas and New Year.

The systems went down at 8am yesterday morning and, as ever, customers took to Twitter to vent their anger, with one customer saying, "Nightmare. No mobile or online banking services for over an hour now. Telephone queues unreasonable. Bills to pay," while another said they were struggling to get through to telephone banking too "Oh @HSBC_UK I'm phoning *because* your website is down. Recorded message suggesting I try online at time of high call volumes not helping,".

HSBC announced the outage on Twitter shortly after it discovered the problems, but this did not seem to appease many of the company's customers.

It opted to fix business services first, followed by personal banking shortly after. Even after the company had claimed to fix everything, it was clear the network was still being battered by people trying to get online to make payments and check their balance after the Christmas splurge.

"Business and retail customers have been unable to access online and mobile banking on Monday 4 January since 8am. Personal Mobile Banking and Business Internet Banking have been restored, but due to high demand may be slow," HSBC said in a statement yesterday afternoon.

"We apologise for any inconvenience this may have caused and our teams continue to work non-stop to restore all other services. Regular updates will be provided. We will ensure customers do not lose out as a result of today's problems."

HSBC has not revealed what went wrong, although it would seem the problems have been going on for a few days, with the biggest complaint being the speed of service.

The bank came under fire last August when its systems suffered a hiccup, stopping BACS payments from leaving customers' accounts. However, a day after the problems were reported, it said it had fixed the issue.

Planes were grounded at a Paris airport last weekend, after a comedy of errors emanating from software underpinned by the ancient operating system, Windows 3.1.

Paris Orly airport closed its runway on the morning of 7 November, with flights diverted to Paris Charles de Gaulle and Lyon airports, following a critical infrastructure failure caused by a system called DECOR.

DECOR links air traffic control systems directly with Meteo France and is vital for taking off and landing in low-visibility situations like mist and fog.

It also runs on Windows 3.1, which was released in 1992 and retired on 31 December 2001.

Le Canard Enchain, a French magazine that first reported the cause of the incident, said that only three engineers are allowed to touch the "capricious beast".

These engineers had been working to restore DECOR for many hours before the runway was closed for an hour for safety after fog set in.

The outage also had political ramifications - the Republicains, a political party set up by former president Nicolas Sarkozy, had to do without two of their big hitters, former Prime Minister Alain Jupp and MP Eric Ciotti, who were left stranded at their points of origin.

But the story doesn't end there.

After the system was eventually restored and the runway reopened, a spokesperson for the Ministry of Transport issued a statement promising the system will be modernised, but not until 2017.

An anonymous source in Air Traffic Control told Le Canard Enchain: "Saturday morning, air traffic was not that busy but think about it, if during COP21 [a sustainable development forum in Paris held the week of 7 December], the coming and going of heads of state gets disrupted by a prehistoric piece of software, what are we going to look like?"

While this is not the first time technological failure has led to flights being grounded, to the best of IT Pro's knowledge it is the only time such an incident was caused by such an ancient operating system.

The French Ministery of Civil Aviation did not deny the issues when contacted by IT Pro, but said it was unable to provide a response until Monday - the article will be updated with any new information then.

Image credit: Leonid Andronov / Shutterstock.com

Mobile operator O2 was left red-faced after thousands of customers lost signal across the UK on bank holiday Monday.

The company was told to "sort it out" by Business Secretary Sajid Javid, who took to Twitter along with hundreds of others to criticise the service.

After several hours of downtime, the network for affected customers in London, Manchester, Glasgow and Northern Ireland was back online at 11.35pm yesterday.

O2 blamed phone mast issues around the UK for the problem, adding: "We apologise for any inconvenience caused to those customers affected and we will now begin a full investigation to identify the root cause."

While Cabinet minister Javid had tweeted "No signal @O2. Please sort it out", telecoms expert Dan Howdle said it was unlikely that angry customers venting on Twitter would actually switch providers.

The editor-in-chief of Cable.co.uk said: "As infuriating as it may be during the disruption, I would be very surprised if many jump ship from O2 as a result."

He pointed to his own research of 2,500 UK mobile users, which found 50 per cent are still with the same provider they signed up with years ago.

Howdle added that instead of just venting frustrations on social media, customers should be prepared to move for better service.

"If reliability is our number one consideration, as many O2 users affected by the outage now claim, we must be prepared to vote with our feet," he said.

O2 is in the process of being bought by Three owner Hutchinson for 10.25 billion.

Three itself suffered an outage in Ireland last month.

Tesco has apologised to customers after its broadband service crashed yesterday, leaving thousands without internet access.

The supermarket blamed the outage on a technical fault with its network, but said the problem has now been fixed.

A spokeswoman told IT Pro: "The problem was a fault at a network level and was resolved last night for all customers. I'm afraid I can't confirm how many customers may have been affected at this stage."

However, the BBC claimed up to 10,000 of Tesco broadband's 75,000 users around the country lost connectivity after the incident occurred at 12pm yesterday.

Mobile operator TalkTalk is buying Tesco's broadband network, along with video streaming service Blinkbox, as it moves to become a quad-play operator similar to BT, offering landline, mobile, pay-TV and broadband services.

But Tesco said the outage wasn't related to the sale.

The spokeswoman said yesterday: "We are liaising with our network partner [Vodafone] to get the issue resolved as quickly as possible and apologise to any customers affected for the inconvenience."

However, the problem has echoes of Three's network outage across Ireland earlier this month, which affected 1.5 million users.

Many of those users appeared to be O2 customers who'd been transferred to Three's service following the 10.25 billion acquisition of O2 by Thrtee owner Hutchinson in March.

Three released a statement at the time blaming the problem on an overheated data centre, rather than on the O2 buy-out.

This article was originally published on 21/04/2015 at 5pm, and was subsequently updated on 22/04/2015 at 1pm after Tesco brought its service back online.