GitHub outage blamed on misconfigured policy as firm pledges resilience improvements

GitHub has revealed the root cause behind a nearly eight-hour long outage on Monday: a misconfigured policy that triggered a cascade of failures.

On Monday, GitHub users began struggling to use the service around 1:28pm UTC, reporting issues with API requests, Actions, Webhooks, Pull Requests, and GitHub Copilot. At the time, GitHub said it was seeing error rates between 20% to 50% for different tasks, with other services also impacted.

"At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%," GitHub said via an update on its status report.

"SAML/OIDC authentication, SCIM, and Team Sync were also affected, as well as Actions workflows in GHEC with Data Residency that depend on public workflow step definitions hosted on GitHub.com."

Within a few minutes the problem was spotted, but it took a total of seven hours and 47 minutes for the issues to be totally resolved, the company said.

"Most services recovered by 16:36 UTC as our Central US datacenter recovered; Actions was degraded until approximately 18:03 UTC; and Copilot Token Service fully recovered by 21:02."

What happened with the GitHub Outage?

After an investigation, GitHub said the issue was due to “network saturation” of load balancers due to new peak traffic in the Central US hosting region.

"Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits,” the company explained in a post-mortem report.

That failure cascaded, the post noted, with four HAProxy nodes exhausting their flow limits, degrading the gateway authentication path — all of which led to further authentication latency and failures, the GitHub said.

"The problem was worsened by optimistic retry logic which overloaded internal load balancers," the post added. "Pausing HAProxy on those nodes simultaneously produced immediate broad recovery."

Slow recovery

GitHub immediately moved some traffic away from the failing Central US data center location to Northern Virginia to give time to debug and resolve the issues. But Northern Virginia was quickly flooded with traffic too because of a retry bug.

That was pinned on "client retry behaviour" – in other words, client software kept trying to make it work over and over again, leading to failed requests and amplified load.

GitHub managed to temporarily block the "retry storm" but still saw a huge increase in Copilot authentication.

"Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS. Reducing gateway authentication retries and blocking retry-triggering responses stabilized Copilot Token Service and completed recovery," the post noted.

External factors exacerbated the issue, according to GitHub, including a number of “scraping attacks on codeload endpoints”.

What's next?

GitHub said it had solutions lined up to avoid a similar outage. That includes fixing the initial autoscaling policies and auditing Istio request, concurrency ,and scaling limits, in order to avoid the same flaw recurring.

It also plans to review "retry limits and backoff behaviour across gateways and clients," and address the VS Code retry behaviour that led to the massive leap in Copilot token traffic.

More widely, GitHub plans to improve load balancer capacity monitoring and failover safeguards.

FOLLOW US ON SOCIAL MEDIA

Follow ITPro on Google News and add us as a preferred source to keep tabs on all our latest news, analysis, views, and reviews.

You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.