When protections outlive their purpose: A lesson on managing defense systems at scale

Source: GitHub Blog

Maintaining Platform Availability

The Problem

Sometimes, protections that were added as emergency responses can outlive their usefulness and start blocking legitimate users. This typically happens when rapid incident response requires broader controls that aren’t intended to be permanent.

What We Learned

• Outdated mitigations can degrade the user experience.
• Observability is as critical for defenses as it is for features.
• Prompt feedback from users is vital for identifying and removing stale protections.

Our Apology

We’re sorry for the disruption. We should have detected and removed these protections sooner.

Next Steps

• Review and retire any emergency mitigations that are no longer needed.
• Enhance monitoring to catch unintended side‑effects of defensive controls.
• Continue gathering user feedback to ensure our defenses remain transparent and effective.

What Users Reported

We saw reports on social media from people receiving “Too many requests” errors during normal, low‑volume browsing—for example, when following a GitHub link from another service or app, or simply browsing without any obvious pattern of abuse.

Users encountered a “Too many requests” error during normal browsing.

These were users making a handful of normal requests that hit rate limits which should not have applied to them.

What We Found

Investigating the reports revealed the root cause: protection rules added during past abuse incidents were left in place. These rules were created from patterns that, at the time, were strongly associated with abusive traffic. Unfortunately, the same patterns also matched some logged‑out requests from legitimate clients.

Composite Signals & False Positives

• The patterns combine industry‑standard fingerprinting techniques with platform‑specific business logic.
• Composite signals can occasionally produce false positives.

Key Statistics

Note: Only fingerprint matches that also satisfied business‑logic patterns resulted in a block.

Visual Summary

Although the percentages are low, any incorrect blocking disrupts real users and is therefore unacceptable.

Why Temporary Mitigations Can Become Permanent

During an active incident we must act quickly, often accepting trade‑offs to keep the service available. Emergency controls are correct at that moment, but they do not age well as threat patterns evolve and legitimate usage changes. Without active maintenance, temporary mitigations become permanent and their side effects compound silently.

Tracing Through the Stack

When users reported errors, we traced the requests across multiple infrastructure layers to pinpoint where the blocks occurred.

Multi‑Layer Protection Overview

GitHub’s custom, multi‑layered protection infrastructure (built on open‑source projects like HAProxy) applies defenses at several points:

1. Edge tier – DDoS protection, IP reputation, rate limits.
2. Application tier – Authentication, per‑user/service rate limits.
3. Service tier – Business‑logic validation, feature‑specific throttling.
4. Backend tier – Resource‑level quotas, internal safety checks.

Diagram (simplified to avoid disclosing specific mechanisms):

Each layer can independently block or rate‑limit a request. Determining which layer performed the block requires correlating logs that have different schemas.

Investigation Steps

By moving backward from the user‑visible error to the underlying rule configuration, we confirmed that the lingering mitigation rules were the source of the false positives.

Lifecycle of Incident Mitigations

The following diagram illustrates how a mitigation can outlive its usefulness:

What Went Wrong

• Controls were added during an incident and functioned as intended.
• No expiration dates, post‑incident reviews, or impact monitoring were applied.
• Over time the rules became technical debt, eventually blocking legitimate traffic.

Recommended Practices

1. Set explicit expiration dates for any emergency rule.
2. Conduct post‑incident reviews to assess ongoing impact.
3. Automate impact monitoring (e.g., alert on unexpected block rates).
4. Document ownership so the responsible team can retire or adjust rules.

Applying these practices ensures that temporary defenses remain temporary and that legitimate users are not inadvertently impacted.

What We Did

• Reviewed the mitigations – compared what each rule is blocking today with its original intent.
• Removed outdated rules – eliminated any protections that no longer serve a purpose.
• Retained essential safeguards – kept the controls that continue to defend against ongoing threats.

What We’re Building

Beyond the immediate fix, we’re improving the lifecycle management of protective controls:

• Better visibility across all protection layers to trace the source of rate limits and blocks.
• Treat incident mitigations as temporary by default; making them permanent should require an intentional, documented decision.
• Post‑incident practices that evaluate emergency controls and evolve them into sustainable, targeted solutions.

Defense mechanisms— even those deployed quickly during incidents—need the same care as the systems they protect. They require observability, documentation, and active maintenance. When protections are added during incidents and left in place, they become technical debt that quietly accumulates.

Thanks to everyone who reported issues publicly! Your feedback directly led to these improvements. And thanks to the teams across GitHub who worked on the investigation and are building better lifecycle management into how we operate. Our platform, team, and community are better together!

Tags

• developer experience
• incident response
• Infrastructure
• observability
• site reliability

Written by

Thomas Kjær Aabo – Engineer on the Traffic team at GitHub.

Explore More from GitHub