index

What Actually Broke

Here’s the chain of events:

A permissions change in a ClickHouse database caused a query to return a much larger file than expected—one filled with duplicate entries. That file fed into Cloudflare’s Bot Management system, which then propagated it to the core proxy layer (the “FL” system).

The proxy had a hardcoded memory limit: 200 features max.

When the new file arrived with way more than 200 features, the proxy didn’t log an error, throttle gracefully, or fall back to a cached version.

It panicked. And when it panicked, it returned HTTP 5xx errors to users across the web.

This cascaded through services like Workers KV, Cloudflare Access, and even their own login system (via Turnstile). For a while, their team couldn’t diagnose the issue because the errors fluctuated—only happening when the query hit an updated part of the database cluster every five minutes. They initially suspected a DDoS attack.

By the time they identified the real cause and manually deployed a known good configuration, hours had passed.

The Database Didn’t Fail—The Software Did

Let’s be clear: the database did exactly what it was told to do. A permissions change altered query results. That’s not a bug—that’s how databases work.

The failure was in how the downstream system handled unexpected input.

A well-defended system would have:

• Validated the input size before loading it into memory
• Logged a warning and continued with the last known good configuration
• Degraded gracefully rather than crashing entirely
• Had circuit breakers to isolate the failure and prevent cascading errors

Instead, the proxy hit a limit, panicked, and brought down a significant portion of Cloudflare’s network.

This is a defense programming problem, not a data problem.

Single Points of Failure Are Expensive

The other issue here is architectural: the core proxy layer was a single point of failure.

When that layer went down, it took Workers KV, Access, and other services with it—because they all depended on the same broken component. Even Cloudflare’s own dashboard became inaccessible.

In distributed systems, you can’t eliminate every failure mode. But you can design around them. The fix Cloudflare applied during recovery—patching Workers KV to bypass the core proxy—is exactly the kind of escape hatch that should have existed from the start.

When a critical system fails, you need an alternative path. A bypass. A way to keep the lights on while you diagnose and fix the root cause.

In racing, if your engine overheats, you don’t keep redlining it until it seizes. You back off, let it cool, and figure out what’s wrong. The same principle applies to software: when things go wrong, the system should degrade, not explode.

What This Means for the Rest of Us

If a company as sophisticated as Cloudflare can have this kind of outage, it’s a reminder that everyone is vulnerable to these mistakes.

Here’s what I think about after reading post-mortems like this:

Panic is not an acceptable failure mode. If your system encounters unexpected input—larger files, malformed data, missing dependencies—it should log, degrade, and continue. Crashing should be a last resort, not the default.

Build escape hatches. When a critical component fails, what’s your bypass? Can users still access core functionality? Can you manually override or roll back without rebuilding the entire system?

Test your limits. Cloudflare’s proxy had a hardcoded limit of 200 features. Did anyone ever test what happened when that limit was exceeded? If you don’t know how your system behaves under stress, you’re flying blind.

Failure modes cascade. One bad configuration file took down the proxy, which took down Workers KV, which took down Access, which locked engineers out of their own dashboard. Map your dependencies. Know what breaks when something fails.

The Fix Is Already Known

Cloudflare’s follow-up plan is solid: harden configuration ingestion, add global kill switches, and review failure modes. These are the right steps.

But here’s the thing—none of this is new. Defensive programming, circuit breakers, and bypass mechanisms have been best practices for decades. The lesson isn’t that these techniques exist. It’s that even experienced teams forget to apply them when building fast or under pressure.

I’ve been there. At a previous startup, we had a service that processed user uploads. One day, a malformed file caused the parser to hang indefinitely, which blocked the entire upload queue. We didn’t have a timeout. We didn’t have a fallback. We had to manually restart the service and patch it live.

The fix took five minutes. The outage lasted hours. The real cost wasn’t the downtime—it was the trust we lost with users who were trying to get work done.

Build Like Things Will Break

The Cloudflare outage is a reminder that systems fail. Databases change. Queries return unexpected data. Memory limits get exceeded. Networks partition. Disks fill up.

The question isn’t whether something will break. It’s whether your system is designed to handle the break gracefully or collapse under it.

Defense programming isn’t glamorous. It doesn’t show up in demos or launch announcements. But it’s the difference between an incident that gets resolved in minutes and one that takes down half the internet for six hours.

When you’re building, ask yourself: What happens when this breaks?

If the answer is “it panics,” you’ve got work to do.