SRE Weekly Issue #515

Source: SRE Weekly

Highlights

“Is your dashboard always green because everything is working, or because your metrics are lying?”
— Barnadeep Bhowmik, Stackademic

“But when we rolled out the new query, disk writes doubled and Write‑Ahead Logging (WAL) syncs quadrupled. We discovered that even when an upsert doesn’t change any values, it still locks the conflicting row, which is recorded in the WAL.”
— Anthonin Bonnefoy, Datadog

“It’s important not just to try to prevent incidents but to be fully ready for them when they inevitably happen anyway.”
— Joe Mckevitt, Uptime Labs

“Queues absorb spikes but not sustained overload. Without backpressure, limits, and monitoring, backlogs grow until systems fail.”
— David Iyanu Jonathan, DZone

“The code exhausted all ephemeral ports and then they logged itself to death complaining about it. I love the workaround. Loopback is a /8!”
— Jim Calabro, Bluesky

“AI didn’t solve the problem for them, but it sped up the grunt‑work and significantly reduced their iteration time, letting them get to an answer much faster.”
— Tristan Streichenberger, Mixpanel

“It’s interesting to me that this is essentially an outage/degradation report, but the definition of system degradation for an LLM tool is much more subjective than with traditional services. The ‘ablation testing’ concept is really neat.”
— Anthropic