Why averages lie: p99 latency is what users actually feel

Source: Dev.to

Introduction

Most performance dashboards look fine:

• Average latency: low
• CPU: stable
• Memory: healthy

Yet users complain that “the app feels slow”. This isn’t a mystery—it’s a metrics problem.

The problem with averages

Imagine this system:

• 99 % of requests complete in 10 ms
• 1 % of requests take 1000 ms

The average latency is ~20 ms, which looks great on a chart.
But 1 out of every 100 users experiences a full‑second pause. That’s not an edge case; it’s a real user.

Why percentiles matter

• p50 – tells you what’s typical
• p90 / p95 – show rising variability
• p99 – shows where systems actually break down

If your p99 is bad, your system feels bad — even if averages look perfect.

Variance and jitter

High variance is often more damaging than uniformly slow performance.

• A stable 40 ms system feels faster than one that jumps between 5 ms and 200 ms.
• This variability (jitter) causes UI stutter, audio glitches, and dropped frames.

Latency Lens – a lightweight analysis tool

I built Latency Lens, a small CLI tool focused on tail behavior rather than averages. It provides:

• p50 / p90 / p95 / p99
• Jitter (standard deviation and MAD)
• Spike detection
• Worst‑case time windows

Installation

• GitHub:
• PyPI:

How to diagnose “slow feeling”

If users say your system feels slow:

1. Don’t look at averages
2. Examine p99
3. Check variance
4. Look for spikes over time

That’s usually where the truth lies.