Why p95 and p99 matter more than average latency
Average latency often hides real production issues. Why p95 and p99 latency give a more accurate view of system reliability and real user experience in high-traffic environments.
Average latency often creates a false sense of confidence. Dashboards look green, SLAs appear healthy and systems are assumed to be performing well. However, real production incidents almost never occur at the “average” level.
They happen at the edges.
That is why, in high-traffic, mission-critical systems, percentile latency — especially p95 and p99 — provides a far more accurate picture of system behaviour than simple averages.
p95 represents the latency under which 95% of requests complete, while p99 reflects the experience of the slowest 1%. This small percentage is where most real-world failures originate.
Why averages are misleading
Averages hide outliers. If 99 requests complete in 50 ms and one takes 5 seconds, the average still looks acceptable. But for some users, the system feels broken.
This discrepancy explains why teams often see healthy dashboards while customers report timeouts and degraded experiences.
In API gateways, IAM layers, payment flows and integration platforms, these outliers tend to cascade across services, amplifying the impact.
p95 vs p99: when to use which
p95 is useful for understanding general system performance. It reflects what most users experience. However, in regulated and high-availability environments, p95 alone is insufficient.
p99 captures rare but critical events:
- Timeouts
- Circuit breaker activations
- Retry storms
- Thread pool exhaustion
These events almost always begin in the p99 range. Monitoring p99 allows teams to detect problems before they escalate into outages.
p99 from an API gateway perspective
API gateways are often blamed for latency issues, but in practice they usually expose problems rather than cause them.
Common scenarios include:
- Rare but extremely slow downstream responses
- Intermittent latency in IAM or token validation flows
- Low cache miss rates with very expensive misses
- Traffic approaching rate limits
In all these cases, average latency remains stable while p99 degrades sharply. Observing p99 at the gateway layer provides early visibility before application teams notice the issue.
Defining SLAs and SLOs with p99
Mature organisations avoid defining SLAs based on averages. Instead, they rely on percentile-based targets:
- SLAs defined on p95 or p99
- Tighter SLOs for internal objectives
- Error budgets calculated from p99 violations
For example:
- “API response time p99 < 800 ms”
- “Monthly error budget calculated on p99 timeouts”
This approach aligns performance metrics with real user experience rather than theoretical averages.
Conclusion
p95 and p99 do not measure how fast your system is on good days. They measure how reliable it is on bad days.
In production systems with high traffic and complex dependencies, p99 is not just a metric — it is an operational instinct.
Because users never experience the average. They experience your system at its slowest moments.
Frequently asked questions
Why are p95 and p99 latency more important than the average?
Average latency can mask the real user experience because it combines high and low values together. p95 and p99 show the worst-case times experienced by 95% and 99% of users. Particularly in sectors like e-commerce and finance, the slowest 1–5% typically represents the most valuable customers (high transaction volume, complex queries).
How is p99 latency measured?
p99 latency is measured with histogram-based metric collection tools. All request response times within a given time window are sorted, and the 99th percentile value is taken. Adequate sample size, appropriate bucket intervals and a distinction between client-side and server-side measurements are important for accurate measurement.
What should the ideal p99 latency value be?
The ideal p99 value depends on the use case and SLA requirements. For the API gateway layer, under 500 ms is typically targeted. For internal calls between microservices, under 100 ms is expected. For database queries, under 50 ms is ideal. What matters as much as the absolute value is keeping the gap between p50 and p99 (tail latency amplification) at an acceptable level.
What techniques are used to improve latency percentiles?
Tail latency improvement techniques include reducing connection overhead with connection pooling, bypassing slow backends with hedged request patterns, preventing cascading failures with circuit breakers, managing dynamic deadlines with adaptive timeouts and optimising the hot path with caching strategies. Runtime optimisations such as GC tuning and JIT warm-up also significantly affect p99.