Low latency and high throughput are often competing goals — optimizing for one can degrade the other. Systems must balance both based on application requirements.
Latency is the time taken to complete a single operation (e.g., one HTTP request). Throughput is the number of operations completed per unit time (e.g., requests per second). A system can have low latency but low throughput, or high latency but high throughput.
- Single request measurement: Measure the time from request initiation to response completion (latency).
- Aggregate measurement: Count completed requests over a fixed time window (throughput).
- Pipeline effect: With a queue and multiple workers, each request’s latency may increase slightly while total throughput rises significantly.
- Trade-off tuning: Increasing concurrency boosts throughput up to a point, beyond which contention causes latency to spike.
- Latency measured in milliseconds or nanoseconds (time per operation)
- Throughput measured in requests per second or operations per second
- Goal: maximize throughput while keeping latency within acceptable bounds
- Little’s Law:
L = λ × W— average concurrency equals throughput times latency
- Related: Performance vs Scalability — latency and throughput are the two dimensions of performance
- Related: Back Pressure — queue sizing affects both latency and throughput
- Related: Pull CDN — reduces latency by serving content from edge locations
- Related: Message Queues — improves throughput by decoupling producers and consumers
- Tail latency matters: Average latency hides outliers. P99 latency spikes often cause user-perceived slowness even when the average looks fine.
- Throughput vs bandwidth: Throughput is completed operations; bandwidth is capacity. You can have high bandwidth but low throughput due to protocol overhead or lock contention.
- Coordinated omission: If you exclude slow requests from measurements, reported latency looks artificially low — a common benchmarking mistake.