How rate limiting works
Rate limiting caps how many requests a client can make in a given time. A token bucket is one way to do it: tokens drip into a bucket at a steady speed, and every request has to spend one.
The short version
Picture a bucket that fills with tokens at a fixed rate, up to its capacity. A request takes one token. If a token is there, the request goes through. If the bucket is empty, the server answers 429 Too Many Requests, and it may add a Retry-After header that says how long to wait. The capacity is what lets a short spike through: a full bucket can pay for many requests at once.
Step by step
- Tokens are added to the bucket at a steady rate, up to its capacity.
- A request arrives and asks for a token.
- If a token is left, it is spent and the request is allowed.
- If the bucket is empty, the server rejects the request with 429.
- The 429 may carry
Retry-After, either a number of seconds or an HTTP date. - The client waits, and tries again.
| Setting | What it controls |
|---|---|
| Rate (AWS: throttling rate) | how fast tokens are added, so the long-run request rate |
| Burst (AWS: throttling burst) | the bucket's capacity, so the most requests a sudden spike can use |
Why it matters
The burst allowance suits real traffic, which arrives in spikes. The catch is that RFC 6585 leaves open how a server recognises a client and counts its requests, so every service decides that for itself. When you get a 429, look for Retry-After before you retry.
Sources
- RFC 6585, section 4: 429 Too Many Requests: the 429 status for too many requests in a given time, and that the specification does not say how clients are identified or requests counted
- MDN, Retry-After: how long to wait before a new request, as seconds or an HTTP date
- AWS API Gateway, request throttling: the token bucket, one token per request, throttling rate and throttling burst, and 429 when the limit is exceeded