How rate limiting works

· 2 min read

#how-it-works#web

Rate limiting caps how many requests a client can make in a given time. A token bucket is one way to do it: tokens drip into a bucket at a steady speed, and every request has to spend one.

The short version

Picture a bucket that fills with tokens at a fixed rate, up to its capacity. A request takes one token. If a token is there, the request goes through. If the bucket is empty, the server answers 429 Too Many Requests, and it may add a Retry-After header that says how long to wait. The capacity is what lets a short spike through: a full bucket can pay for many requests at once.

A token bucket decides whether a request is allowed A request arrives at a bucket of tokens. A dashed arrow from a token refill box adds tokens to the bucket at a steady rate. If a token is left, the request is allowed. If the bucket is empty, the server answers 429 with a Retry-After header. Token refill Request Bucket capacity = burst Allowed one token spent 429 + Retry-After wait, then retry steady rate token left bucket empty
Simplified: each service decides for itself who counts as one client.

Step by step

  1. Tokens are added to the bucket at a steady rate, up to its capacity.
  2. A request arrives and asks for a token.
  3. If a token is left, it is spent and the request is allowed.
  4. If the bucket is empty, the server rejects the request with 429.
  5. The 429 may carry Retry-After, either a number of seconds or an HTTP date.
  6. The client waits, and tries again.
Setting What it controls
Rate (AWS: throttling rate) how fast tokens are added, so the long-run request rate
Burst (AWS: throttling burst) the bucket's capacity, so the most requests a sudden spike can use

Why it matters

The burst allowance suits real traffic, which arrives in spikes. The catch is that RFC 6585 leaves open how a server recognises a client and counts its requests, so every service decides that for itself. When you get a 429, look for Retry-After before you retry.

Sources