Rate Limiting

Restricting the number of requests a client can make to a service within a time window, rejecting or delaying excess requests to protect the system and ensure fair usage.

429 Too Many Requests, typically accompanied by a Retry-After header telling the client when to try again.

• Protection: prevent overload and abuse (DoS, scraping, brute force).
• Fairness/cost: stop one client from monopolizing resources and control operational cost.

By user/account ID, API key, IP address, session, or endpoint — or a combination. The key defines the scope of "who" the limit applies to.

Deliberately slowing or limiting request processing. It's closely related to rate limiting; throttling may delay/queue requests rather than reject them outright, smoothing load instead of hard-blocking.

Commonly at the edge — the API gateway or reverse proxy — so bad traffic is rejected before reaching backends. It can also be applied per-service or even in client SDKs.

A short spike of many requests in a brief period. Some algorithms (token bucket) intentionally allow bursts up to a limit, while others (leaky bucket) smooth them into a steady rate.

A bucket holds up to B tokens and refills at R tokens/second. Each request removes one token; if the bucket is empty, the request is rejected (or waits). This allows bursts up to B (using accumulated tokens) while enforcing an average rate of R. It's memory-efficient (store token count + last-refill timestamp) and widely used.

Requests are added to a queue (the bucket) and processed ("leak") at a fixed rate; if the bucket overflows, requests are dropped. It smooths bursts into a constant output rate. Difference: token bucket allows bursts (spending saved tokens) while capping the average; leaky bucket enforces a strictly steady output and doesn't allow bursts. Token bucket is better for APIs that tolerate bursts; leaky bucket for protecting a downstream that needs constant pacing.

It counts requests per fixed calendar window (e.g. each minute) and resets at the boundary. A client can send the full limit at the end of one window and again at the start of the next — up to 2× the intended rate across the boundary. It's simple and memory-cheap but allows these edge bursts, which the sliding window fixes.

Store a timestamp for each request; to check the limit, count timestamps within the last window (e.g. last 60s), discarding older ones. It's precise (true rolling window, no edge bursts) but memory-heavy since it stores every request timestamp per key — costly at high volume. The sliding-window counter is an approximation that's much cheaper.

It combines two fixed-window counts to approximate a rolling window cheaply. Estimate = current window count + previous window count × (fraction of the window still overlapping). E.g. 25% into the current minute, weight the previous minute by 75%. It removes most of the fixed-window edge-burst problem while storing only two counters per key — a good accuracy/cost balance used in practice.

On success: X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset (or the standardized RateLimit-* headers) so clients can self-throttle. On a 429: Retry-After indicating how long to wait. Clear signaling lets well-behaved clients back off instead of hammering and getting repeatedly blocked.

Rate limiting enforces a predefined quota per client regardless of current system load (a policy/fairness mechanism). Load shedding drops requests reactively when the system is actually overloaded (a survival mechanism), often prioritizing important traffic and rejecting the rest to stay up. They complement each other: quotas for fairness, shedding for protecting availability under unexpected surges.

Depends on latency tolerance and traffic shape. Rejecting (429) is best for user-facing APIs where a fast failure and client retry is preferable to unbounded waiting. Queuing/throttling suits background or batch traffic that can tolerate delay and benefits from smoothing. Queues must be bounded (with timeouts) or they add latency and memory pressure and can mask overload. Often: reject at the edge, throttle internally.

A shared counter is needed so N servers enforce one global limit. Use a central store (Redis) with atomic operations — e.g. a Lua script that reads, checks, and updates the token bucket/window in one round trip to avoid races. To reduce load and latency on the store, use a two-tier scheme: each node keeps a local bucket and periodically reserves a batch of tokens from the central budget, trading exactness for scale. Handle clock skew by having the store own time. Decide fail-open vs fail-closed if the store is unavailable, and make the store HA.

Use token bucket keyed by user. Because a single global counter per user at that rate is a hotspot, prefer per-region enforcement with a fraction of the global budget allocated per region, or a two-tier limiter (local buckets reserving from a regional Redis). Shard the counter store by user key to spread load, use atomic Lua updates, and store bucket state as {tokens, last_refill_ts}. Return standard rate-limit headers. Accept small overshoot for scalability. If a strict global limit is required, funnel each user's traffic to one owning region/shard (at the cost of cross-region latency) — usually not worth it versus approximate global limits.

It's a trade-off between availability and protection. Fail open (allow traffic when the limiter can't check) keeps the API available but removes protection, risking overload/abuse during the outage. Fail closed (reject) preserves protection but can cause a self-inflicted outage from a limiter dependency failing. Common pragmatic choice: fail open for availability but keep a coarse local fallback limit (per-node) so you're not completely unprotected, and make the limiter store highly available so this rarely triggers. High-risk endpoints (auth, payments) may fail closed.

The check-and-increment must be atomic; otherwise two concurrent requests both read "under limit" and both proceed, exceeding it. Use a Lua script (executed atomically on the Redis server) that performs the entire read-decide-update in one step, or Redis atomic primitives like INCR with EXPIRE for fixed windows, or INCRBYFLOAT/sorted-set operations for sliding windows. Avoid client-side read-modify-write. Set TTLs so keys expire and don't leak memory, and ensure the script computes refill from stored timestamps rather than relying on client clocks.

Model limits as policies keyed by (subject, dimension): e.g. per plan tier, per endpoint cost, per API key, plus a global safety cap. Store quotas in a config/plan service the limiter consults (cached). Apply multiple limits simultaneously (a request must pass all: per-user, per-endpoint, per-IP) and return the most restrictive result. Weight expensive endpoints so they consume more tokens. Make limits hot-reloadable without redeploy, expose per-tier headers, and allow temporary overrides/bursts for trusted clients. Keep the evaluation O(1) per request to avoid the limiter itself becoming a bottleneck.

Naive clients that immediately retry on 429 create a feedback loop that amplifies load exactly when the system is stressed — a retry storm. Fixes: clients must honor Retry-After and use exponential backoff with jitter so retries spread out; servers can implement retry budgets (only allow retries to be a small % of traffic) and circuit breakers. On the server side, count each retry against the limit so retries don't bypass it. Idempotency keys let safe retries avoid duplicate side effects. The goal is for backoff to converge, not oscillate.

Per-IP limits fail when an attacker uses thousands of IPs (botnet) or is behind shared NAT/CDN. Layer defenses: authenticate and limit per account/API key (attacker needs many accounts); apply global and per-endpoint limits to bound total load; use anomaly detection and reputation scoring to identify coordinated behavior; challenge suspicious traffic (CAPTCHA/proof-of-work); and push filtering to the edge/CDN/WAF and upstream DDoS scrubbing so volumetric floods never reach your servers. Combine rate limiting with load shedding that prioritizes known-good, authenticated traffic during an attack. Be careful with IP limits behind proxies — use the correct client IP (e.g. X-Forwarded-For handling).

Fixed window: 1 counter/key, tiny memory, but boundary bursts (low accuracy). Sliding window log: stores every request timestamp — highest accuracy but O(requests) memory, impractical at scale. Sliding window counter: 2 counters/key, cheap, good approximation. Token/leaky bucket: store {count/level, timestamp} per key, O(1) memory, allow (bucket) or smooth (leaky) bursts with precise average rate. In distributed settings, all incur per-key state in a shared store, so key cardinality (number of users/IPs) drives memory; TTLs and the counter-based algorithms keep it bounded. Choose bucket or sliding-counter for the best accuracy/cost balance at scale.

A rate limit caps requests over a short rolling window (e.g. 100/minute) to control instantaneous load. A quota caps total usage over a long period (e.g. 1,000,000 calls/month), often tied to billing/plan. They're complementary: rate limits protect against bursts, quotas enforce overall consumption.

A hard limit is strictly enforced — requests over it are rejected. A soft limit is a warning threshold: exceeding it may trigger alerts, throttling, or notifications but still allows requests (perhaps temporarily), giving clients a chance to react before a hard cutoff.

Anonymous traffic can only be keyed by IP (coarse, spoofable, shared behind NAT), so give it a conservative limit and rely on additional edge defenses. Authenticated users are keyed by account/API key, allowing higher, per-plan limits and accurate attribution. Typically anonymous requests get a low IP-based limit while authenticated requests get generous per-user limits — encouraging sign-in and letting you identify abusers precisely.

In a fast shared store (usually Redis) so all server instances enforce one consistent limit. Purely per-instance in-memory counters mean N servers each allow the full limit, so the effective limit is N× too high, and counters vanish on restart/scale events. A shared store gives a global view; a two-tier design (local buckets reserving from the shared budget) reduces store load while staying approximately correct.

Run in monitor/shadow mode first: evaluate the proposed limit and log how many requests would be blocked, per client, without actually rejecting — so you see the impact and find false positives. Analyze which legitimate clients would be affected, notify them, and set limits above real usage with headroom. Roll out gradually (a fraction of traffic or specific tiers) with dashboards on 429 rates, and keep the limit hot-configurable for instant rollback. Provide clear headers and documentation so clients adapt. Never flip a strict new limit globally at once.

Publish limits in docs and return standard headers (RateLimit-Limit/Remaining/Reset, Retry-After) so well-behaved clients self-throttle. For clients that keep hammering after 429s: escalate penalties (temporary bans / exponential lockouts), since responding to abusive traffic still costs resources; enforce at the edge/WAF so rejected requests are cheap; and consider tarpitting or connection-level throttling for persistent offenders. Track repeat offenders and, for authenticated abuse, revoke or throttle the API key. The key idea is to make ignoring 429s progressively more costly for the client while keeping rejection cheap for you.

GCRA is a precise rate-limiting algorithm (from ATM networking) equivalent to a leaky bucket but implemented without a background drain process. It tracks a single value — the "theoretical arrival time" (TAT) — and, on each request, checks whether now is at or beyond TAG minus an allowed burst tolerance; if so it admits the request and advances TAT by the emission interval, otherwise rejects. Benefits: O(1) time and memory (one timestamp per key), smooth pacing with configurable burst tolerance, and no separate refill job — making it efficient for distributed limiters (e.g. Redis cell modules). It's more precise than fixed-window and cheaper than a sliding-window log.