Dev Enock

Designed a Distributed Rate Limiter as part of my System Design practice. 🚀 Here is how I would design one for a distributed API system. 1. API Gateway Apply rate limiting at the API Gateway. This allows us to reject requests before they reach backend services. 2. Token Bucket Algorithm Use the Token Bucket algorithm. It allows clients to send short bursts of requests while keeping their average request rate within the configured limit. 3. Redis Cluster Store token bucket state in Redis. All API Gateway instances use the same Redis state. This prevents each instance from maintaining a separate request count. 4. Redis Lua Scripts Use Lua scripts to make token refill and request validation atomic. This prevents two requests from consuming the same available token at the same time. 5. PostgreSQL Store rate limit rules in PostgreSQL. For example: 100 requests per minute. 5,000 requests per hour. Different limits for different API keys or endpoints. 6. Kafka + Transactional Outbox When an admin changes a rate limit, save the change and an event in the same database transaction. A background worker reads the event and publishes it to Kafka. All API Gateway instances receive the update and refresh their local configuration. 7. Local Configuration Cache Each API Gateway keeps a local copy of the rate limit rules. This avoids a database query for every API request. Important design trade-offs: 1. Shared Redis State vs Local State With Redis, all API Gateway instances use the same token bucket for a customer. This helps enforce one common rate limit across all gateways. However, every request needs to contact Redis. This adds network latency and increases Redis traffic. With local state, each gateway maintains its own token bucket. This removes the Redis call, but each gateway has an independent limit. A customer can exceed the intended global limit by sending requests through different gateways. 2. Centralised Token Buckets vs Local Token Leases With centralised token buckets, every request checks Redis before it can proceed. This provides more accurate rate limiting, but increases the number of Redis calls. With local token leases, a gateway reserves a batch of tokens from Redis and uses them locally. This reduces Redis calls, but some reserved tokens may remain unused on one gateway while another gateway needs them. We gain lower latency, but give up some flexibility in how tokens are shared. 3. Fail Open vs Fail Closed What happens if Redis becomes unavailable? Fail open: Allow requests without checking the rate limit. The API remains available, but customers may exceed their limits. Fail closed: Reject requests because the rate limit cannot be checked. This protects our infrastructure, but valid customers may lose access. The choice depends on the cost and risk of allowing unchecked requests. 4. Global vs Regional Token Buckets With a centralised Redis cluster, all regions share the same token bucket for a customer. This provides a common view of token usage, but requests from distant regions may have higher network latency. With regional token buckets, each region maintains its own bucket and quota. This reduces network latency, but unused tokens in one region cannot automatically be used by another region. A shared global limit requires coordination between regions, which adds latency and complexity. Key takeaway: System design is not only about choosing the right components. It is about understanding what we gain, what we give up, and why a particular design makes sense for our requirements.