3 posts tagged with "adaptive-rate-limiting"

In modern engineering organizations, service owners don't just build services, but are also responsible for their uptime and performance. Ideally, each new feature is thoroughly tested in development and staging environments before going live. Load tests designed to simulate users and traffic patterns are performed to baseline the capacity of the stack. For significant events like product launches, demand is forecasted, and resources are allocated to handle it. However, the real world is unpredictable. Despite the best-laid plans, below is a brief glimpse of what could still go wrong (and often does):

• Traffic surges: Virality (Slashdot effect) or sales promotions can trigger sudden and intense traffic spikes, overloading the infrastructure.
• Heavy-hitters and scrapers: Some outlier users can hog up a significant portion of a service's capacity, starving regular user requests.
• Unexpected API usage: APIs can occasionally be utilized in ways that weren't initially anticipated. Such unexpected usage can uncover bugs in the end-client code. Additionally, it can expose the system to vulnerabilities, such as application-level DDoS attacks.
• Expensive queries: Certain queries can be resource-intensive due to their complexity or lack of optimization. These expensive queries can lead to unexpected edge cases that degrade system performance. Additionally, these queries could push the system to its vertical scaling boundaries.
• Infrastructure changes: Routine updates, especially to databases, can sometimes lead to unexpected outcomes, like a reduction in database capacity, creating bottlenecks.
• External API quotas: A backend service might rely on external APIs or third-party services, which might impose usage quotas. End users get impacted when these quotas are exceeded.

Imagine a bustling highway system, a complex network of roads, bridges, tunnels, and intersections, each designed to handle a certain amount of traffic. Now, consider the events that lead to traffic jams - accidents, road work, or a sudden influx of vehicles. These incidents cause traffic to back up, and often, a jam in one part of the highway triggers a jam in another. A bottleneck on a bridge, for example, can lead to a jam on the road leading up to it. Congestion creates many complications, from delays and increased travel times, to drivers getting annoyed over wasted time and too much fuel burned. These disruptions don’t just hurt the drivers, they hit the whole economy. Goods are delayed and services are disrupted as employees arrive late (and angry) at work.

But highway systems are not left to the mercy of these incidents. Over the years, they have evolved to incorporate a multitude of strategies to handle such failures and unexpected events. Emergency lanes, traffic lights, and highway police are all part of the larger traffic management system. When congestion occurs, traffic may be re-routed to alternate routes. During peak hours, on-ramps are metered to control the influx of vehicles. If an accident occurs, the affected lanes are closed, and traffic is diverted to other lanes. Despite their complexities and occasional hiccups, these strategies aim to manage traffic as effectively as possible.

We’ve been hearing about rate limiting quite a lot these days, being implemented throughout popular services like Twitter and Reddit. Companies are finding it more and more important to control the abuse of services and keep costs under control.

Before I started working as a developer advocate, I built quite a few things, including integrations and services that catered to specific business needs. One thing that was common while building integrations was the need to be aware of rate limits when making calls to third-party services. It’s worth making sure my integration doesn't abuse the third-party service API. On the other hand, third-party services also implement their own rate-limiting rules at the edge to prevent being overwhelmed. But how does all this actually work? How do we set this up? What are the benefits of rate limiting? We’ll cover these topics, and then move on to the reasons why adaptive rate limiting is necessary.