Concurrency Scheduler

See Also

Concurrency Scheduler Reference

The Concurrency Scheduler is used to schedule requests based on importance while ensuring that the application adheres to concurrency limits.

The Concurrency Scheduler can be thought of as a combination of a Scheduler and a Concurrency Limiter. It essentially provides scheduling capabilities atop a Concurrency Limiter. Similar to the Concurrency Limiter, this component takes max_concurrency as an input port which determines the maximum number of in-flight requests in the global request ledger.

The global request ledger is shared among Agents in an agent group. This ledger records the total number of in-flight requests across the Agents. If the ledger exceeds the configured max_concurrency, new requests are queued until the number of in-flight requests drops below the limit or until timeout.

note

Only accepted requests are counted towards the in-flight concurrency.

In a scenario where the maximum concurrency is known upfront, the Concurrency Scheduler becomes particularly beneficial to enforce concurrency limits on a per-service basis.

The Concurrency Scheduler also allows the definition of workloads, a property of the scheduler, which allows for strategic prioritization of requests when faced with concurrency constraints. As a result, the Concurrency Scheduler ensures adherence to the concurrency limits and simultaneously offers a mechanism to prioritize requests based on their importance.