Concurrency Scheduler Reference
The Concurrency Scheduler is used to schedule requests based on importance while ensuring that the application adheres to concurrency limits.
The Concurrency Scheduler can be thought of as a combination of a
Scheduler and a
Concurrency Limiter. It essentially provides
scheduling capabilities atop a Concurrency Limiter. Similar to the
Concurrency Limiter, this component takes
max_concurrency as an input port
which determines the maximum number of in-flight requests in the global request
The global request ledger is shared among Agents in an
agent group. This ledger records the total number
of in-flight requests across the Agents. If the ledger exceeds the configured
max_concurrency, new requests are queued until the number of in-flight
requests drops below the limit or
Only accepted requests are counted towards the in-flight concurrency.
In a scenario where the maximum concurrency is known upfront, the Concurrency Scheduler becomes particularly beneficial to enforce concurrency limits on a per-service basis.
The Concurrency Scheduler also allows the definition of workloads, a property of the scheduler, which allows for strategic prioritization of requests when faced with concurrency constraints. As a result, the Concurrency Scheduler ensures adherence to the concurrency limits and simultaneously offers a mechanism to prioritize requests based on their importance.