Rate limits
Per-key request and concurrency caps.
A key can have:
- Requests per minute, counted over a sliding window.
- Concurrency, the number of calls running at the same time.
Going over returns 429 rate_limited with a Retry-After header in seconds. Upstream rate limits also come back as 429 with their own Retry-After, so retry after that many seconds either way.
Aliases with more than one model retry the next model on an upstream 429 or 5xx, as long as nothing was streamed yet.