LLM API Rate Limits: Handle 429s, Retries, and Concurrency Safely
By Nicho
A production LLM application needs more than a retry loop. It needs to know why a request was rejected, how much work it can admit, and when to stop trying.
The practical starting point is simple: classify the error, control request admission, and retry only failures that can plausibly recover within the operation's deadline. Increasing retries without these controls can turn a small traffic spike into a longer outage.
This guide covers provider-neutral application design. Exact quotas, error codes, headers, and accounting rules depend on your provider, model, account, and gateway.
Understand which limit you reached
These controls answer different questions:
| Control | What it measures | Application response |
|---|---|---|
| Request rate | Requests admitted within a time window | Pace starts and smooth bursts |
| Token rate | Token usage or estimated token demand | Reserve token capacity and constrain request size |
| Concurrency | Requests currently in flight | Limit workers or use a semaphore |
| Spend or balance | Financial availability | Check billing or budget controls |
OpenAI, for example, documents separate request and token limits, with scopes that can include projects and shared model families. A new API key is therefore not necessarily new capacity. Its documentation also distinguishes transient rate-limit errors from hard spending limits, even when both use 429. Read the error body rather than making decisions from the status alone. OpenAI rate-limit documentation
The HTTP standard defines 429 but does not prescribe how a service identifies callers or counts requests. It also makes Retry-After optional. Do not assume every OpenAI-compatible endpoint has identical throttling behavior. HTTP 429 specification
Size traffic around the tighter constraint
Consider a simplified, hypothetical allowance of 60 requests per minute and 120,000 tokens per minute. If each request consumes 4,000 counted tokens, token capacity supports only 30 such requests per minute:
120,000 tokens/minute ÷ 4,000 tokens/request = 30 requests/minute
This is a planning example, not a provider quota or exact scheduler. A service may reserve estimated output, use shorter enforcement windows, or account for different token categories differently.
Measure your own request-size distribution. Averages alone conceal large document uploads and long generations. Keep headroom for these outliers instead of targeting the mathematical maximum.
Use separate controls for starts and in-flight work. A concurrency cap alone does not enforce requests per minute: fast responses can free slots quickly enough to create excessive starts. Conversely, a request-rate limiter does not stop slow streams from accumulating and exhausting connections.
Put admission control before retries
A useful request path is:
Authenticated request → bounded queue → rate/token admission → worker → provider
Design the queue deliberately:
- Set a maximum queue length and maximum waiting time.
- Give interactive requests a shorter deadline than background jobs.
- Reject or defer excess work with a clear application-level response.
- Remove canceled and expired jobs before dispatch.
- Share admission state across replicas that consume the same upstream quota.
Otherwise, adding application servers can multiply traffic while upstream capacity stays unchanged. For mixed workloads, reserve some capacity for user-facing requests so a document-import job cannot occupy every slot.
These are architectural recommendations, not promises that a particular gateway exposes a queue or adjustable concurrency controls.
Use one bounded retry policy
Retries should have an owner. Inspect SDK behavior before adding another loop. In a hypothetical configuration, three application attempts wrapped around three SDK attempts can produce nine upstream attempts for one logical operation.
A practical policy is:
- Classify the failure using the provider's documented error information.
- Stop immediately for errors requiring changed credentials, permissions, billing, or request content.
- Retry eligible transient failures only while both attempt and time budgets remain.
- Honor valid server retry guidance; otherwise use capped exponential backoff with jitter.
- Return the final failure with enough context to investigate it safely.
Retry-After can specify a delay or a date. Treat an applicable server delay as a minimum, not a suggestion to shorten. If it exceeds the operation's remaining deadline, defer the work or fail clearly rather than retrying early. HTTP Retry-After semantics
Randomized waits reduce synchronized retry waves. Timeouts and bounded attempts keep retries from holding resources indefinitely. AWS's engineering guidance explains why retries can amplify overload and why a timeout does not prove an operation had no effect. AWS guidance on timeouts and retries
Do not blindly replay streams or agent actions
A stream that stops after displaying text is not the same as a request rejected before work began. Your interface needs an explicit incomplete state. Automatically starting again can duplicate text, consume additional tokens, or produce a different answer.
Agent workflows need an additional boundary: retrying model generation must not automatically replay completed tool actions. Record action outcomes and use idempotency mechanisms where the action service supports them. A failed response does not justify sending the same email or creating the same order twice.
Monitor logical work, not just HTTP success
Track these separately:
- Logical operations and upstream attempts.
- Queue wait and in-flight requests.
- Rejections by documented error reason.
- Input/output usage and cost per completed operation.
- Timeouts, canceled work, and incomplete streams.
Use request identifiers and sanitized metadata. Avoid logging authorization headers or complete prompts by default. A falling error rate is not a win if retries doubled the cost of each completed task.
Check gateway controls separately from capacity
cheapaimodel's published pricing and account information describes per-key budgets, expiration, model/IP allowlists, and a 403 response when account balance is exhausted. That balance condition is not something a short backoff resolves. Those controls also do not establish unlimited request or token throughput.
Before scaling an integration, check the current cheapaimodel documentation and confirm applicable limits for the route you intend to use. Keep admission control and retry policy configurable so changes do not require rewriting application logic.
The goal is predictable completed work: bounded waiting, explainable failures, controlled spending, and retries that help recovery instead of fighting it.