How to Handle 429 Too Many Requests: Retry-After, Backoff and Real Concurrency

A 429 Too Many Requests response is not an instruction to sleep for five seconds and submit the same request again.

It tells you the server is limiting new work. Before choosing a retry delay, you need to know which limit you hit.

RFC 6585 defines 429 as a rate-limiting response. Provider contracts can narrow that behavior further. each::labs predictions, for example, document 429 as an account concurrency cap rather than a per-second prediction limit.

Those mechanisms recover differently.

A time-based rate limit clears because time passes. A concurrency limit clears because running work finishes. Exponential backoff controls how aggressively you probe again, but it cannot manufacture a free concurrency slot.

That distinction matters with generation APIs because image and video jobs may remain active long after the submission request returns. If an account can run two predictions at once and both slots are occupied, another 20 POST requests do not increase throughput. They create 20 more admission decisions.

The better place to handle predictable excess work is your scheduler.

Not every 429 means the same thing in practice

RFC 6585 says a 429 Too Many Requests response may include Retry-After, while leaving details such as how requests are counted to the implementation.

That implementation detail matters. Some APIs limit requests over a time window. Others also use 429 when too much work is already running. each::labs uses the latter interpretation for prediction concurrency.

For a client, three cases are worth separating:

Limit type	What is capped?	What creates headroom?	Primary client response
Rate window	Work started over time	Time/window recovery	Retry-After, pacing, backoff
Concurrency	Work simultaneously in flight	Existing work finishing	Queue + concurrency limiter
Temporary capacity	Immediate admission capacity	Service recovery	Bounded backoff with jitter

Time-window rate limiting

A conventional rate limit might allow 60 requests per minute or 10 requests per second. When the window advances, capacity returns.

Elapsed time is part of the mechanism. If the server tells you to retry after three seconds, waiting can make the next request valid even when nothing else in your application changes.

If the limit is known in advance, pacing is better than discovering it through repeated rejection. A client that knows it can start ten requests per second does not need to send 50 at once.

Concurrency limiting

API concurrency is the number of accepted operations that are still in flight at the same time. With asynchronous generation, that lifetime usually extends beyond the HTTP request that created the job.

Imagine the limit is two:

slot 1: job A — processing
slot 2: job B — processing

There is no third slot. One second later, there is still no third slot if A and B are both processing.

Capacity returns when active work leaves the in-flight set.

This is where a queue or worker pool matters. Your application should control how much work it admits instead of relying on rejected requests to tell it the obvious: all available slots are occupied.

Temporary admission throttling

There is also a transient case. Your nominal concurrency state can look healthy while a burst still exceeds immediate service capacity.

Backoff fits naturally here. Reduce retry pressure, add jitter so workers do not all return together, and stop after a bounded number of attempts.

Same status code. Different scheduling problem.

Read Retry-After before calculating your own delay

When an API returns Retry-After, use that signal before inventing a more aggressive local delay.

HTTP defines two forms. The value can be a number of seconds:

HTTP/1.1 429 Too Many Requests
Retry-After: 7

Or an HTTP date.

A generic parser should therefore handle both:

from datetime import datetime, timezone
from email.utils import parsedate_to_datetime


def parse_retry_after(value: str | None) -> float | None:
    if not value:
        return None

    value = value.strip()

    # RFC delay-seconds is a non-negative decimal integer.
    if value.isdigit():
        return float(int(value))

    try:
        retry_at = parsedate_to_datetime(value)
        if retry_at.tzinfo is None:
            retry_at = retry_at.replace(tzinfo=timezone.utc)

        now = datetime.now(timezone.utc)
        return max(0.0, (retry_at - now).total_seconds())
    except (TypeError, ValueError, OverflowError):
        return None

The parser is not the interesting part. The precedence is.

If the server asks you to wait seven seconds, a locally calculated one-second backoff is not a reason to retry sooner.

Retry-After also does not tell you what creates capacity. It tells you when a follow-up request may make sense. An unfinished generation still has to finish before a concurrency slot is actually free.

And not every 429 carries the header. In each::labs, the balance endpoint explicitly documents Retry-After: 1 for its own request-rate guard. The prediction concurrency documentation does not make the same guarantee.

Exponential backoff controls retry pressure

If a 429 is transient and the server has not supplied usable retry timing, exponential backoff is a reasonable fallback.

The basic shape is:

first failure  → short delay
second failure → longer delay
third failure  → longer again
...
cap the delay
stop after a finite number of attempts

A common base calculation is:

delay = min(max_delay, base_delay × 2^attempt)

With a base delay of 0.5 seconds:

attempt 0 → 0.5s
attempt 1 → 1.0s
attempt 2 → 2.0s
attempt 3 → 4.0s

That is enough to stop a tight retry loop and give the constrained service room to recover.

Plain exponential backoff still has one nasty property: it can preserve synchronization.

Why backoff needs jitter

Say 100 workers hit the same limit at the same moment.

Without jitter:

t=0s  → 100 workers fail
t=1s  → 100 workers retry
t=2s  → 100 workers retry again
t=4s  → 100 workers retry again

The traffic is less frequent, but it still arrives in waves.

Jitter spreads those attempts across the interval:

worker A → 0.8s
worker B → 1.4s
worker C → 0.5s
worker D → 1.7s

The exact jitter family is less important here than the principle: workers that fail together should not automatically return together.

A compact full-jitter helper looks like this:

import random


def backoff_delay(
    attempt: int,
    base_delay: float = 0.5,
    max_delay: float = 30.0,
) -> float:
    cap = min(max_delay, base_delay * (2 ** attempt))
    return random.uniform(0, cap)

Keep a retry budget

Backoff needs an end.

An infinite retry loop can turn one product action into an unbounded stream of requests. Put limits around maximum attempts and maximum delay. In many products, a maximum total retry window is useful too.

Then check the layers below your own code. If an SDK retries three times and your application wraps that call in another retry loop, the real number of attempts may be much larger than the code nearest the product action suggests.

Generative APIs add another constraint.

The semantics have to come before the pattern.

An idempotent read retried after a network failure still asks for the same resource. A generation is not automatically the same operation the second time you submit it.

If you know the API rejected the request before creating any work, resubmission is straightforward. If the request may already have been accepted, another call can create another inference run and a different artifact.

So the first question is not “How many times should I retry?” It is “Do I know this operation is safe to repeat?”

Only then does the backoff formula matter.

Why backoff alone does not solve a concurrency limit

Consider a generation API with a concurrency cap of two.

You have 20 jobs ready.

The naïve architecture does this:

submit job 1  → accepted
submit job 2  → accepted
submit job 3  → 429
submit job 4  → 429
...
submit job 20 → 429

Now you have two real jobs and 18 retry timers.

That is a strange way to build a queue.

If your application knows that it owns a two-slot concurrency budget, the normal state should be:

20 jobs ready
2 concurrency slots

Active:
  job 1
  job 2

Queued:
  jobs 3–20

When job 1 finishes:

Active:
  job 2
  job 3

Queued:
  jobs 4–20

Predictable excess work never reaches the API.

Keep these three mechanisms separate:

A queue holds work that cannot start yet.

A concurrency limiter decides how much work can start.

Backoff controls when a rejected attempt may probe again.

A concurrency slot represents unfinished work

For asynchronous generation, the important lifetime is not the POST request.

A submission can return an ID while the expensive work continues in the background. Holding a semaphore only until that response arrives limits simultaneous submissions, not simultaneous generations.

The slot needs to represent the whole in-flight lifetime:

accepted
   ↓
created / starting / processing
   ↓
success / error / cancelled
   ↓
slot reusable

For each::api predictions, the current prediction status reference documents created, starting, processing, success, error, and cancelled. The final three are terminal states.

A client-side worker should not think, “The POST returned, so my worker is free.”

It should think, “The generation was accepted. This local capacity slot stays occupied until that prediction settles.”

That is the difference between request concurrency and real asynchronous concurrency.

Do as little scheduling work as possible after the 429

One infrastructure rule I keep coming back to is simple: do as little cold work as possible after something breaks.

Applied here, your application should not receive a concurrency 429 and only then begin deciding how many jobs it ought to have admitted.

The queue should already exist. The concurrency ceiling should already be represented in the scheduler. Your application should already know which local work is active and which work is waiting.

Then a 429 tells you something useful: your view of available capacity was incomplete. Another application instance may share the account. The limit may have changed. Another form of throttling may be active.

That is a much better use for the error than treating it as your primary scheduler.

The normal path is completion-driven:

job finishes
    ↓
capacity becomes available
    ↓
scheduler admits next queued job

The protection path is rejection-driven:

submit
    ↓
429
    ↓
bounded wait
    ↓
probe again

You need both.

What an each::labs prediction 429 means today

The current Error Reference documents prediction 429 as an account concurrency cap. It explicitly says there is no per-second request limit on predictions.

While an organization's balance is $10.00 or less, the documented low-balance caps are:

Model pricing type	Concurrent predictions
Fixed list price	10
Metered / cannot be priced before execution	2

This is more specific than “low balance means two concurrent jobs.” Two applies to the documented metered case; fixed-list-price models use ten under the same balance condition.

The response's details field names the number that applied. Client code should inspect or surface that value instead of hard-coding one explanation for every 429.

The documentation also makes two useful guarantees for a request rejected by this concurrency check:

1. No prediction is created.
2. The rejected request is not billed.

That gives you a clean boundary:

submission rejected with concurrency 429
        ↓
no prediction exists
        ↓
work still needs to be submitted later

Compare it with:

prediction accepted
        ↓
generation starts
        ↓
prediction eventually returns error

The first is work that never entered execution. The second is an execution that existed. Retrying the second case is therefore a new product decision, not an automatic replay.

For the documented concurrency case, a slot becomes available when an in-flight prediction reaches a terminal state. The same Error Reference warns that backoff by itself does not clear that cap.

The documentation says this specific low-balance cap does not apply when the balance is above $10.00. It does not establish an unlimited-concurrency guarantee.

Queue large batches instead of turning the API into your queue

Batch size and concurrency are different numbers.

You may have:

2,000 jobs to process

without wanting:

2,000 simultaneous API calls

The first is a backlog. The second is an admission policy.

For a simple application that owns the full account quota, a bounded worker pool can be enough:

                 ┌─ worker 1 ─→ API
backlog / queue ─┼─ worker 2 ─→ API
                 └─ ...

If the relevant concurrency ceiling is two, at most two workers should own active asynchronous jobs. When one accepted job settles, that worker can take another item.

The model gets more complicated when several processes or services share the same account. A per-process semaphore cannot enforce an account-wide ceiling if ten processes each believe they own two slots.

That distributed-control problem deserves its own article. For this one, the operating rule is enough: put the backlog somewhere deliberate and make admission capacity explicit.

If a bounded worker still receives 429

Do not let every queued item spin up its own retry loop.

Instead:

1. inspect the response;
2. identify the limiting condition from the endpoint contract and response details;
3. use Retry-After if it is supplied;
4. otherwise use a bounded jittered delay before probing again;
5. keep the rejected work inside your bounded admission path;
6. reduce or pause new admissions if the response tells you your concurrency assumption is wrong.

For a concurrency limit, the delay does not free capacity. It only controls how often your application probes again when it cannot directly observe every other consumer of the quota.

Bulk Trigger groups asynchronous workflow work

For each::workflows, the Bulk Trigger endpoint starts the same workflow with multiple input objects.

The current documentation accepts 1–10 inputs. Each accepted input becomes its own workflow execution, and the executions share a bulk_id for correlation. The immediate response is 202 Accepted; accepted entries initially report queued. Invalid entries can fail independently without discarding the accepted executions.

That is useful for submission and grouping:

bulk request
    ↓
execution A — queued
execution B — queued
execution C — queued
    ↓
shared bulk_id

But keep the product claim narrow.

The public Bulk Trigger documentation covers parallel workflow submission, queueing, and grouping. It does not document Bulk Trigger as the account-level governor for the prediction concurrency caps described above.

Use bulk execution for the job it is documented to perform. Treat downstream model capacity as a separate constraint.

Use completion events instead of polling your way into another limit

Long-running work needs a completion signal.

Polling can be appropriate, and each::api exposes GET /v1/prediction/{id} for prediction status. But a high-volume application should avoid replacing one controlled generation workload with a second uncontrolled stream of status requests.

For each::workflows, the workflow webhook documentation lets you supply a webhook_url and receive a POST when an execution completes successfully or fails.

Workflow status vocabulary is separate from prediction status vocabulary. Workflow executions use states such as running, completed, failed, and cancelled; prediction polling uses created, starting, processing, success, error, and cancelled.

For bulk-triggered workflows, each execution sends its own notification and includes the shared bulk_id.

That gives your application a natural reconciliation event:

submit execution
       ↓
queued / running
       ↓
backend reaches terminal state
       ↓
webhook received
       ↓
mark local job terminal
       ↓
advance or reconcile backlog

The webhook tells your application that the terminal transition occurred.

The workflow webhook documentation recommends idempotent handlers using execution_id as a deduplication key and bulk_id to correlate bulk work.

The current first-party documentation is not consistent enough to justify reproducing an exact webhook retry schedule or a webhook_secret signature implementation here. Those details should come from the contract you verify when implementing the webhook.

One implementation: queue + concurrency + bounded retry

The Python below is a single-process scheduling example, not a copy-paste each::labs client. API-specific request parsing is intentionally separated from the scheduler.

The important property is that an accepted asynchronous generation keeps its worker occupied until it reaches a terminal state.

A request rejected with 429 also stays inside that bounded worker rather than spawning an independent retry loop.

import asyncio
import random
from dataclasses import dataclass


class RateLimited(Exception):
    def __init__(self, retry_after: float | None = None):
        super().__init__("rate limited")
        self.retry_after = retry_after


@dataclass
class Job:
    id: str
    payload: dict


async def submit_generation(job: Job) -> str:
    """
    API-specific function.

    Return a remote prediction/execution ID only after the API
    has unambiguously accepted the job.

    Raise RateLimited(...) only for an unambiguous 429 rejection.

    Do not automatically turn a timeout or dropped connection into
    a retry: first determine whether the server may already have
    accepted the generation.
    """
    raise NotImplementedError


async def wait_until_terminal(remote_id: str) -> dict:
    """
    API-specific completion function.

    A small implementation might poll conservatively.
    A workflow implementation might resolve completion from a
    webhook-backed local registry.

    Return only when the accepted job reaches a terminal state.
    """
    raise NotImplementedError


def jittered_backoff(
    attempt: int,
    base_delay: float = 0.5,
    max_delay: float = 30.0,
) -> float:
    cap = min(max_delay, base_delay * (2 ** attempt))
    return random.uniform(0, cap)


async def run_job(
    job: Job,
    *,
    max_retries: int = 4,
) -> dict:
    for attempt in range(max_retries + 1):
        try:
            remote_id = await submit_generation(job)

            # The asynchronous job was accepted.
            # Keep this worker occupied until the remote job settles.
            return await wait_until_terminal(remote_id)

        except RateLimited as exc:
            if attempt >= max_retries:
                raise

            # Retry-After controls the next probe when supplied.
            # Otherwise use a bounded jittered delay.
            delay = (
                exc.retry_after
                if exc.retry_after is not None
                else jittered_backoff(attempt)
            )

            # This worker stays occupied while waiting, so it does
            # not pull another item from the backlog and add pressure.
            await asyncio.sleep(delay)

    raise RuntimeError("unreachable")


async def worker(
    queue: asyncio.Queue,
    results: dict,
):
    while True:
        job = await queue.get()

        try:
            results[job.id] = await run_job(job)
        except Exception as exc:
            results[job.id] = {"error": str(exc)}
        finally:
            queue.task_done()


async def process_jobs(
    jobs: list[Job],
    concurrency: int,
) -> dict:
    queue = asyncio.Queue()
    results = {}

    for job in jobs:
        await queue.put(job)

    workers = [
        asyncio.create_task(worker(queue, results))
        for _ in range(concurrency)
    ]

    await queue.join()

    for task in workers:
        task.cancel()

    await asyncio.gather(*workers, return_exceptions=True)
    return results

With:

concurrency = 2

this process can have at most two workers owning asynchronous jobs or controlled admission attempts at once.

The important sequence is:

worker starts
    ↓
submission accepted
    ↓
worker remains occupied
    ↓
generation reaches terminal state
    ↓
worker becomes available
    ↓
next queued job starts

If you release the worker immediately after submit_generation() returns an ID, you are limiting simultaneous POST requests—not the asynchronous generations those POSTs created.

Where Retry-After fits

The API-specific submission layer can attach a parsed Retry-After value to RateLimited.

When it exists:

delay = exc.retry_after

Otherwise the worker uses bounded jitter:

delay = jittered_backoff(attempt)

For a real concurrency cap, neither delay creates headroom. It only controls the next probe. Capacity still comes from existing work settling.

The single-process caveat

This example controls one process.

If four application instances each configure:

concurrency = 2

the shared account can still see up to eight attempts or active jobs if nothing coordinates those instances.

Once several processes share one external quota, use an account-wide scheduler, distributed admission control, shared worker allocation, or another mechanism that represents the quota globally.

Do not mistake a local semaphore for a global limit.

Common 429 mistakes in generation systems

Retrying every 429 on the same timer

If every worker sleeps two seconds, they all wake together.

Use server timing when available and jitter locally calculated delays.

Assuming sleeping frees concurrency

Time passing and work finishing are not equivalent.

A five-second delay only helps a concurrency cap if relevant active work happens to settle during that interval. Build the normal path around completion.

Sending the entire batch and treating rejection as flow control

If you already know the concurrency ceiling, predictable excess work belongs in your queue.

A 429 is useful protection when your assumptions are wrong. It is a poor substitute for an admission policy.

Retrying a generation that may already have started

A clear pre-admission 429 is different from a network timeout after submission.

With the timeout, the server may have accepted the generation even though your client did not receive the acknowledgement.

Do not automatically create another nondeterministic generation until you know which state you are in.

Accidentally stacking retry layers

Check SDK behavior, HTTP-client retries, job-runner retries, workflow retries, and application retries together.

The system's real retry count can be much larger than the nearest for attempt in range(3) suggests.

429 and 402 can both tell you to wait, for different reasons

each::labs can also return 402 Payment Required when the available balance cannot cover a request, including reserved estimated cost from work already in flight.

Status	Constraint
429	Admission/concurrency capacity
402	Available balance / reserved estimated cost

Both may sometimes clear after existing predictions settle, but they require different handling.

What to do when you receive a 429

1. Inspect the response and endpoint contract.
2. Identify whether the constraint is a rate window, concurrency, or transient capacity.
3. Respect Retry-After when supplied.
4. Use capped exponential backoff with jitter for transient throttling.
5. Queue predictable excess work when concurrency is the limit.
6. Treat terminal completion as the event that creates new concurrency headroom.
7. Keep retries finite, and never blindly repeat a generation that may already have started.

For each::labs prediction 429, inspect details for the cap that applied. A submission rejected by the documented concurrency check creates no prediction and is not billed; see the Error Reference.

For asynchronous workflow batches, Bulk Trigger groups executions under a bulk_id, while workflow webhooks provide the completion signal your application can use for reconciliation.

Do not choose a retry delay until you know what resource the API is protecting.

FAQ

What does 429 Too Many Requests mean?

RFC 6585 defines HTTP 429 as a rate-limiting response. The precise implementation is endpoint-specific, and APIs may attach narrower admission semantics; each::labs predictions currently use it for an account concurrency cap.

Should I retry every 429?

No.

Read the endpoint contract and response details first. A transient time-window throttle usually supports a later retry. A concurrency cap needs capacity to become available as existing work settles. Some quotas cannot be fixed by retries at all.

How long should I wait after a 429?

Use Retry-After when the server supplies it.

Without that signal, the answer depends on what is limited. Use bounded jittered backoff for transient throttling. For a concurrency cap, fixed sleep is only a controlled probe; actual headroom comes from active work settling.

Does exponential backoff fix a concurrency limit?

No.

Backoff decides when you try again. A free concurrency slot appears when existing in-flight work leaves the active set.

What is jitter in API retries?

Jitter adds controlled randomness to retry timing so clients that fail together do not all retry together.

How do I process a batch larger than my concurrency limit?

Keep excess work queued and admit only as much asynchronous work as the concurrency budget allows.

When an active job settles, admit the next queued item.

Does an each::labs prediction rejected with 429 charge my account?

No, for the documented prediction concurrency rejection. The current each::labs Error Reference says no prediction is created and the rejected request is never billed.