Throttling, retries and rate limits for AI workflows

An AI workflow depends on other services: the model provider, your CRM, a payment API, a mailbox. Each can be slow, busy or briefly down. The way you handle that decides whether you get a quiet second attempt or a duplicate payment and a flooded support inbox.

This post covers the patterns that matter: when to retry, how to space retries, how to keep a retry from doing the work twice, and what to do when retries run out.

Start with the failure type#

Not every error deserves a retry. Sort them first.

• Temporary: timeouts, connection resets, 5xx server errors, 429 "too many requests". A retry may well succeed.
• Permanent: 400 bad request, 401 or 403 permission errors, 404 not found. Repeating the same request gives the same result. Fix the cause.
• Content problems: the model returned malformed output or refused. A retry may help, but change something (a clearer instruction, a lower temperature), or you are just paying for the same failure again.

A workflow that retries permanent errors wastes money and hides the real fault. Route them straight to a failure path with a useful message.

Backoff with jitter#

If a service is struggling and every client retries immediately, the retries become the load. The usual remedy is to wait longer between each attempt (exponential backoff) and to randomise the wait (jitter).

An AWS engineering article on the subject puts the reason bluntly: retries are "selfish" because they spend more of the server's capacity to raise one client's chance of success. It also explains why jitter matters: without randomness, clients that fail together retry together. A simple, safe pattern is waits of roughly 1, 2, 4 and 8 seconds, each with some random variation, then stop.

Two more points from the same article are worth borrowing:

• Retry at one layer. If your workflow retries, the tool inside it retries and the API client retries too, the attempts multiply. The article's example shows three retries across five layers amplifying load on a database by 243 times. Decide where retrying happens and turn it off elsewhere.
• Limit total retries. Cap attempts per run, and ideally cap retries as a share of overall traffic, so an outage does not triple your request volume.

Respect the rate limit headers#

When a provider says "slow down", listen. Many APIs return a 429 status and a Retry-After header telling you how long to wait. The HTTP standard (RFC 9110) defines Retry-After as a response header that indicates how long the client should wait before making a follow-up request, given either as a number of seconds or a date. If the header is present, use it instead of your own backoff guess.

For model providers, rate limits often apply per minute to requests or tokens. A workflow that fans out over 500 documents at once will hit them. Control the pace instead of hoping:

• Process items in small batches or one at a time inside a loop.
• Add a short wait between batches.
• Use a cheaper or higher-limit model for bulk steps, and keep the premium model for the steps that need it.

Idempotency: make a retry safe#

The scariest retry is a request that actually succeeded but whose reply was lost. You retry, and the payment is taken twice.

The fix is idempotency. You attach a unique key to the request, and the service uses it to recognise a repeat and return the original result instead of repeating the work. Stripe's API documents this clearly. You send an Idempotency-Key header, Stripe saves the status code and body of the first request made with that key, and later requests with the same key return the saved response. Keys can be up to 255 characters, Stripe suggests random values such as V4 UUIDs, and in API v1 keys can be removed after they are at least 24 hours old.

The AWS article makes the matching point: calls with side effects aren't safe to retry unless the API provides idempotency.

In practice:

1. Generate one key per logical action (not per attempt) and store it with the run.
2. Send the same key on every attempt of that action.
3. If a service has no idempotency support, check first: look up whether the thing already exists before creating it.
4. Never retry "send message" or "charge card" steps blindly. Look first or gate with a person.

Inbound is different: webhooks and queues#

When another system calls your workflow, it will retry too. Stripe's webhook documentation tells receivers to return a 2xx status quickly before doing complex work, to expect the same event more than once, and to track processed event IDs to avoid duplicates. It also says events may arrive out of order, so do not assume sequence.

The same rules apply to any workflow triggered from outside: acknowledge fast, process once, and record what you have seen.

What to do when retries run out#

Plan the ending before you build the loop.

• Fall back. Try a second model or an alternative route for the same step. Some model gateways let you list several models in priority order and move to the next on errors like rate limiting or downtime, billing at the rate of the model that handled the request.
• Park it. Put the item in a "needs attention" queue with the error, so nothing is lost silently.
• Tell someone. A failed run that nobody hears about is a data-quality problem waiting to happen.
• Compensate. If step 3 failed after step 2 had an effect, decide whether to undo step 2 or leave it and flag it.

Building it in Cadrix#

Cadrix does not hide retries behind a single switch, so you build the pattern from visible steps. That is a feature when something goes wrong, because every attempt is in the trace.

1. Make the call with an HTTP step (or a connected tool) and follow it with a Condition that reads the result. Split success, temporary failure and permanent failure into three branches.
2. On the temporary-failure branch, add a Custom Function step that works out the next wait (doubling each attempt, plus a random amount) and pauses for that long, then send the flow back through a Loop step to the call. Set the Loop's maximum so it stops after a handful of attempts. Check that your deployment allows the timer in a Custom Function before relying on this.
3. Pass an idempotency key, built from the run's identifier, in the HTTP step's headers where the service supports one.
4. For bulk work, use an Iteration over the items so they are handled one at a time, with the same short pause between them, instead of firing everything at once.
5. Configure an Error workflow, which can be set as a default for a whole workspace, to alert a person and park the failed item.
6. When a run fails, open the trace to see which step failed and what it received. You can retry the step or re-run with edits, and the result is a new linked run rather than an overwrite.

Checklist#

• Classify errors: retry only the temporary ones.
• Backoff with jitter, a hard cap on attempts, retries at one layer.
• Honor Retry-After.
• One idempotency key per logical action, reused across attempts.
• Acknowledge inbound webhooks fast and de-duplicate by event ID.
• A defined end state: fallback, parking queue, alert.

Sources#

• AWS Builders' Library, Timeouts, retries, and backoff with jitter (Dec 2019)
• Stripe API docs, Idempotent requests
• Stripe docs, Receive Stripe events in your webhook endpoint
• RFC 9110, HTTP Semantics, section 10.2.3 Retry-After
• OpenRouter docs, Model fallbacks