How do you handle rate limits and retries with AI APIs?

AI APIs enforce per-minute and per-day token/request limits. Handle them with exponential backoff retries, request queues, and model fallbacks so rate limit errors never reach your users.

AILLM

Overview

Every AI API enforces rate limits: requests per minute (RPM), tokens per minute (TPM), and sometimes tokens per day. When you exceed them you get a 429 response. In production these errors are inevitable during traffic spikes. The right architecture handles them transparently with automatic retries, queuing, and fallback models so end users never see an error.

Key Points

•HTTP 429 Too Many Requests means you hit a rate limit. The response header Retry-After tells you how long to wait.
•Exponential backoff: wait 1s, then 2s, then 4s, then 8s between retries - with jitter to avoid thundering herd.
•The OpenAI Node.js SDK has built-in retries - set maxRetries in the client constructor.
•Token limits are harder to hit but more dangerous: a single large document can consume your entire TPM allowance.
•Use a queue (Bull, Inngest, pg-boss) for batch workloads so requests are metered, not burst.
•Model fallback: if GPT-4o is rate-limited, automatically retry with GPT-4o-mini.

Examples

OpenAI SDK built-in retries

typescript

import OpenAI from 'openai';

// The SDK retries automatically on 429 and 5xx errors
const client = new OpenAI({
  apiKey:     process.env.OPENAI_API_KEY,
  maxRetries: 3,          // default is 2
  timeout:    30_000,     // 30 second request timeout
});

// All API calls on this client automatically retry with exponential backoff
const response = await client.chat.completions.create({
  model:    'gpt-4o',
  messages: [{ role: 'user', content: 'Hello' }],
});

Manual exponential backoff with jitter

typescript

async function withRetry<T>(
  fn: () => Promise<T>,
  options = { maxAttempts: 5, baseDelayMs: 1000 }
): Promise<T> {
  for (let attempt = 1; attempt <= options.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error: unknown) {
      const isRateLimit =
        error instanceof Error &&
        'status' in error &&
        (error as { status: number }).status === 429;

      const isLastAttempt = attempt === options.maxAttempts;

      if (!isRateLimit || isLastAttempt) throw error;

      // Exponential backoff with random jitter
      const delay = options.baseDelayMs * Math.pow(2, attempt - 1);
      const jitter = Math.random() * 500;
      const waitMs = delay + jitter;

      console.warn(`Rate limited. Attempt ${attempt}/${options.maxAttempts}. Waiting ${Math.round(waitMs)}ms...`);
      await new Promise(resolve => setTimeout(resolve, waitMs));
    }
  }
  throw new Error('Max retries exceeded');
}

// Usage
const response = await withRetry(() =>
  client.chat.completions.create({ model: 'gpt-4o', messages })
);

Model fallback chain

typescript

const MODEL_FALLBACK_CHAIN = [
  'gpt-4o',
  'gpt-4o-mini',
  'gpt-3.5-turbo',
] as const;

async function completionWithFallback(
  messages: OpenAI.ChatCompletionMessageParam[]
): Promise<OpenAI.ChatCompletion> {
  for (const model of MODEL_FALLBACK_CHAIN) {
    try {
      const result = await client.chat.completions.create({
        model,
        messages,
        timeout: 20_000,
      });
      if (model !== MODEL_FALLBACK_CHAIN[0]) {
        console.warn(`Used fallback model: ${model}`);
      }
      return result;
    } catch (error: unknown) {
      const isRateLimitOrOverload =
        error instanceof Error &&
        'status' in error &&
        [429, 503].includes((error as { status: number }).status);

      if (!isRateLimitOrOverload || model === MODEL_FALLBACK_CHAIN.at(-1)) {
        throw error;
      }
      console.warn(`Model ${model} unavailable, trying next...`);
    }
  }
  throw new Error('All models in fallback chain failed');
}

Request queue for controlled throughput (using p-queue)

typescript

import PQueue from 'p-queue';

// Limit to 50 concurrent requests and 3000 requests per minute
const queue = new PQueue({
  concurrency: 50,
  intervalCap: 3000,
  interval: 60_000,  // 1 minute window
});

// All LLM calls go through the queue
async function queuedCompletion(messages: Message[]) {
  return queue.add(() =>
    client.chat.completions.create({ model: 'gpt-4o', messages })
  );
}

// Process 10,000 items without hitting rate limits
const results = await Promise.all(
  items.map(item => queuedCompletion([{ role: 'user', content: item.text }]))
);

Pro Tip

Check your API dashboard for your actual rate limits before writing retry logic. OpenAI tier 1 gives you 500 RPM and 30k TPM on GPT-4o - a single large document can hit the TPM limit in one request. Monitor your usage in the dashboard and request a tier upgrade before traffic spikes, not during them.