Making LLM API Calls Reliable with Bounded Retries and Timeouts

Introduction

Interacting with Large Language Model (LLM) APIs introduces unique challenges compared to standard REST services. Due to high latency and the risk of rate limiting, your application must be prepared to handle transient failures gracefully. Without proper retry logic and timeouts, a single network hiccup or a busy model endpoint can cause your entire application flow to hang or crash.

The Importance of Timeouts

LLM requests are often long-running. However, you should never allow an API call to hang indefinitely. Setting a strict timeout ensures that your application releases resources if the provider is unresponsive. Using the native fetch API or libraries like axios, you can enforce these limits.

const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), 30000); // 30 second limit

try {
  const response = await fetch("https://api.openai.com/v1/chat/completions", {
    signal: controller.signal,
    // ... headers and body
  });
} catch (err) {
  if (err.name === "AbortError") {
    console.error("Request timed out");
  }
}

Implementing Exponential Backoff

Simply retrying immediately after a failure often exacerbates the problem, especially if the API is rate-limited. Exponential backoff increases the wait time between retries, giving the service time to recover. A common pattern is to wait 2^n * base_delay milliseconds.

async function fetchWithRetry(url, options, retries = 3, backoff = 1000) {
  try {
    return await fetch(url, options);
  } catch (error) {
    if (retries <= 0) throw error;
    await new Promise((resolve) => setTimeout(resolve, backoff));
    return fetchWithRetry(url, options, retries - 1, backoff * 2);
  }
}

Handling Rate Limits (429 Errors)

When dealing with LLM providers, you will eventually encounter HTTP 429 "Too Many Requests" errors. Your retry logic should specifically look for this status code. If the response header provides a Retry-After value, you should respect it rather than using your own calculated backoff.

Best Practices for Reliability

1. Bounded Retries: Always set a maximum number of retries (usually 3 to 5) to prevent infinite loops.
2. Jitter: Add a small amount of random delay to your backoff to prevent "thundering herd" problems where multiple clients retry at the exact same time.
3. Circuit Breakers: If the LLM provider is consistently failing, use a circuit breaker pattern to stop making requests entirely for a cooling-off period.

Conclusion

Building reliable AI-powered features requires moving beyond simple fetch calls. By implementing timeouts and exponential backoff, you protect your infrastructure and ensure a smoother experience for your users. For more on managing external dependencies, check out our guide on postgresql-connection-pooling-serverless to see how similar reliability patterns apply to database connections.