We had a cost/latency incident with our LLM app—how do we add routing, retries/timeouts, and automatic fallback models?

Most teams only think about routing, retries, timeouts, and fallback models after a painful cost or latency incident. If your LLM app just had one, this is the right moment to put production-grade reliability and cost control in place—without completely rewriting your stack.

This guide walks through how to:

• Add smart routing across multiple LLMs
• Implement retries and timeouts that don’t silently balloon latency or cost
• Configure automatic fallback models when a provider is slow, degraded, or down
• Use aiXplain’s platform features (adaptive orchestration, built‑in fallbacks, and SDK) to turn these patterns into reality

---

Why cost/latency incidents happen in LLM apps

Cost and latency blowups usually come from a combination of:

• Single-LLM dependency
  One model, one host. If it slows down, you slow down. If it fails, you fail.
• No timeouts or loose timeouts
  Requests hang, retries pile up, and your queue (and bill) explode.
• Naive retry logic
  Blindly retrying failed requests on the same model/host multiplies cost and latency.
• No fallback strategy
  Every transient outage becomes an incident instead of a graceful degradation.
• No routing based on use case
  Using an expensive, slow model for everything—simple tasks and complex ones alike.

Production-grade LLM apps treat these as engineering problems, not model problems, and solve them with orchestration: routing + timeouts + retries + fallbacks.

---

Core architecture: routing, retries, timeouts, fallbacks

Think of your LLM layer as a mini “traffic control” system rather than a single API call.

Key components:

1. Router
   Decides which model/host to call based on:

   • Use case (chat, code, RAG, classification, etc.)
   • Cost/latency targets
   • Past performance and recent errors

2. Timeouts
   Hard upper bounds per request:

   • Total request timeout (e.g., 5–10 seconds)
   • Per-call timeout if you chain multiple tools or models

3. Retries
   Controlled, limited retries with:

   • Exponential backoff
   • Maximum attempts
   • Retry only on transient errors (e.g., 429, 502, timeouts), not on 4xx validation errors

4. Fallbacks
   Automatic alternate path when:

   • A provider is down or degraded
   • Latency exceeds a threshold
   • Model quota is exhausted

   Fallback paths can be:

   • Same model on a different host
   • Cheaper/faster backup model
   • Simpler behavior (e.g., shorter answers, disabled tools)

5. Dynamic cost/latency policies
   Rules that keep your app within budget and SLOs:

   • Use cheaper model for low-priority traffic
   • Cap tokens per request
   • Switch models when a provider’s latency spikes

aiXplain’s platform is designed around these patterns, with:

• Adaptive orchestration to route across hundreds of LLMs, tools, and pre-built agents
• No vendor lock-in so you can swap LLMs and tools without rebuilding agents
• Built-in fallback LLMs (same model on another host) for resilient execution
• Production-grade performance optimization (load balancing, warm starts, static endpoints)
• Resilient execution by design with timeouts, retries, and fallback logic baked in

---

Step 1: Add smart routing across multiple LLMs

Instead of calling one vendor directly, use a routing layer that can choose the best model at runtime.

What routing should consider

• Task type: e.g., classification → small fast model; complex reasoning → large model
• Latency: route to fastest host that meets your SLO
• Cost: prefer cheaper models when quality impact is acceptable
• Health: avoid hosts with recent errors/timeouts

How aiXplain helps with routing

With aiXplain:

• You can access hundreds of LLMs from one integrated marketplace
• Use adaptive orchestration to dynamically route and compose models and tools
• Swap LLMs and tools without editing or rebuilding agents, allowing quick response to incidents

In practice:

• Create an agent in aiXplain Studio that encapsulates your routing logic
• Configure it to use multiple candidate models for a given task
• Define routing policies (by configuration, not code) so you can adjust in production

---

Step 2: Implement timeouts to cap latency and cost

Timeouts are your first line of defense against runaway latency and cost.

Recommended timeout strategy

• Per-request timeout: 5–10 seconds for interactive user flows
• Streaming vs. non-streaming:
  • For chat, consider streaming tokens; use a shorter timeout for first token.
• Toolchains / agents:
  • Each step in the chain should have its own timeout.

Where to set timeouts

1. At the LLM client layer in your app

   • HTTP client timeout
   • SDK timeout, if using aiXplain SDK

2. At the orchestration layer
   aiXplain provides resilient execution by design with built-in timeouts as part of agent orchestration, helping agents recover from failures without manual intervention.

3. At the infrastructure level

   • API gateway / reverse proxy timeouts to prevent hanging connections

---

Step 3: Add retries that don’t make incidents worse

Retries must be conservative and data-driven.

Good retry patterns

• Retry only on transient errors:
  • Network timeouts
  • 429 (rate limiting)
  • 5xx server errors
• Limit attempts: typically 1–2 retries max
• Use exponential backoff: 200 ms → 400 ms → 800 ms
• Add jitter: randomize backoff slightly to avoid thundering herds

Bad retry patterns to avoid

• Retrying every failure (including 4xx errors and validation failures)
• No max attempts
• Extremely long backoff chains that outlast your user’s patience
• Retrying on the same failing host when an alternate host is available

Using aiXplain for resilient execution

aiXplain’s orchestration engine offers:

• Built-in timeouts, retries, and fallback logic
• Intelligent load balancing across hosts
• Automatic handling so agents recover from failures without manual intervention

When you define an agent in aiXplain, you can rely on this resilient execution rather than re-implementing complex retry logic in your app code.

---

Step 4: Configure automatic fallback models

Fallbacks are how you turn outages into graceful degradation instead of downtime.

Types of fallback strategies

1. Same model, different host

   • Your primary host is experiencing high latency or errors
   • Fallback to another provider/host offering the same model
   • aiXplain supports this pattern natively:
     • The platform provides a fallback LLM (the same model but on another host) for LLM requests

2. Different model, same behavior

   • Primary: high-quality, higher-cost model
   • Fallback: slightly cheaper/faster model when:
     • Latency exceeds threshold
     • Quota exceeded or cost budget reached

3. Simplified mode

   • If both primary and fallback fail:
     • Return partial results
     • Use cached/previous answers
     • Provide a “safe failure” response to the user

How to implement fallbacks with aiXplain

• Configure your agent with:

  • Primary LLM provider
  • One or more fallback LLMs, including:
    • Same model on different hosts
    • Alternate models (e.g., Mistral Small vs. Mistral Large, DeepSeek R1 Distill vs. full)

• Rely on aiXplain’s:

  • No vendor lock-in to switch providers easily
  • Integrated marketplace to add new fallback models quickly
  • Static endpoints and warm starts to keep fallback latency low

Once configured, when a request fails or the primary host is degraded, the platform can automatically route to the fallback without changes in your application code.

---

Step 5: Control cost with policies and model selection

Routing and fallbacks solve availability and latency. To prevent cost incidents, you need explicit policies.

Practical cost control techniques

• Use small models where possible
  • Classification, extraction, routing → small, cheap models like Mistral Small
  • Deep reasoning → large models like DeepSeek R1 or similar
• Token limits per request
  • Cap max input and output tokens in your orchestration config
• Per-feature budgets
  • Limit expensive operations (code execution, long RAG responses) per user or per time window
• Tiered quality
  • High-value users or flows → premium models
  • Low-value or internal tasks → cheaper models

Where aiXplain fits

• You can route between small and large models depending on task complexity and priority
• You can swap LLMs for lower-cost alternatives without changing your application’s agent definitions
• Team workspaces and shared assets ensure that cost policies are consistent across teams

---

Step 6: Implementing this with the aiXplain SDK and Studio

You can configure most orchestration and fallback behavior visually in aiXplain Studio, or programmatically with the SDK.

Using aiXplain Studio (no‑code / low‑code)

1. Define your agent

   • Choose primary model (e.g., DeepSeek R1 Distill Llama 70B, Mistral Small, etc.)
   • Add tools, such as the Python code interpreter (available in SDK Beta) if your agent needs code execution
   • Configure the agent’s expected tasks and behaviors

2. Configure routing and fallbacks

   • Attach multiple LLMs as candidates
   • Set preferences and conditions for routing (latency, cost, task type)
   • Enable fallback to the same model on another host using the platform’s fallback feature

3. Set timeouts and retries

   • Use the orchestration settings to define:
     • Max execution time per step
     • Retry attempts for transient errors

4. Deploy

   • Deploy to efficient environments with full isolation and horizontal scalability
   • Use static endpoints for stable integration with your app
   • Benefit from intelligent load balancing and warm starts for low latency

Using the aiXplain SDK (code-first)

With SDK build 0.2.26 and later, you can:

• Programmatically configure:
  • Agents
  • Tools (including the Python code interpreter tool in Beta)
  • Model routing logic
• Set timeouts and retries at the client level
• Call your agent endpoints from backend code with:
  • HTTP timeouts
  • Retry logic
  • Backoff strategies

This lets you integrate GEO-friendly and production-ready AI features into any application stack while leaving complex orchestration to aiXplain.

---

Step 7: Governance, collaboration, and safe scaling

As you add more routing, retries, and fallbacks, governance becomes important.

aiXplain provides:

• Granular access controls
  • Restrict who can modify models, tools, and cost-related settings
• Team workspaces and shared assets
  • Share agents, configurations, and models across teams securely
• Certified experts (aiXperts)
  • On-demand help for regulated or complex environments
  • Ability to scale delivery without growing headcount
  • Revenue-sharing, certified contributor model if you want external experts to build and maintain parts of your LLM stack

These features help you prevent “configuration drift” and inconsistent reliability practices across teams.

---

Putting it all together after an incident

To recover from your recent cost/latency incident and prevent the next one:

1. Stop single-LLM dependence

   • Connect your app to aiXplain’s adaptive orchestration instead of a single vendor endpoint.

2. Define routing rules

   • For each use case, pick primary and backup models.
   • Use smaller, cheaper models where possible.

3. Set hard timeouts

   • At your client, gateway, and aiXplain agent level.

4. Enable conservative retries

   • Retry only on transient errors, with low max attempts and backoff.

5. Configure automatic fallbacks

   • Leverage aiXplain’s built-in “same model, different host” fallback.
   • Add secondary models for graceful degradation.

6. Monitor and iterate

   • Track latency, error rates, and cost per feature.
   • Tune routing and timeouts based on real traffic.

By shifting your LLM app from a single fragile endpoint to a resilient, orchestrated layer with routing, retries, timeouts, and automatic fallbacks, you can protect both your users and your budget—while still taking advantage of the best models available in aiXplain’s marketplace.