How do I autoscale a FastAPI inference service to hundreds or thousands of workers without rewriting my app?

Most FastAPI inference services break long before the model does—not because Python can’t keep up, but because the underlying infrastructure can’t autoscale cleanly from a single node to hundreds or thousands of workers. The usual “solution” is a rewrite: new container layout, new job queue, new deployment stack. You don’t need that. You can keep your FastAPI app and plug it into an autoscaling runtime that’s designed for spiky, GPU-heavy workloads.

Quick Answer: You can autoscale a FastAPI inference service to hundreds or thousands of workers by wrapping your existing app in a Modal @modal.fastapi_endpoint or @modal.asgi_app, then defining hardware, scaling, and lifecycle in Python. Modal handles container builds, cold-start optimization, and elastic GPU/CPU scaling—so you keep your FastAPI code and swap out the underlying infrastructure.

Why This Matters

If your LLM or model server needs to jump from a few requests per second to hundreds or thousands, you quickly run into all the boring-but-hard problems: prewarming replicas, managing GPU quotas, avoiding overprovisioning, and not blowing your latency SLO every time traffic spikes. Hand‑rolled autoscaling around FastAPI usually turns into a mesh of Docker, custom job queues, and YAML that’s hard to reason about.

By defining your FastAPI service as a Modal app, you get:

• autoscaling from zero to thousands of replicas in minutes,
• sub‑second cold starts for most workloads, and
• GPU capacity pooled across clouds,

all from Python. Your FastAPI routes stay the same; you just tell Modal how to run them.

Key Benefits:

• No rewrite of your FastAPI app: Wrap your existing ASGI app; keep your routes, Pydantic models, and middleware.
• Elastic GPU and CPU scaling: Scale replicas from zero to thousands across a multi‑cloud capacity pool without managing nodes or quotas.
• Production‑grade operations built in: Timeouts, retries, job queues, metrics, and logging via Modal primitives instead of ad‑hoc glue code.

Core Concepts & Key Points

Concept	Definition	Why it's important
Code-defined infra (Modal App)	A Python file where you declare your image, hardware, scaling, and FastAPI endpoints using Modal decorators.	Keeps infrastructure in the same language and repo as your FastAPI app, so you don’t juggle Terraform/YAML to change capacity or hardware.
Autoscaling replicas	Modal containers running copies of your FastAPI app that auto‑scale up/down based on traffic.	This is how you go from one process to hundreds or thousands of workers. Modal can spin up many replicas in minutes, then scale back to zero.
Stateful model servers	@app.cls classes with @modal.enter used to load models once per container and reuse them across requests.	Prevents reloading large models on every request, which is critical for LLM/ML inference latency and throughput.

How It Works (Step-by-Step)

At a high level you:

1. Wrap your existing FastAPI app in a Modal app.
2. Tell Modal what environment and hardware to use.
3. Deploy and let Modal handle autoscaling.

Let’s walk through it.

1. Start from your existing FastAPI app

Suppose you have a typical FastAPI inference server:

# app_fastapi.py
from fastapi import FastAPI
from pydantic import BaseModel
import torch

app = FastAPI()

class InferenceRequest(BaseModel):
    text: str

class InferenceResponse(BaseModel):
    output: str

# Imagine this wraps your LLM or other model.
model = torch.jit.load("model.pt")
model.eval()

@app.post("/infer", response_model=InferenceResponse)
def infer(req: InferenceRequest):
    with torch.inference_mode():
        # dummy example
        output = model([req.text])[0]
    return InferenceResponse(output=output)

On a single VM, you’d run this behind Uvicorn/Gunicorn and hit a wall when CPU/GPU is saturated. You can’t just “add more pods” without wrestling with orchestration and model loading.

2. Define a Modal Image and app

We’ll keep app as-is and define a Modal app in a new file. Modal Images are like Dockerfiles expressed in Python—pin Python version and dependencies once, and Modal builds a reproducible container for you.

# modal_fastapi_autoscale.py
import modal

app = modal.App("fastapi-autoscale-example")

# Build an image with your dependencies.
image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install(
        "fastapi==0.110.0",
        "uvicorn[standard]==0.29.0",
        "torch==2.2.0",
    )
)

Best practice: pin versions tightly so you can reproduce behavior and avoid dependency drift.

3. Wrap your FastAPI app for autoscaling

Modal has built‑in support for FastAPI via @modal.fastapi_endpoint and generic ASGI via @modal.asgi_app. The decorator turns your FastAPI app into a production endpoint that Modal can autoscale.

from fastapi import FastAPI
from pydantic import BaseModel
import torch

# Let’s get the imports out of the way first; then we’ll define the server.

class InferenceRequest(BaseModel):
    text: str

class InferenceResponse(BaseModel):
    output: str

def create_app(model):
    app = FastAPI()

    @app.post("/infer", response_model=InferenceResponse)
    def infer(req: InferenceRequest):
        with torch.inference_mode():
            output = model([req.text])[0]
        return InferenceResponse(output=output)

    return app


@app.cls(
    image=image,
    gpu="A10G",          # or "A100", "H100", or just CPU
    keep_warm=1,         # keep at least one replica hot
    concurrency_limit=8, # max concurrent requests per container
)
class InferenceServer:
    def __init__(self):
        self._app = None
        self._model = None

    @modal.enter()
    def load_model(self):
        # This runs once per container, not per request.
        self._model = torch.jit.load("model.pt")
        self._model.eval()
        self._app = create_app(self._model)

    @modal.asgi_app()
    def fastapi_app(self):
        # Modal mounts this ASGI app on an HTTPS endpoint.
        return self._app

A few things going on here:

• @app.cls makes a stateful server. Modal keeps instances of InferenceServer alive inside containers.
• @modal.enter is a lifecycle hook; it runs once when the container starts. That’s where you load the model.
• @modal.asgi_app tells Modal that fastapi_app is an ASGI app it should serve. Your FastAPI routes are unchanged; you just instantiate them inside the class.

This pattern avoids the classic “reload the model per request” problem. The model is loaded once per container, then reused for all requests that hit that replica.

4. Configure scaling and deployment

By default, Modal will autoscale replicas based on in‑flight requests and load. You can also tune concurrency and container counts in the decorator:

@app.cls(
    image=image,
    gpu="A100:2",        # 2x A100 GPUs per container
    keep_warm=2,         # keep 2 warm for low latency
    concurrency_limit=16,
    timeout=60 * 5,      # 5 minute timeout for long requests
)
class InferenceServer:
    ...

To run locally against Modal’s cloud:

modal serve modal_fastapi_autoscale.py

modal serve gives you a dev endpoint that reloads on code changes. Once you’re happy with it:

modal deploy modal_fastapi_autoscale.py

You’ll get a stable, HTTPS endpoint in the Modal UI (or via CLI output). Traffic to that endpoint will:

• scale your containers from zero to many (hundreds+),
• spread load across regions/hardware you selected,
• and reuse model-loaded containers as long as they’re warm.

You don’t have to change your client: it’s still just HTTP requests hitting /infer with the same JSON payload.

5. Scaling out to hundreds or thousands of workers

When you say “thousands of workers,” what you usually mean is “thousands of replicas concurrently processing requests or jobs.” On Modal:

• A replica is one container running InferenceServer.
• A replica can use one or multiple GPUs (e.g., A100:2).
• A high‑throughput LLM service is built by scaling out these replicas.

Modal can scale services from zero to thousands of replicas in minutes. Under load, Modal’s scheduler:

1. Measures current in‑flight requests and per‑container concurrency.
2. Requests new containers in its multi‑cloud capacity pool (GPUs and CPUs).
3. Starts containers with sub‑second cold starts for typical Python apps, faster than a full Docker pipeline.
4. Tears them down as load drops, scaling back to zero.

You don’t need to explicitly manage node pools or autoscaling groups; your knob is concurrency and hardware in the Python decorator.

If you also have background jobs (e.g., evals, batch scoring) tied to this service, capture them in the same app and scale with .spawn() and a job queue.

@app.function(
    image=image,
    gpu="A10G",
    timeout=60 * 10,
    retries=modal.Retries(max_retries=3),
    max_containers=1000,  # fan-out upper bound
)
def batch_infer(texts: list[str]) -> list[str]:
    import torch
    model = torch.jit.load("model.pt")
    model.eval()

    with torch.inference_mode():
        outputs = model(texts)
    return outputs

From another function or an external client you can:

# Fan-out many inference jobs in parallel
calls = [batch_infer.spawn(batch) for batch in batches]
results = [c.get() for c in calls]

Modal’s job queue will spin up additional containers as needed to process jobs concurrently.

Common Mistakes to Avoid

• Loading the model on every request:
  Doing torch.jit.load() inside your FastAPI route obliterates throughput. Instead, use @app.cls + @modal.enter to load once per container and reuse across requests.

• Treating CPU and GPU services identically:
  A small FastAPI JSON API can stick with CPU autoscaling. For LLMs or heavy models, attach GPUs explicitly (gpu="A10G" or "A100") and tune concurrency_limit based on profiling. Otherwise you either leave GPU underutilized or overload it and blow your latency budget.

• Ignoring timeouts and retries:
  Production inference sees tail latencies, transient GPU hiccups, and upstream issues. Set timeout and retries=modal.Retries(...) on your functions so you fail fast or retry cleanly instead of silently hanging clients.

• Not pinning dependencies in the Image:
  Leaving dependencies floating (torch, fastapi, etc.) can change behavior across deployments. Always pin versions in your Image definition and only bump when you intend to.

• Doing cross-region data access by accident:
  If you’re fetching models or data from external storage (e.g., S3), avoid hardcoding a region that doesn’t match your Modal region. Use Modal Volumes for large cached assets when possible; they’re colocated with compute and avoid surprise latency/cost.

Real-World Example

Imagine you’re running a FastAPI-based LLM inference service that currently sits on a single A100 VM. It handles ~5 RPS fine, but when you run evals or a marketing campaign lands, traffic spikes to 200–300 RPS and latency jumps from 200 ms to several seconds. You’ve tried vertical scaling and a bit of Kubernetes, but managing GPU node pools, prewarming, and YAML isn’t where you want to spend your week.

With Modal, you:

1. Move your model load into an @app.cls with @modal.enter so each container loads weights once.
2. Wrap your existing FastAPI app in @modal.asgi_app() and choose gpu="A100" plus a sane concurrency_limit (say 8).
3. Deploy with modal deploy and point your clients at the new HTTPS endpoint.

Under normal traffic, Modal keeps 1–2 warm replicas alive. When evals kick off and your MCP servers or agents flood the endpoint, Modal scales out replicas into the multi‑cloud GPU pool—up to hundreds of containers if needed. Because containers are gVisor-sandboxed, you get isolation. Because autoscaling is baked into the runtime, you don’t touch node counts or AutoScalingGroups. Logs and metrics land in the Modal apps page, so you can watch concurrency and latency distributions in real time.

The net result: same FastAPI app, same route signatures, but now you can run “massive spikes in volume for evals, RL environments, and MCP servers” without pre-provisioned GPU fleets or manual capacity management.

Pro Tip: Start by profiling a single replica’s throughput: find the max safe concurrency_limit where your p95 latency stays within budget. Then let Modal scale out replicas horizontally instead of over‑stuffing each GPU. This usually beats exotic single-node optimization for overall throughput and keeps tail latency predictable.

Summary

You don’t need to rewrite your FastAPI inference service to scale it to hundreds or thousands of workers. You need a runtime that treats FastAPI as an ASGI app, not a special snowflake, and lets you define environment, hardware, scaling, and lifecycle in code.

On Modal, that means:

• wrapping your existing FastAPI app with @modal.fastapi_endpoint or @modal.asgi_app,
• using @app.cls + @modal.enter to load models once per container,
• defining GPU/CPU hardware and autoscaling behavior in Python decorators,
• and relying on Modal’s multi‑cloud capacity pool to scale replicas from zero to thousands as load spikes.

You keep your FastAPI code and client contracts; you swap out the infrastructure for something built for sub‑second cold starts, elastic GPU scaling, and production-grade observability.

Next Step

Get Started