Why is the first request to my GPU inference endpoint so slow, and how do people get cold starts under a second?

Most teams hit the same wall the first time they deploy a GPU inference endpoint: the first request takes multiple seconds, and only after that does latency drop to something reasonable. That “why is my first token so slow?” moment is usually not about your network or framework; it’s about physics, GPU memory, and how you initialize your model server.

Quick Answer: The first request to your GPU endpoint is slow because you’re paying two big one-time costs: starting a container and loading several GB of model weights from disk into GPU memory and then into on-chip SRAM. To get cold starts under a second, you either (a) keep “warm” containers around with weights already loaded, or (b) use an AI-native runtime that hides container startup and weight loading behind aggressive caching, pre-initialization, and fast autoscaling.

Why This Matters

If your first request takes 10–20 seconds, you can’t safely use that endpoint in anything user-facing: chat UIs, integrations, agentic workflows, or evals that spike from 0 to thousands of requests. Cold starts turn into timeouts, dropped traffic, and defensive overprovisioning (“just leave 20 GPUs idling”) that kills your infra budget.

If you understand where the latency comes from, you can shape your deployment strategy instead of cargo-culting random “optimizations.” That means picking the right GPU, preloading weights the right way, and using infrastructure that actually respects how GPU workloads behave rather than treating them like generic HTTP servers.

Key Benefits:

• Predictable latency for the first user: Avoid 10–30s stalls on the first token by keeping weights hot and instances ready to serve.
• No-idle-cost autoscaling: Scale to zero between bursts without paying the full cold start penalty every time.
• Higher throughput per dollar: Spend GPU time on tokens, not on repeatedly loading the same 10–40GB of weights.

Core Concepts & Key Points

Concept	Definition	Why it's important
Container cold start	Time to start your inference container, import libraries, and initialize your framework before handling the first request.	Adds hundreds of milliseconds to seconds before your app code even runs, especially on generic serverless platforms.
Model weight loading	The process of reading multi-GB model weights from storage, copying to GPU VRAM, and then into on-chip SRAM when generating tokens.	Dominant source of first-request latency; memory bandwidth-bound and fundamentally non-trivial.
Warm / pre-initialized containers	Containers kept alive with model weights already on the GPU, ready to serve traffic without reloading.	The main technique behind “sub-second cold starts” for LLMs and other large models.

What Actually Makes That First Request So Slow?

Let’s break the first request into its components. In a naïve setup, you pay all of these, serially:

1. Container startup & runtime boot

   • Time for your platform to:
     • Provision a node (or wake one up).
     • Start a container.
     • Attach a GPU.
     • Boot your Python runtime.
   • On typical serverless/container platforms, this can be 1–10 seconds, depending on image size, base image, and node provisioning.

2. Environment & framework initialization

   • Importing torch, transformers, vllm, CUDA libraries, etc.
   • JIT compilation, kernel loading, cuDNN/cuBLAS initialization.
   • Cost: 100ms–several seconds, depending on how much work happens in import and initialization.

3. Model weight loading from storage to GPU VRAM

   • Reading multiple gigabytes of weights from:
     • Local disk, network filesystem, or object storage (e.g., S3).
   • Copying them into GPU VRAM.
   • This is fundamentally bounded by I/O (storage + PCIe/NVLink) bandwidth:
     • 10–40GB of weights at effective ~5–20GB/s can easily be 0.5–5 seconds.

4. Token generation: VRAM → SRAM in Streaming Multiprocessors

   • Even after weights are in VRAM, per-token latency is dominated by moving those GB of weights from GPU RAM into on-chip SRAM inside the Streaming Multiprocessors.
   • This has to happen at least once for the request — naïvely, once per token per request.
   • Modern data center GPUs have memory bandwidth on the order of terabytes per second, but with GB-scale models, that still translates into:
     • Milliseconds per token.
     • Seconds for the kilotoken responses users like.

5. Network & service overhead

   • If you measure from the client, you’re also including:
     • TLS handshakes, routing, load balancer.
     • Client–server latency.
   • In practice, ~150ms of network and service delays is common baked into “time to first token” (TTFT) numbers.

This is why bare “cold TTFT” for LLMs often lands in the 200ms+ range even in highly-optimized setups and explodes into multi-second delays in naïve deployments.

How People Get “Cold Starts Under a Second”

When people claim “sub-second cold starts” for GPU inference, they’re almost never doing a full “from image pull to first token” cycle. They’re doing one or more of:

• Avoiding container cold starts entirely for the hot path.
• Preloading and caching weights in GPU VRAM.
• Using infra that keeps a pool of pre-initialized containers ready.
• Measuring from server to server, not from end-user client.

On Modal, for example, the architecture is explicitly built around this:

• Container cold starts are aggressively minimized.

  • Modal’s runtime spins up containers in seconds, with an internal architecture tuned for high churn.
  • “100x faster than Docker” is not magic; it’s the result of building only for containerized workloads and throwing away the general-purpose baggage.

• Model servers run as stateful classes (@app.cls) and load weights once per container.

  • You define a class whose __enter__ (@modal.enter) method loads the weights into GPU memory.
  • That cost is paid once per container, not per request.

• Autoscaling keeps a warm pool.

  • You set concurrency and scaling limits in code.
  • Modal keeps enough containers hot so new requests can usually land on already-initialized instances.

Combine these, and you can get “cold requests” (from client perspective) under a second, even with multi-GB models, as long as there’s at least one warm container around.

How It Works (Step-by-Step)

Let’s walk through the typical lifecycle and then how to optimize it.

1. Define a GPU inference function in Python

Start with a normal Python function. Choose a GPU (e.g., H100, A100, A10G) with enough VRAM for your model.

import modal

app = modal.App("fast-llm-inference")

image = (
    modal.Image.debian_slim()
    .pip_install(
        "torch==2.2.0",
        "transformers==4.38.0",
        "accelerate==0.27.0",
    )
)

@app.cls(
    gpu="H100",
    image=image,
)
class LLMServer:
    def __enter__(self):
        from transformers import AutoModelForCausalLM, AutoTokenizer

        model_name = "meta-llama/Llama-2-7b-chat-hf"
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype="auto",
            device_map="cuda",
        )
        # Weights are now resident in GPU VRAM

    @modal.method()
    def generate(self, prompt: str) -> str:
        import torch

        inputs = self.tokenizer(prompt, return_tensors="pt").to("cuda")
        outputs = self.model.generate(
            **inputs,
            max_new_tokens=128,
        )
        return self.tokenizer.decode(outputs[0], skip_special_tokens=True)

Here’s what matters:

• @app.cls(gpu="H100", image=image) says: give this class a GPU-backed container with our dependencies.
• __enter__ loads the model once. Every method call runs with the model already in VRAM.
• generate is your “handle a request” function.

2. Expose it as a web endpoint

Turn it into an HTTP endpoint with a couple more lines:

from fastapi import FastAPI
from pydantic import BaseModel

fastapi_app = FastAPI()
server = LLMServer()

class GenerateRequest(BaseModel):
    prompt: str

@modal.fastapi_endpoint(app=app)
@fastapi_app.post("/generate")
async def generate_endpoint(req: GenerateRequest):
    return {"output": await server.generate.remote(req.prompt)}

Deploy it:

modal deploy llm_app.py

Modal now:

• Builds the image (once).
• Spins up GPU containers.
• Calls __enter__ to load the model into VRAM before serving traffic.
• Routes HTTP requests to generate_endpoint, which calls into server.generate.remote(...).

3. Configure autoscaling and keep containers warm

You usually want at least one warm container:

@app.cls(
    gpu="H100",
    image=image,
    concurrency_limit=8,
    keep_warm=1,   # keep at least one container hot
)
class LLMServer:
    ...

Now, even when traffic drops to near-zero:

• Modal keeps one GPU container alive with the model loaded.
• A “first” request after a lull hits a warm instance, so TTFT is bounded by:
  • Prompt tokenization.
  • Initial kernel launches.
  • Network overhead (often ~150ms).

In practice, this is how you get cold starts under a second for real users—without paying for 10 idle GPUs 24/7.

Common Mistakes to Avoid

• Loading the model on every request:

  • Pattern: model = AutoModelForCausalLM.from_pretrained(...) inside your request handler.
  • Result: You reload GBs of weights for every call.
  • Fix: Load once per container with a class-based server (@app.cls + @modal.enter) or process-level init.

• Relying on generic FaaS with no GPU-awareness:

  • Pattern: Lambda/Cloud Run spinning up containers with big images and no persistent GPU state.
  • Result: 5–30s cold starts on every scale-from-zero, plus no control over warm pools.
  • Fix: Use an AI-native platform where you can set GPU, warm counts, and lifecycle hooks in code.

• Pulling weights from slow or cross-region storage:

  • Pattern: Loading from S3 or HF Hub in a different region each time a container boots.
  • Result: Weight load times balloon due to cross-region bandwidth and latency.
  • Fix: Cache weights in a local volume or region-local storage; avoid cross-region endpoints.

• Overspecifying GPUs and underspecifying concurrency:

  • Pattern: “Just use A100:4 and hope it’s fast.”
  • Result: High cost with no guaranteed latency benefits; poor utilization.
  • Fix: Start with a single H100 or A100, benchmark, and tune concurrency (concurrency_limit) and batch sizes.

Real-World Example

Imagine you’re running evals for an LLM agent. You might run zero traffic for an hour, then spike to 10,000 requests in a few minutes. On a naïve GPU endpoint:

• First eval job:
  • Waits 15–30 seconds while:
    • The node spins up.
    • The container boots.
    • PyTorch and the model initialize.
  • You blow your latency budget on the very first call.

To avoid this, you move to a Modal app:

• You define LLMServer as above with keep_warm=2 and concurrency_limit=16.
• You run your evals by calling server.generate.spawn(prompt) thousands of times; Modal spreads them across warm containers.
• During quiet periods, you still keep 2 containers alive, eating some idle cost but avoiding 10–20s cold starts.
• When the spike hits:
  • Requests go into queues attached to already-initialized containers.
  • Modal autoscaler increases container count; each new container pays the initial weight load once, then goes into the warm pool.

From the client perspective, the first request of the spike lands on one of the existing warm containers and returns in under a second. The fact that new containers are spinning up in the background is invisible to your users.

Pro Tip: Treat “warm capacity” as an SLO knob, not an accident. For anything user-facing, explicitly set keep_warm and concurrency_limit in code, then inspect actual latencies in your Modal apps dashboard and adjust. It’s usually cheaper to keep 1–2 GPUs warm than to chase 30-second cold starts with retries and timeouts.

Summary

The first request to your GPU inference endpoint is slow because you’re doing all the heavy lifting at once: starting a container, initializing the runtime, and loading multi-gigabyte weights from storage into GPU VRAM and then into on-chip SRAM. This workload is fundamentally memory-bandwidth-bound, so naïve deployments pay seconds of latency even before tokens start flowing.

To get cold starts under a second in practice, you don’t cheat physics—you change the shape of the problem:

• Load weights once per container, not per request.
• Keep a small pool of warm containers with models already on the GPU.
• Use an AI-native runtime that makes container startup and autoscaling fast, predictable, and programmable from Python.

Do that, and “first request” stops being a problem; it becomes just another request landing on a warm model server.

Next Step

Get Started