pay-per-second GPU hosting that scales to zero (A100/H100/T4) — who offers this?

Most teams looking for pay-per-second GPU hosting that scales to zero hit the same wall: big cloud GPUs are priced by the hour, require quotas and reservations, and don’t naturally shrink back to zero when your traffic is quiet or your experiments finish. What you actually want is serverless-style GPUs (A100, H100, T4) that you pay for only when your code is running—and that can also burst hard when you need thousands of parallel containers.

Quick Answer: Yes, there are pay-per-second GPU platforms that scale to zero on A100/H100/T4-class hardware—Modal is one of them, built specifically for AI workloads with instant autoscaling and sub-second cold starts. Instead of renting fixed GPU instances, you define your environment in Python; Modal runs your functions on elastic GPUs, scales them out or down to zero automatically, and bills by the second.

Why This Matters

Hourly, always-on GPU instances are fine when you’re training one big model for days. They’re terrible when you’re:

• Running LLM inference that’s spiky across the day
• Doing evals or RL with bursts to thousands of parallel rollouts
• Hosting internal tools or agents with unpredictable usage
• Running secure sandboxes that only live for seconds

If your infrastructure can’t scale to zero and bill per second, you end up overprovisioning or constantly babysitting autoscaling groups. That’s wasted money and wasted engineering time.

Key Benefits:

• True pay-per-use: You’re billed per second of GPU runtime instead of pre-paying for idle instances.
• Scale to zero by default: When no traffic is hitting your endpoints or jobs, GPU usage (and cost) drops to zero.
• Burst to thousands of GPUs: When you need to fan out evaluations or batch jobs, you can spin up massive parallelism without tickets, reservations, or capacity planning.

Core Concepts & Key Points

Concept	Definition	Why it's important
Pay-per-second GPU hosting	Billing based on actual runtime seconds of GPU containers, not hourly instances	Avoids paying for idle capacity; matches cost to workload shape
Scale to zero	Infrastructure automatically deprovisions when there are no active requests or jobs	Cuts your baseline GPU cost to effectively zero between bursts
Python-defined serverless GPUs	Defining environment, hardware, scaling, and endpoints in Python code (e.g., Modal Images + Functions)	You get GPU infra with simple decorators and scripts instead of clusters, YAML, and manual orchestration

How It Works (Step-by-Step)

At a high level, pay-per-second GPU hosting that scales to zero needs three things:

1. Ephemeral containers instead of fixed instances:
   You don’t own a GPU node; you submit work that gets attached to a GPU for as long as it runs, then the container disappears.

2. Autoscaling based on demand, not reservations:
   The platform allocates GPUs as your functions are invoked, and tears them down when idle—no warm pools or manual scale-down logic you have to maintain.

3. Code-first configuration of hardware & environment:
   You describe what you need (e.g., “A100:2 with CUDA + PyTorch + this pip set”) in code; the platform handles drivers, images, and scheduling.

On Modal, that looks like this in practice.

1. Define a GPU Image in Python

Let’s get the imports out of the way and build a Modal Image that pins your dependencies:

import modal

image = (
    modal.Image.debian_slim()
    .pip_install(
        "torch==2.3.1",
        "transformers==4.40.0",
        "accelerate==0.30.0",
    )
)

app = modal.App("gpu-pay-per-second-demo")

This Image is your reproducible environment: OS, libraries, and ML stack baked once, reused across thousands of containers. No Dockerfiles, no manual pushes.

2. Attach a GPU and Scale Policy to a Function

Now define a function that runs on a GPU, with compute and concurrency expressed in Python:

@app.function(
    image=image,
    gpu="a100",           # or "h100", "t4", "a10g", "a100:2"
    timeout=600,          # max 10 minutes per call
    concurrency_limit=100 # safety limit per function
)
def run_inference(batch):
    import torch
    from transformers import AutoModelForCausalLM, AutoTokenizer

    model_name = "meta-llama/Llama-3-8B-Instruct"

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.float16,
        device_map="cuda",
    )

    inputs = tokenizer(batch, return_tensors="pt", padding=True).to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=128)
    return tokenizer.batch_decode(outputs, skip_special_tokens=True)

A few important details:

• gpu="a100" (or "h100", "t4", "a10g") tells Modal what hardware you want.
• The function becomes an autoscaled, GPU-backed unit of work:
  • You can run it ad-hoc via .remote(...)
  • Fan it out with .map(...)
  • Queue work with .spawn() and FunctionCall.get()
• When there are no calls, there are no GPU containers running—your usage scales to zero.

3. Expose a Web Endpoint (Optional)

To serve requests from your app or other services, expose a FastAPI endpoint:

from fastapi import FastAPI
from pydantic import BaseModel

web_app = FastAPI()

class InferenceRequest(BaseModel):
    prompts: list[str]

@app.function(
    image=image,
    gpu="t4",               # cheaper GPU for lighter workloads
    timeout=60,
)
@modal.fastapi_endpoint(web_app, method="post")
async def infer_endpoint(body: InferenceRequest):
    return run_inference.remote(body.prompts)

Deploy it:

modal deploy gpu_pay_per_second_app.py

What you get:

• A live HTTPS endpoint that can autoscale across many T4s/A100s/H100s.
• Containers spin up when real traffic arrives (with sub-second cold starts for prebuilt Images).
• When traffic stops, infrastructure shrinks back to zero. You’re only billed for actual runtime.

Common Mistakes to Avoid

• Treating GPUs like static VMs instead of ephemeral compute:
  If you bring a “long-lived instance” mindset, you’ll try to keep containers hot forever and lose the cost advantage. Instead, load heavy assets once per container with @app.cls + @modal.enter, and let autoscaling do its job.

• Not pinning dependencies tightly in Images:
  Leaving torch or transformers unpinned means “works today, breaks quietly tomorrow.” Always pin versions in your Modal Image to get reproducible behavior and predictable GPU performance.

Real-World Example

Say you’re running evals on a new agent stack: for each model version you run 50k prompts with long context windows. Traffic pattern:

• Massive burst of jobs for 10–20 minutes
• Then nothing for hours while you analyze results
• Repeat whenever you tweak the agent or prompt strategy

On instance-based GPUs, you either:

• Overprovision 10–20 A100s for hours “just in case,” or
• Underprovision and watch jobs queue endlessly

On Modal, you implement your eval as a GPU function:

@app.function(image=image, gpu="a100", timeout=900)
def run_eval_case(case):
    # your eval logic here
    ...

@app.local_entrypoint()
def main():
    cases = load_eval_cases()
    results = list(run_eval_case.map(cases))
    save_results(results)

When you run:

modal run eval_app.py

Modal fans run_eval_case across many A100s in parallel:

• Jobs complete in minutes, not hours.
• Once the map job is done, all those GPU containers are gone.
• Your GPU cost for the eval is minutes of A100 time, not hours of idle instances.

Pro Tip: For heavy model loading, wrap your model server in @app.cls with @modal.enter to load weights once per container, then reuse that state across many calls. You’ll hit the sweet spot of low latency and high throughput on A100/H100 without wasting GPU time on repeated initialization.

Summary

If you’re looking for pay-per-second GPU hosting that scales to zero on A100/H100/T4, you’re really asking for a serverless GPU runtime with:

• Ephemeral containers scheduled onto a global GPU pool
• Autoscaling up for bursts, down to zero when idle
• Pricing based on actual runtime seconds, not hourly instances

Modal is designed specifically around that model: you define everything in Python—Images, GPUs, endpoints, batch jobs—and Modal’s AI-native runtime handles sub-second cold starts, instant autoscaling, and multi-cloud GPU capacity. You get the performance characteristics of dedicated GPUs with the economics and ergonomics of serverless.

Next Step

Get Started