What’s a good pattern for scheduled GPU batch jobs (nightly/weekly) that scale to zero when done?

Most production AI workloads have a boring but important sibling: the nightly or weekly GPU batch job. Think eval sweeps, retraining, embeddings refresh, or log crunching over terabytes of data. You want them to start on schedule, fan out across GPUs, finish, and then disappear—no warm pools, no idle clusters, no quota babysitting. And you want all of this defined in code, in the same repo as the job itself.

Quick Answer: The best pattern is to define your GPU job as a Modal Function, wire it to a modal.Cron schedule, and let Modal spin up GPUs on demand, fan out the workload with .map() or .spawn(), then scale back to zero when the run finishes. All environment, hardware, and scheduling live in Python, so you ship one code-defined batch app instead of a pile of YAML and cron glue.

Why This Matters

Nightly and weekly GPU jobs are where infrastructure entropy tends to accumulate. A “simple” cron on a GPU node becomes a dedicated cluster you’re afraid to touch, or some semi-manual Airflow pipeline that nobody fully understands. The result: idle GPUs, fragile schedules, and a lot of operational drag just to run code a few times per week.

A clean pattern for scheduled GPU batch jobs gives you:

• Code-defined schedules that version with your application
• Elastic GPU capacity that appears when needed and costs $0 when idle
• A single runtime model (functions, images, logs) for both online inference and offline batch

Key Benefits:

• Scale to zero by default: Modal spins GPUs up when the schedule triggers and tears everything down when the run completes—no long‑lived clusters to manage.
• Code, not cron glue: You define your environment, hardware, schedule, and fan‑out logic in Python using Modal decorators, not in separate YAML or infra tooling.
• Built‑in observability and retries: Every run is a first‑class app in the Modal dashboard, with logs, metrics, and production primitives like modal.Retries and timeouts.

Core Concepts & Key Points

Concept	Definition	Why it's important
Code‑defined schedules	Using modal.Cron or modal.Period in Python to declare when a function should run (e.g., nightly at 02:00 UTC).	Your schedule lives next to the job logic, can be code‑reviewed, and deploys with modal deploy like everything else.
GPU‑backed Functions	Modal Functions with a GPU configuration (e.g., gpu="A100:2") that run inside an Image with your dependencies baked in.	You get deterministic environments and explicit hardware selection without manual cluster management.
Fan‑out with .map() / .spawn()	Running the same Function across many inputs in parallel, with Modal autoscaling containers to match demand.	Lets a single scheduled trigger orchestrate thousands of parallel GPU tasks and then scale back to zero when done.

How It Works (Step‑by‑Step)

At a high level, the pattern looks like this:

1. Define a GPU‑enabled Modal Image and Function for your batch workload.
2. Define a scheduled “driver” Function using modal.Cron that orchestrates fan‑out and aggregation.
3. Deploy the app once; Modal handles the nightly/weekly triggers, autoscaling, and scale‑to‑zero.

Let’s walk through an example: a nightly evaluation sweep of a model across a large dataset on GPUs.

1. Define your Image and GPU worker Function

First, get the environment and hardware story straight. You want everything pinned and reproducible.

# app.py
import modal

app = modal.App("nightly-gpu-batch")

image = (
    modal.Image.debian_slim()
    .pip_install(
        "torch==2.2.0",
        "transformers==4.39.3",
        "datasets==2.18.0",
    )
)

GPU_TYPE = "A10G"  # or "A100", "H100", etc.
TIMEOUT = 60 * 60  # 1 hour per task max (Modal max is 24h per Function call)

@app.function(
    image=image,
    gpu=GPU_TYPE,
    timeout=TIMEOUT,
    retries=modal.Retries(max_retries=3),
)
def run_eval_shard(shard_id: int, num_shards: int) -> dict:
    """Run evaluation on a shard of a dataset and return metrics."""
    from datasets import load_dataset
    from transformers import AutoModelForSequenceClassification, AutoTokenizer
    import torch

    model_name = "distilbert-base-uncased-finetuned-sst-2-english"
    device = "cuda" if torch.cuda.is_available() else "cpu"

    model = AutoModelForSequenceClassification.from_pretrained(model_name).to(device)
    tokenizer = AutoTokenizer.from_pretrained(model_name)

    ds = load_dataset("imdb", split="test")

    shard = ds.shard(num_shards=num_shards, index=shard_id)
    correct = 0

    for item in shard:
        inputs = tokenizer(
            item["text"],
            truncation=True,
            padding="max_length",
            max_length=256,
            return_tensors="pt",
        ).to(device)
        with torch.no_grad():
            logits = model(**inputs).logits
        pred = logits.argmax(dim=-1).item()
        if pred == item["label"]:
            correct += 1

    accuracy = correct / len(shard)
    return {"shard_id": shard_id, "n": len(shard), "accuracy": accuracy}

This is your basic GPU worker: it runs on a single GPU, processes a slice of data, and returns metrics. No scheduling logic yet.

2. Add a scheduled driver Function with modal.Cron

Now you need something that runs nightly/weekly, fans out the shards, waits for results, and maybe writes a report.

# app.py continued

NUM_SHARDS = 64  # how many parallel eval tasks to run

@app.function(
    schedule=modal.Cron("0 2 * * *"),  # every day at 02:00 UTC
    # or for weekly: modal.Cron("0 3 * * 1")  # Mondays at 03:00 UTC
)
def nightly_eval_driver():
    """Scheduled entrypoint. Fan out shards to GPUs, aggregate metrics."""
    # Fan out the workload across GPUs
    calls = [run_eval_shard.spawn(i, NUM_SHARDS) for i in range(NUM_SHARDS)]

    # Collect results
    results = [call.get() for call in calls]

    # Aggregate
    total_n = sum(r["n"] for r in results)
    weighted_acc = sum(r["accuracy"] * r["n"] for r in results) / total_n

    # You can push this somewhere persistent: S3, DB, Slack, etc.
    print(f"[nightly_eval] total_n={total_n}, weighted_accuracy={weighted_acc:.4f}")

Key points:

• schedule=modal.Cron("0 2 * * *") wires this Function to a cron‑style schedule.
• The Function itself has no GPU attached; it’s just orchestration. The GPU work is in run_eval_shard.
• .spawn() creates Function Calls that Modal runs across as many containers as needed, on demand. Once all the calls finish and the driver exits, Modal scales those containers back to zero.

3. Deploy once, let Modal handle the rest

To ship this:

modal deploy app.py

After deploy:

• Modal registers the nightly_eval_driver schedule.
• At 02:00 UTC every day (or weekly, depending on your cron), Modal starts the driver.
• The driver fans out 64 eval jobs; Modal allocates GPUs elastically from its multi‑cloud capacity pool.
• When all tasks complete, all containers are torn down. You’re back to zero running resources and zero idle cost.

You can watch runs and logs in the Modal dashboard under the app name (nightly-gpu-batch).

Common Mistakes to Avoid

• Keeping a “warm” GPU cluster alive:
  Many teams spin up a “batch” GPU cluster and then leave it running to make sure cron jobs have capacity. On Modal, this defeats the point. Instead, rely on autoscaling and scale‑to‑zero: declare GPUs per Function (gpu="A100") and let the scheduler acquire GPUs only when a run is actually in flight.

• Stuffing everything into the scheduled Function:
  It’s tempting to put all your compute, fan‑out, and I/O into the scheduled Function itself. This makes it harder to reuse components and to scale parts independently. Keep the scheduled Function thin—use it as a driver that calls GPU Functions via .spawn() or .map() and keeps orchestration separate from compute.

Other pitfalls worth avoiding:

• Forgetting timeouts or retries on long GPU work—explicitly set timeout and retries on your GPU Functions.
• Overloading a single Function call with hours of work—prefer many smaller calls that can be retried individually.

Real‑World Example

Imagine you retrain a recommendation model once per week on fresh interaction logs, and you want to run a fairly heavy embedding + training pipeline on GPUs:

• Step 1: Precompute embeddings. A scheduled driver Function, running CPU‑only, enumerates input buckets from object storage and uses embed_shard.spawn() to send work out to GPU Functions (gpu="A10G"). Each GPU Function writes embeddings to a Modal Volume or to your object store.
• Step 2: Kick off training. When embeddings are ready, the same driver (or a second scheduled Function with a later cron) calls train_model.remote() configured with a larger GPU like "A100:2". The training Function periodically checkpoints to a Volume per Modal’s recommended pattern—save often so you can resume if preempted or interrupted.
• Step 3: Evaluate and publish. Another scheduled Function runs eval shards in parallel, aggregates metrics, and if they clear a threshold, writes a “ready” marker or triggers a separate deployment pipeline.

In all three phases:

• Jobs only run when the schedule fires.
• GPUs are acquired transiently and released automatically.
• You have logs and metrics for each run in the dashboard, and runs are independent—no long‑running cluster state to babysit.

Pro Tip: For long‑running training steps in a scheduled pipeline, implement checkpointing to a Modal Volume and make the training Function reentrant. That way, if the job hits a timeout or is preempted, a new scheduled run (or manual retry) can resume from the latest checkpoint instead of starting from scratch.

Summary

The clean pattern for nightly/weekly GPU batch work is to treat “schedule” as just another piece of Python code, not a separate infrastructure layer. Define a GPU worker Function with explicit image, hardware, timeouts, and retries. Wrap it in a thin scheduled driver using modal.Cron that fans out work with .spawn() or .map(), aggregates results, and exits. Deploy once with modal deploy, and let Modal handle GPU provisioning, autoscaling, and scale‑to‑zero after each run.

You get deterministic GPU jobs that run on schedule, absorb big batch spikes, and then vanish until next time—no idle clusters, no glued‑together cron boxes.

Next Step

Get Started