best way to run 10k parallel Python container jobs for ML evals (no Kubernetes)

Quick Answer: The best way to run 10k parallel Python container jobs for ML evals without Kubernetes is to define your workload as plain Python functions on Modal, then fan them out with .map() or .spawn() over elastic CPU/GPU containers. You let Modal handle autoscaling, scheduling, and retries while you stay in Python, not YAML or cluster ops.

Why This Matters

ML evaluation is inherently bursty: you might sit idle for hours, then suddenly need to hammer 10,000 prompts through an LLM or run a big sweep of reward models. If you solve this with Kubernetes, you’re signing up for cluster sizing, node pools, autoscalers, and a lot of YAML you’ll forget how to maintain. If you solve it with something like plain Docker + EC2, you end up either massively overprovisioned or rate-limited by manual orchestration.

A better pattern is: treat the infra as a programmable runtime you control from Python. You should be able to say “run this function 10k times in parallel” and have the platform spin up containers, distribute work, retry failures, and shut everything down when you’re done—without you touching a single manifest.

Key Benefits:

• True elastic parallelism: Fan out to thousands of containers (CPU or GPU) with .map() / .spawn() and let Modal autoscale up, then back to zero.
• Code-defined infra, no Kubernetes: Declare Images, hardware, concurrency, and timeouts in Python decorators instead of maintaining clusters and YAML.
• Production-ready controls: Built‑in retries, timeouts, isolation via gVisor, and integrated logs let you treat ML evals like real production jobs, not one‑off scripts.

Core Concepts & Key Points

Concept	Definition	Why it's important
Python-defined infrastructure	Describing containers, hardware, scaling, and endpoints in Python using Modal Images and decorators.	Keeps infra close to code, avoids Kubernetes and YAML sprawl, and makes changes easy to review and test.
Massive fan-out (.map() / .spawn())	Modal primitives for running the same function over thousands of inputs in parallel on elastic containers.	This is the core building block for “run 10k parallel Python container jobs” without managing workers or queues yourself.
Autoscaling + container lifecycle	Modal’s runtime that boots containers in seconds, reuses them when possible, and scales to thousands of concurrent workers.	Turns spiky ML eval workloads into simple function calls that complete within your latency/throughput budget without overprovisioning.

How It Works (Step-by-Step)

Let’s walk through how you’d actually run 10k parallel Python container jobs for ML evals on Modal, with no Kubernetes in sight.

1. Define your environment as an Image

First, you describe the container environment in Python. Pin dependencies tightly; that’s how you avoid “works on my laptop” surprises.

import modal

app = modal.App("ml-evals-10k")

image = (
    modal.Image.debian_slim()
    .pip_install(
        "torch==2.2.1",
        "transformers==4.39.3",
        "accelerate==0.28.0",
        "numpy==1.26.4",
    )
)

This replaces a Dockerfile. Modal will build and cache this image, and cold starts are designed to be fast—containers typically launch in seconds.

2. Declare the worker function that runs in containers

Now define the actual evaluation job—this is the unit of work that will run 10,000 times.

from typing import Dict, Any

@app.function(
    image=image,
    timeout=600,              # max 10 minutes per eval
    concurrency_limit=100,    # avoid accidental blowups
)
def run_single_eval(example: Dict[str, Any]) -> Dict[str, Any]:
    from transformers import AutoModelForCausalLM, AutoTokenizer
    import torch

    # For real workloads, load model in a class-level server (see below).
    model_name = "meta-llama/Meta-Llama-3-8B-Instruct"
    device = "cuda" if torch.cuda.is_available() else "cpu"

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.float16 if device == "cuda" else torch.float32,
        device_map="auto" if device == "cuda" else None,
    )

    prompt = example["prompt"]
    inputs = tokenizer(prompt, return_tensors="pt").to(device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=256,
            temperature=0.2,
            do_sample=False,
        )

    completion = tokenizer.decode(outputs[0], skip_special_tokens=True)

    return {
        "id": example["id"],
        "prompt": prompt,
        "completion": completion,
    }

You can run this locally with:

modal run ml_evals.py::run_single_eval

No cluster. No YAML. Just Python.

3. Wrap the model in a stateful server for better throughput (optional but recommended)

If you call run_single_eval 10,000 times as-is, you’ll reload the model 10,000 times. That’s wasteful. Instead, use a class-based server with @app.cls and @modal.enter so each container loads the model once, then handles many requests.

@app.cls(
    image=image,
    gpu="A10G",            # or "A100:2", "H100", etc. for heavy evals
    concurrency_limit=10,  # per-container concurrent calls
    timeout=600,
)
class LlamaEvalServer:
    def __init__(self):
        self.tokenizer = None
        self.model = None
        self.device = "cuda"

    @modal.enter()
    def load_model(self):
        from transformers import AutoModelForCausalLM, AutoTokenizer
        import torch

        model_name = "meta-llama/Meta-Llama-3-8B-Instruct"

        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype=torch.float16,
            device_map="auto",
        )
        self.model.eval()

    @modal.method()
    def eval_one(self, example: Dict[str, Any]) -> Dict[str, Any]:
        import torch

        inputs = self.tokenizer(
            example["prompt"], return_tensors="pt"
        ).to(self.device)

        with torch.no_grad():
            outputs = self.model.generate(
                **inputs,
                max_new_tokens=256,
                temperature=0.2,
                do_sample=False,
            )

        completion = self.tokenizer.decode(outputs[0], skip_special_tokens=True)

        return {
            "id": example["id"],
            "prompt": example["prompt"],
            "completion": completion,
        }

This is the pattern you want for 10k parallel jobs: each GPU container becomes a multi-request worker.

4. Fan out 10k eval jobs with .map() or .spawn()

Now we orchestrate the 10,000 jobs from a single Python function. You choose between .map() for simple bulk parallelism or .spawn() if you want explicit lifecycle control over each call.

Using .map() for bulk evals

@app.function(
    image=image,
    timeout=24 * 60 * 60,  # max 24 hours for the whole eval run
)
def run_10k_evals(examples: list[Dict[str, Any]]) -> list[Dict[str, Any]]:
    # Create a handle to the class-based server
    server = LlamaEvalServer()

    # Modal will create as many containers as needed and
    # distribute calls to `eval_one` across them.
    results = list(server.eval_one.map(examples))
    return results

You can run this with:

modal run ml_evals.py::run_10k_evals

Modal will:

• Spin up a pool of GPU containers (for LlamaEvalServer).
• Call eval_one over your 10,000 examples in parallel.
• Reuse containers while jobs are in flight.
• Tear everything down when done.

Using .spawn() for more control (queues, retries, partial results)

If you want to stream results, handle failures manually, or inspect jobs as they run:

@app.function(image=image, timeout=24 * 60 * 60)
def run_10k_evals_spawn(examples: list[Dict[str, Any]]) -> list[Dict[str, Any]]:
    server = LlamaEvalServer()

    calls = [
        server.eval_one.spawn(example)
        for example in examples
    ]

    # Get results (with basic retry semantics handled by Modal)
    results = [call.get() for call in calls]
    return results

.spawn() gives you a FunctionCall handle you can poll, cancel, or inspect later. This is handy when:

• You want to avoid waiting on the full set.
• You need to track progress in a dashboard.
• You’re comfortable with partial failures and want to requeue specific items.

5. Run detached and let the platform handle the long tail

For long evaluation runs, start them detached so they keep going even if your laptop sleeps:

modal run --detach ml_evals.py::run_10k_evals

A few operational notes:

• Detached runs continue even if you close your terminal.
• You can inspect logs in the Modal apps page: every container, every function call.
• .remote() calls (and the FunctionCall objects they create) are limited to 24 hours; design your evaluation batches accordingly.

6. Tune concurrency and container sizing

To really squeeze performance out of 10k parallel jobs, you need decent defaults:

• Use concurrency_limit on your @app.cls and @app.function to avoid overloading a single container.
• Batch requests: instead of doing 1 example per call, consider running 4–16 examples per eval_one call if your model’s throughput is good enough. That reduces routing overhead and model overhead.
• Use appropriate GPUs: A10G is a good midrange choice; A100/H100 for larger models or multi-GPU eval servers (gpu="A100:2").

You can hardcode these in your decorators and adjust over time:

@app.cls(
    image=image,
    gpu="A10G",
    concurrency_limit=16,
    timeout=900,
)
class LlamaEvalServer:
    ...

Common Mistakes to Avoid

• Reloading the model on every call: This kills throughput. Use @app.cls + @modal.enter so you load the model once per container and then call .map() over @modal.methods.
• Trying to hand-roll a queue or worker pool: You don’t need to build your own job queue + worker pool on top of containers. Use .map() for straight fan-out, and .spawn() + FunctionCall.get() if you want explicit control and progress tracking.

Real-World Example

Imagine you want to evaluate a new reward model over 10,000 candidate completions from a baseline LLM. Each eval is independent but heavy enough to justify GPUs. With Modal, you:

1. Package the reward model and dependencies in an Image (Python code, not Dockerfiles).
2. Implement a RewardServer with @app.cls(gpu="A10G") and a score_batch @modal.method that processes a list of examples at once.
3. Run RewardServer.score_batch.map(batched_examples) over 10,000 items, using maybe 50–100 GPU containers in parallel.
4. Watch jobs in the Modal dashboard, inspect logs when something fails, and tweak timeouts or batch sizes directly in your Python code.

If the evaluation suddenly needs to run over 100,000 examples instead, you don’t change infra at all—you just feed more items into .map(). Modal’s autoscaler handles the extra parallelism, then scales back to zero when you’re done.

Pro Tip: For large eval sweeps, store your dataset and intermediate outputs in a Modal Volume or cloud storage, and only pass lightweight IDs through .map() / .spawn(). That keeps per-call payloads small and reduces serialization overhead while still letting you run 10k+ parallel jobs.

Summary

To run 10k parallel Python container jobs for ML evals without Kubernetes, structure your workload as:

• Python-defined Images that encode dependencies.
• Stateful eval servers via @app.cls that load models once per container.
• Massive fan-out over .map() or .spawn() for your eval methods.

You get elastic CPU/GPU scaling, sub-second container launches, and built-in retries and observability—without building a cluster, writing YAML, or hand-rolling job queues.

Next Step

Get Started