How do I fan out an eval run to thousands of parallel Python workers and aggregate results without building a cluster?

Most teams running serious evals eventually hit the same wall: you can generate 10k–100k prompts locally, but actually running them through a model in a reasonable time means fanning out to thousands of workers. The catch is you don’t want to build or operate a Kubernetes cluster just to parallelize a Python for-loop.

Quick Answer: Use Modal’s Python-first job fan-out pattern: define a single @app.function that runs one eval, then call it thousands of times with .spawn() or .map() to fan out across elastic containers, and aggregate results back in Python when the FunctionCall objects complete. You get thousands of parallel workers, automatic retries, and aggregation without touching cluster config, autoscalers, or queues.

Why This Matters

Eval runs are where all the sharp edges of your infrastructure show up at once: spiky, bursty workloads; expensive GPUs; and a bunch of Python code you’d like to keep simple so you can iterate quickly. If you’re spending days wiring up job queues, Kubernetes operators, and monitoring just to run a big eval, you’re burning time on infra instead of improving your models.

With Modal, you treat “thousands of workers” as a parameter on a function call, not a separate cluster project. You write the eval logic once, as a plain Python function, then let Modal fan it out across CPUs/GPUs on demand and pull the results back into a single process for scoring and analysis.

Key Benefits:

• Massive parallelism with just Python: Use .map() and .spawn() to fan out to thousands of workers without writing any cluster config or queue plumbing.
• Elastic capacity for spiky eval loads: Burst to thousands of CPUs/GPUs across clouds, then scale back to zero when the eval is done—no reservations or idle nodes.
• End-to-end observability and reliability: Automatic retries, timeouts, and per-call logs mean you can trust big eval runs and debug failures quickly.

Core Concepts & Key Points

Concept	Definition	Why it's important
Modal Function	A Python function decorated with @app.function that runs in a container on Modal’s AI-native runtime.	This is your “worker.” You write eval logic once and scale it from 1 to thousands of containers.
Fan-out via .map() / .spawn()	Call patterns on Modal Functions: .map() parallelizes over an iterable; .spawn() creates async jobs you can poll with FunctionCall.get().	They are your “job queue” and “cluster” in one—no extra infrastructure required.
Aggregation in Python	Collecting results in your driver process (or another Modal Function) once workers finish, then computing metrics.	Keeps the orchestration logic legible and testable—your aggregation is just a loop over Python objects.

How It Works (Step-by-Step)

At a high level, you have three pieces:

1. A driver that defines the eval dataset and orchestrates fan-out.
2. A worker function that runs a single eval example (or a small batch).
3. Aggregation code that gathers the results and computes metrics.

Let’s walk through a concrete setup.

1. Define the Modal app and environment

First, describe your environment and app in Python: Python version, dependencies, and hardware.

# eval_app.py
import modal

image = (
    modal.Image.debian_slim()
    .pip_install(
        "openai",        # or vLLM, transformers, etc.
        "tqdm",
        "pydantic",
    )
)

app = modal.App("llm-eval-fanout")

This replaces a bunch of Dockerfile + YAML. You can run and deploy everything with:

modal run eval_app.py
# later:
modal deploy eval_app.py

2. Implement a single eval worker

Write a plain Python function that runs one eval example (or a small batch of examples). Decorate it with @app.function so Modal can run it in containers.

from typing import Dict, Any

@app.function(
    image=image,
    timeout=600,          # per-example timeout (seconds)
    retries=modal.Retries(
        max_retries=3,
        backoff_coefficient=2.0,
    ),
    max_concurrency=1024,  # allow large parallelism
)
def run_eval_example(example: Dict[str, Any]) -> Dict[str, Any]:
    """
    Run model inference + scoring for a single eval example.
    Return structured metrics.
    """
    import openai
    import time

    prompt = example["prompt"]
    expected = example.get("expected")

    t0 = time.time()
    # Replace this with your model call: OpenAI, vLLM endpoint, local HF, etc.
    completion = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        timeout=30,
    )
    text = completion.choices[0].message["content"]
    latency = time.time() - t0

    # Simple example metric
    correct = int(expected is not None and expected in text)

    return {
        "id": example["id"],
        "latency": latency,
        "correct": correct,
        "output": text,
    }

This function will be executed in a sandboxed container (gVisor), with retries and timeouts handled by Modal. You can put any Python here: HF models, custom scoring, HTTP calls, etc.

3. Build the eval dataset

Now define how you generate or load the examples. Do this in a “driver” function.

def build_eval_examples(n: int):
    for i in range(n):
        yield {
            "id": i,
            "prompt": f"Translate this number to French: {i}",
            "expected": "trente" if i == 30 else None,  # toy example
        }

You can load from Parquet, a DB, or an S3 bucket. For large datasets, prefer streaming (a generator) rather than building a giant list.

4. Fan out: .map() vs .spawn()

You have two main options to fan out to thousands of workers:

Option A: Use .map() for simple, high-throughput fan-out

.map() is the simplest way to parallelize an eval run. Give it an iterable, and it will:

• Stream inputs to workers.
• Scale containers up to meet demand.
• Yield results as they complete.

@app.local_entrypoint()
def run_eval(n: int = 10000):
    examples = list(build_eval_examples(n))

    # Map returns an iterator of results in completion order.
    results_iter = run_eval_example.map(examples)

    num_correct = 0
    latencies = []
    outputs = []

    for result in results_iter:
        num_correct += result["correct"]
        latencies.append(result["latency"])
        outputs.append(result["output"])

    accuracy = num_correct / n
    p95_latency = sorted(latencies)[int(0.95 * len(latencies))]

    print(f"n={n}, accuracy={accuracy:.3f}, p95 latency={p95_latency:.3f}s")
    # You could also write results to a Volume or external store here.

This alone will fan out to thousands of containers if there’s enough work. You don’t configure a cluster; you just let Modal’s scheduler pull from the multi-cloud capacity pool.

Option B: Use .spawn() when you want async control and aggregation

If you want more explicit control (e.g., chunked fan-out, custom progress bars, or multiple aggregators), use .spawn().

@app.local_entrypoint()
def run_eval_spawn(n: int = 10000, batch_size: int = 1000):
    from tqdm import tqdm

    calls = []

    # Fan out in chunks if you want to control concurrency
    for batch_start in range(0, n, batch_size):
        batch = list(build_eval_examples(batch_size))
        for ex in batch:
            calls.append(run_eval_example.spawn(ex))

    # Now aggregate results
    num_correct = 0
    latencies = []

    for call in tqdm(calls, desc="Collecting results"):
        result = call.get()  # blocks until this particular call finishes
        num_correct += result["correct"]
        latencies.append(result["latency"])

    accuracy = num_correct / len(calls)
    p95_latency = sorted(latencies)[int(0.95 * len(latencies))]

    print(f"n={len(calls)}, accuracy={accuracy:.3f}, p95 latency={p95_latency:.3f}s")

Here, calls is essentially your job queue. Each spawn() returns a FunctionCall handle you can poll, wait on, or inspect in the Modal UI (per-call logs, retries, exceptions).

You can scale this pattern to:

• Multiple worker types (e.g. CPU scoring vs. GPU inference).
• Multi-phase evals (generation, then judging).
• Cross-model comparisons (spawn jobs for model A and B and merge in the aggregator).

5. Hardware and scaling knobs

Want GPUs? Change the decorator, not your infrastructure:

@app.function(
    image=image,
    gpu="A10G",          # or "A100", "A100:2", "H100", etc.
    timeout=600,
    max_concurrency=2048,
)
def run_eval_example(example: Dict[str, Any]) -> Dict[str, Any]:
    ...

Modal takes care of provisioning GPU containers, shuffling jobs to them, and scaling down when idle. You don’t touch node pools or cluster autoscalers.

Common Mistakes to Avoid

• Pushing massive payloads in and out of workers:
  Shipping large models or datasets with every call will kill performance and cost. Use Modal Images to bake in dependencies, and Volumes or remote storage for large assets. Let workers load heavy data once per container.

• Ignoring timeouts and retries on noisy APIs:
  LLM APIs and network calls fail in real life. Don't leave this to chance. Always configure timeout and modal.Retries on your worker functions so individual eval examples fail fast and retry, instead of stalling the whole run.

Real-World Example

Imagine you’re evaluating a new RAG pipeline on 50k questions against your internal knowledge base. Locally, running this eval takes hours or days. On Modal, you:

1. Package your RAG code and dependencies into an Image.
2. Write a single @app.function that:
   • Builds the retrieval query.
   • Calls your model (maybe a vLLM endpoint also running on Modal).
   • Scores the answer (exact match, BLEU, or a judge LLM).
3. Use .map() to fan that function out to 50k inputs.

Modal spins up thousands of CPU or GPU containers across its capacity pool, executes your eval with sub-second cold starts, and streams results back. If a few calls fail due to transient errors, the configured retries clean them up. You watch progress in the CLI and Modal UI, then compute aggregate metrics inside your driver. No cluster, no custom queue, no YAML.

Pro Tip: For big evals you’ll rerun often, store raw per-example results in a Modal Volume or object store from the worker itself. That way you can re-slice and recompute aggregate metrics later without rerunning the model.

Summary

You don’t need to build or manage a cluster to fan out eval runs to thousands of parallel Python workers. With Modal, your “cluster” is a handful of decorators and method calls:

• Describe your environment and hardware in Python (Image, gpu=...).
• Turn your eval logic into a @app.function.
• Fan out with .map() or .spawn(), then aggregate in a plain Python loop.

You get elastic capacity, sub-second cold starts, retries, and logging out of the box—so you can spend your time improving models, not babysitting infrastructure.

Next Step

Get Started