platforms where infra is defined in Python code (images, secrets, schedules) instead of Kubernetes YAML

Most teams running serious AI workloads eventually hit the same wall: the infra is all Kubernetes YAML and Terraform, but the workload is written in Python. Every deploy means bouncing between languages, tools, and mental models—and your iteration speed pays the price. There’s a growing class of platforms where you define infrastructure directly in Python: images, secrets, schedules, hardware, and endpoints all live next to your application code.

Quick Answer: You don’t have to live in Kubernetes YAML anymore. Platforms like Modal, AWS CDK, Pulumi, and Prefect let you define infrastructure as Python code—covering containers/images, secrets, scheduling, and autoscaling. Modal is specifically optimized for AI workloads, so your Python functions become GPU-backed, autoscaling endpoints without writing YAML or managing clusters.

Why This Matters

When infra is defined in Python code instead of Kubernetes YAML, your team gets a single language and toolchain for both application logic and operations. That means you can:

• Version, test, and refactor infra like any other Python module.
• Keep environment, hardware, and scaling rules in sync with the functions that actually run.
• Cut out the “throw it over the wall to DevOps” loop, which kills iteration speed on ML, LLM, and data workloads.

This is especially critical for AI systems that need to scale elastically on GPUs, run spiky evals or RL rollouts, and ship new variants several times a day. Defining infra in Python code puts the knobs you actually care about—images, secrets, schedules, and concurrency—directly in the hands of the engineers building the models and services.

Key Benefits:

• Single-language workflow: Define endpoints, queues, schedules, images, and secrets all in Python instead of juggling YAML, Helm, and Terraform.
• Tighter feedback loops: Spin up, modify, and redeploy infra with the same tools you use to run and test your code locally, keeping iteration loops fast.
• Production-grade by default: Get autoscaling, GPU selection, observability, and security controls as standard primitives instead of bespoke Kubernetes glue.

Core Concepts & Key Points

Concept	Definition	Why it's important
Programmable Infra in Python	Expressing infrastructure (containers, networks, schedulers, secrets) via Python APIs and objects instead of static YAML manifests.	Eliminates context-switching and lets you use the full Python ecosystem (testing, refactors, type checking) for infra code.
Image-Defined Environments	Defining runtime environments (OS packages, Python deps, CUDA, tools) as Python-constructed images rather than Dockerfiles and CI pipelines.	Keeps your production runtime in lockstep with code, and makes GPU-optimized images reproducible and fast to iterate on.
Code-Level Schedules & Endpoints	Declaring cron jobs, queues, and web endpoints with decorators and Python objects instead of separate config files or dashboards.	Ensures operational behavior lives next to the functions that implement it, reducing drift and surprises in production.

How It Works (Step-by-Step)

Let’s walk through how a Python-defined infra stack typically looks using Modal as the reference point, and then map it to other platforms.

At a high level, Modal lets you:

• Define images in Python (modal.Image) with your dependencies and system packages.
• Declare functions and classes as remotely runnable workloads.
• Attach hardware (CPU/GPU), concurrency, retries, and timeouts declaratively in code.
• Expose them as HTTP endpoints, cron jobs, or batch jobs with decorators.
• Wire in secrets, volumes, and observability without touching YAML or Kubernetes.

1. Define your image, secrets, and hardware in Python

You start by importing Modal and describing your environment:

import modal

image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install("torch", "transformers", "fastapi", "uvicorn")
    .apt_install("git")
)

app = modal.App("llm-service")

openai_secret = modal.Secret.from_name("openai-api-key")

You can also pin CUDA versions or specific GPU types:

gpu_image = (
    modal.Image.debian_slim()
    .apt_install("git")
    .pip_install("torch==2.2.0", "flash-attn==2.5.6")
)

No Dockerfile. No Helm chart. The image, packages, and secrets are all first-class Python objects.

2. Turn Python functions into scalable, GPU-backed workloads

Now take a plain Python function and make it remotely runnable:

@app.function(
    image=gpu_image,
    gpu="A10G",
    timeout=600,
    retries=modal.Retries(max_retries=3),
    concurrency_limit=32,
    secrets=[openai_secret],
)
def generate(text: str) -> str:
    # Load your model lazily or use a class with @modal.enter
    from transformers import AutoModelForCausalLM, AutoTokenizer

    model_name = "meta-llama/Llama-3-8b"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(model_name).cuda()

    tokens = tokenizer(text, return_tensors="pt").to("cuda")
    out = model.generate(**tokens, max_new_tokens=128)
    return tokenizer.decode(out[0], skip_special_tokens=True)

generate.remote("Hello") now runs this in a container with a GPU, your image, and your secrets.

For stateful model servers (load weights once per container), you use classes and lifecycle hooks:

@app.cls(
    image=gpu_image,
    gpu="A10G",
    concurrency_limit=16,
    secrets=[openai_secret],
)
class LlamaServer:
    @modal.enter()
    def load(self):
        from transformers import AutoModelForCausalLM, AutoTokenizer

        model_name = "meta-llama/Llama-3-8b"
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(model_name).cuda()

    @modal.method()
    def generate(self, text: str) -> str:
        tokens = self.tokenizer(text, return_tensors="pt").to("cuda")
        out = self.model.generate(**tokens, max_new_tokens=128)
        return self.tokenizer.decode(out[0], skip_special_tokens=True)

Under the hood, Modal’s “AI-native runtime” handles sub-second cold starts, autoscaling, and multi-cloud GPU capacity. You just define the class; Modal decides where and how to run it.

3. Expose endpoints, schedules, and batch workloads in the same file

Now you wire that server to HTTP and schedules directly in Python:

from fastapi import FastAPI
from pydantic import BaseModel

web = FastAPI()

class GenerateRequest(BaseModel):
    prompt: str

@modal.fastapi_endpoint(app=app, web=web)
async def fastapi_app():
    return web

@web.post("/generate")
async def generate_route(req: GenerateRequest):
    # call the Modal server class
    return {"output": LlamaServer.generate.remote(req.prompt)}

Scheduling a job is also just a decorator:

@app.function(schedule=modal.Cron("0 * * * *"))  # run at top of every hour
def refresh_index():
    # fetch data, compute embeddings, update an index in a Volume
    ...

Fan-out batch workloads use .map() or .spawn() as Python-level primitives:

@app.function(timeout=60 * 60)
def process_item(item_id: int):
    # heavy CPU/GPU work
    ...

def run_batch(item_ids: list[int]):
    # Distribute across thousands of containers
    results = list(process_item.map(item_ids))
    return results

To deploy it all, you run:

modal deploy app.py

No Kubernetes manifests. No separate cron job spec. Infra is just Python you can run, test, and refactor.

---

Other platforms follow similar patterns:

• AWS CDK (Python): Define VPCs, ECS services, Lambda functions, and schedules in Python constructs.
• Pulumi (Python): Use Python to manage infra across AWS/GCP/Azure, including K8s objects if you still need them.
• Prefect (Python): Define workflows and schedules in Python, backing onto its own agents and infra layer.

They differ in focus—Modal is built around Python workloads and AI runtime performance, while CDK/Pulumi target generic cloud infra, and Prefect focuses on orchestration.

Common Mistakes to Avoid

• Treating Python infra as “just code” with no boundaries:
  It’s easy to accidentally mix application logic and infra definitions in a single 2,000-line file. Keep a clear separation between modules that declare images/endpoints/schedules and modules that implement business logic. Use functions and classes as interfaces, not dumping grounds.

• Recreating Kubernetes complexity in Python:
  The goal isn’t to port every knob from a Deployment spec into Python. With Modal, for instance, lean on the high-level primitives—@app.function, @app.cls, .map(), modal.Cron—instead of re-implementing bespoke job queues or homegrown autoscalers. Take advantage of opinionated defaults around timeouts, retries, and concurrency.

Real-World Example

Imagine you’re building an eval harness for a new RAG model. You need to:

• Run 50k eval queries against multiple model variants.
• Scale up to hundreds of GPUs during the eval, then scale back to zero.
• Schedule nightly evals, write results to storage, and keep everything observable.

On a typical Kubernetes stack, you’d glue together:

• Dockerfiles and CI for building images.
• Helm charts for Jobs/Deployments.
• CronJob definitions for scheduling.
• A separate secrets manager (plus wiring).
• Custom scripts for fan-out and result aggregation.

On Modal, you can define the entire pipeline in Python:

import modal

image = (
    modal.Image.debian_slim()
    .pip_install("openai", "pandas", "numpy")
)

app = modal.App("rag-evals")
openai_secret = modal.Secret.from_name("openai-api-key")

@app.function(
    image=image,
    concurrency_limit=1024,
    timeout=600,
    secrets=[openai_secret],
)
def run_single_eval(query: str) -> dict:
    import openai
    # call two models, compute metrics, return row
    ...

@app.function(
    image=image,
    schedule=modal.Cron("0 3 * * *"),  # nightly at 03:00
)
def nightly_eval():
    queries = load_eval_queries()  # pull from S3/db
    results = list(run_single_eval.map(queries))
    write_results(results)

Deploy it with modal deploy evals.py. At runtime:

• Modal spins up thousands of containers across clouds, with no GPU quota gymnastics.
• You see logs, metrics, and duration per function in the Modal dashboard.
• Cold starts stay sub-second because the image is prebuilt and cached.

Pro Tip: Start by modeling your operational units as plain functions (run_single_eval, nightly_eval) and only introduce stateful classes (@app.cls) when you need heavy model initialization amortized across calls. This keeps your mental model simple and your deployments predictable.

Summary

Infra defined in Python code is not a cute abstraction; it’s a way to get Kubernetes out of your hot path so you can ship AI systems faster. Platforms like Modal, AWS CDK, Pulumi, and Prefect all push infra into Python, but Modal is explicitly designed around AI workloads: GPU images, fast autoscaling, sandboxes for untrusted code, and production endpoints all controlled via decorators and simple APIs.

If you’re tired of debugging YAML and maintaining bespoke infra just to run LLMs, RL environments, or big batch jobs, moving to a Python-first infra stack will improve your iteration speed and reduce operational drag—without sacrificing observability, isolation, or capacity.

Next Step

Get Started