How do I connect ZenML to our existing stack (Airflow/Argo/Kubeflow + S3/GCS + W&B) without migrating everything?

The demo era is over. You don’t have time to rewrite every pipeline just to get reproducibility, lineage, and control over ML and GenAI in production. You already have Airflow or Argo or Kubeflow orchestrating jobs. Your data is in S3 or GCS. Your experiments live in Weights & Biases. The question isn’t “how do we replace this?”—it’s “how do we add the missing layer without migrating everything?”

Quick Answer: You connect ZenML as a metadata layer on top of your existing stack: keep Airflow/Argo/Kubeflow as orchestrators, keep S3/GCS as artifact storage, keep W&B for experiment tracking, and let ZenML tie them together into reproducible, diffable pipelines—without moving your data or jobs.

The Quick Overview

What It Is: ZenML is a metadata-first AI engineering layer that standardizes ML and GenAI pipelines across your existing orchestrators and storage, adding lineage, caching, and governance on top of Airflow/Argo/Kubeflow, S3/GCS, and W&B.
Who It Is For: Teams that already run production workloads on Airflow/Argo/Kubeflow and store artifacts in S3/GCS, but are stuck in “prototype wall” territory: fragile scripts, missing lineage, and non-reproducible runs—even with W&B in the loop.
Core Problem Solved: ZenML lets you keep your current stack but stop glue-coding it. You get unified pipelines, versioned artifacts, and audit-ready run histories without rewriting everything or locking into a new orchestrator.

How It Works

ZenML doesn’t replace Airflow, Argo, or Kubeflow. It wraps your existing workflows in a standardized pipeline + step abstraction, tracks the exact code/dependency/container environment, and delegates execution to your orchestrators while reading/writing artifacts to S3 or GCS. W&B stays your experiment dashboard; ZenML becomes the backbone that ties code, data, runs, and infra together.

At a high level, the integration works in three phases:

Connect your infrastructure:
Point ZenML at your existing orchestrator(s), S3/GCS buckets, and W&B project. You keep using the same clusters and storage; ZenML just gains the ability to schedule and track runs there.
Wrap existing logic as ZenML steps:
Instead of rewriting all logic, you wrap key parts (training, evaluation, agent loops) as ZenML steps in Python. Under the hood, those steps still run inside Airflow/Argo/Kubeflow jobs and still log to W&B, but ZenML version-controls code, artifacts, and environments.
Evolve from “jobs” to pipelines:
You gradually bind steps into ZenML pipelines, which execute as DAGs on your chosen orchestrator. ZenML adds lineage, caching, and rollback across runs—while your data stays in S3/GCS and your metrics stay in W&B.

How It Works in Practice

1. Connect ZenML to Airflow/Argo/Kubeflow

You don’t rip out your orchestrator. You configure ZenML to talk to it:

For Airflow:
- Register an Airflow orchestrator with ZenML (point to your Airflow deployment).
- ZenML maps pipeline runs to Airflow DAGs and tasks; Airflow still handles scheduling and retry logic.
For Kubeflow or Argo:
- Register a Kubeflow or Kubernetes-based orchestrator.
- ZenML compiles pipelines into DAGs executed on your existing cluster, respecting the same namespaces, node pools, and autoscaling settings.

From your team’s perspective: they write @pipeline and @step in Python, and the runs just show up as DAGs in Airflow/Argo/Kubeflow as before—no duplicate orchestration layer, no new scheduler to maintain.

2. Point ZenML at S3 or GCS for artifacts

You don’t move data; you standardize how it’s written:

Configure an artifact store in ZenML:
- S3 example: s3://your-ml-artifacts-bucket
- GCS example: gs://your-ml-artifacts-bucket
ZenML then:
- Stores step inputs/outputs there (datasets, models, evaluation reports).
- Records metadata and lineage for each artifact (which code version produced it, which container image, which run).

What changes: instead of every script choosing its own folder inside S3/GCS, ZenML enforces a consistent artifact layout and links each object to an execution trace.

3. Keep Weights & Biases, add ZenML’s lineage and environment tracking

You can keep W&B as the primary experiment UI:

In each ZenML step, you still call wandb.init and log metrics/artifacts.
ZenML tracks:
- Which step and pipeline each W&B run came from.
- The environment: code hash, dependency versions (e.g., Pydantic, PyTorch, LangChain), container image, hardware.
Result: you can navigate from a W&B experiment to:
- The exact ZenML run and pipeline.
- The S3/GCS artifacts it produced.
- The orchestrator job / pod that executed it.

Orchestration gives scheduling. W&B gives nice charts. ZenML makes the whole thing reproducible and auditable.

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Orchestrator-agnostic pipelines	Defines pipelines/steps in Python and executes them on Airflow, Argo, or Kubeflow without rewriting your jobs.	Keep your existing orchestrator; add a consistent pipeline abstraction and metadata layer on top.
Unified artifact & environment tracking on S3/GCS	Stores artifacts in your S3/GCS buckets while tracking code, dependencies, and container state for every step.	Eliminate “it worked on my machine” and missing lineage; make every run diffable and rollbackable.
Native W&B integration	Links W&B runs to ZenML pipeline steps and artifacts.	Keep your experiment dashboard while gaining full end-to-end execution traces across ML and GenAI workloads.

Ideal Use Cases

Best for “don’t touch my Airflow” teams:
Because ZenML doesn’t fight your orchestrator. You keep Airflow DAGs (or Argo/Kubeflow workflows) and let ZenML manage pipeline definitions, metadata, and artifact versioning on top.
Best for “S3/GCS + W&B but no lineage” teams:
Because you already log metrics and store models, but you can’t answer “which code version produced this model?” or “what changed between these two agent runs?” ZenML connects S3/GCS and W&B into a traceable, governed system without migrating storage.

Example Integration Pattern (Step‑by‑Step)

Below is a concrete pattern I’ve used in regulated environments to connect a legacy stack without disrupting workloads.

Bootstrap ZenML in your VPC
- Deploy ZenML where your orchestrator lives (Kubernetes cluster, VM, etc.).
- Configure authentication to S3 or GCS using your existing IAM roles/service accounts.
- Confirm that “Your VPC, your data” holds: data and compute never leave your infra; only metadata is managed by ZenML.
Register your existing resources
- Create an artifact store:
  - Example: zenml artifact-store register prod-s3-store --flavor=s3 --path=s3://ml-artifacts-prod
- Register an orchestrator:
  - Airflow example: zenml orchestrator register prod-airflow --flavor=airflow --...
  - Kubeflow example: zenml orchestrator register prod-kubeflow --flavor=kubeflow --...
- Optionally configure a container registry that you already use for Argo/Kubeflow images.
Add a minimal ZenML pipeline around existing code
- Start with one critical flow: model training, batch evaluation, or a GenAI agent loop.
- Wrap the core logic into ZenML steps:
  - Data loading (from S3/GCS)
  - Training (Scikit-learn, PyTorch, XGBoost, etc.)
  - Evaluation + logging to W&B
- Each step runs as a task in Airflow/Argo/Kubeflow; ZenML intercepts inputs/outputs to version artifacts and environments.
Integrate W&B inside steps
- Use your existing W&B patterns inside ZenML steps:
```
@step
def train_model(config) -> ModelArtifact:
    import wandb
    wandb.init(project="your-project", config=config)
    # train model, log metrics
    ...
    return model
```
- ZenML links that step to:
  - The W&B run ID
  - The artifact stored in S3/GCS
  - The container + dependency versions
Scale out incrementally
- Once one pipeline works end-to-end:
  - Migrate additional jobs into ZenML pipelines at your own pace.
  - Reuse existing Airflow DAG schedules by calling ZenML runs from DAG tasks if you prefer a hybrid approach.
- Over time, your stack evolves from “N untracked jobs” to “N governed pipelines” without a big‑bang migration.

Limitations & Considerations

You still need to model pipelines once:
ZenML won’t infer your DAG from existing Airflow code. You will define pipelines and steps in Python. The payoff is you get a unified, orchestrator-agnostic representation that survives infra changes—but expect some incremental refactoring.
Dual visibility at first:
For a while, you’ll see both Airflow/Argo/Kubeflow and ZenML UIs. That’s intentional: orchestrators own scheduling; ZenML owns lineage, caching, and run metadata. Over time, most teams use the orchestrator UI for ops incidents and ZenML for development, governance, and audits.

Pricing & Plans

ZenML is open source at its core, with a managed cloud offering and enterprise options when you need them.

Open Source / Community: Best for individual engineers or small teams needing full control and willing to manage infrastructure themselves. Ideal when you want to test-drive ZenML on top of an existing Airflow/Argo/Kubeflow + S3/GCS + W&B setup.
ZenML Cloud / Enterprise: Best for teams that want SOC2 Type II / ISO 27001–aligned operations, RBAC, and enterprise support while keeping data and compute in their own clouds. Ideal if you need help wiring multiple orchestrators, hardening RBAC, and standardizing pipelines across ML and GenAI workloads.

(For current pricing details, check the ZenML site—you’ll choose based on user count, support level, and whether you need managed hosting vs full self‑hosting.)

Frequently Asked Questions

Can I keep Airflow/Argo/Kubeflow and just “drop ZenML in”?

Short Answer: Yes. ZenML is designed to sit on top of your existing orchestrators, not replace them.

Details:
ZenML doesn’t take an opinion on the orchestration layer. You register Airflow/Argo/Kubeflow as orchestrators in ZenML, and pipelines you define in Python are executed there as DAGs/workflows. All your existing scheduling, SLAs, and infra investments remain. ZenML adds:

Pipeline and step abstractions that are orchestrator‑agnostic.
Artifact and environment versioning tied to each run.
Execution traces and lineage across ML and GenAI steps.

You can even call ZenML runs from existing Airflow DAGs if you want a gradual adoption path.

Do I have to move my artifacts out of S3/GCS or stop using W&B?

Short Answer: No. You keep S3/GCS and W&B; ZenML standardizes how they’re used and connects them into a coherent lineage.

Details:
For storage, you configure S3 or GCS as the ZenML artifact store. ZenML will:

Use your existing buckets as-is (no data copy).
Organize run artifacts under a consistent layout.
Record metadata about each artifact and its producing run.

For W&B, you continue calling the W&B SDK in your code. ZenML doesn’t compete with W&B’s experiment UI; it enriches it by:

Linking W&B runs to specific pipeline steps and runs.
Capturing code, dependencies, and container versions behind each experiment.
Allowing you to audit from “this W&B run” back to “this orchestrator pod and S3/GCS artifact.”

Summary

If your stack today is “Airflow/Argo/Kubeflow + S3/GCS + W&B,” you already have the right building blocks—but you’re missing the layer that makes it reproducible, governed, and easy to evolve. ZenML plugs into your existing orchestrators, uses your existing buckets, and respects your existing experiment tracking, while turning disconnected jobs into standardized ML and GenAI pipelines.

No migration of data. No orchestrator replacement. Just a metadata layer that tracks every run, every artifact, and every environment so you can ship from notebook to production without glue‑coding and “it worked on my machine” drama.

Next Step

Get Started