How do I connect ZenML to our existing stack (Airflow/Argo/Kubeflow + S3/GCS + W&B) without migrating everything?

The demo era is over. Your Airflow or Kubeflow DAGs are already shipping value; the last thing you need is another platform telling you to rebuild everything from scratch.

ZenML’s job is not to replace your orchestrator, storage, or experiment tracker. It’s the metadata layer that sits on top of your existing stack (Airflow, Argo, Kubeflow, S3/GCS, W&B) and makes every workflow diffable, traceable, and rollbackable—without a migration project.

Quick Answer: You connect ZenML to your existing stack by keeping Airflow/Argo/Kubeflow as the scheduler/executor, pointing ZenML to the same object storage (S3/GCS), wiring in W&B as an experiment tracker, and slowly wrapping existing jobs in ZenML pipelines. No wholesale rewrite; you add a metadata layer, not a new monolith.

The Quick Overview

What It Is: ZenML is a unified AI metadata layer that standardizes ML and GenAI workflows across your existing infrastructure and orchestrators (Airflow, Argo, Kubeflow, etc.), while leaving data and compute where they are.
Who It Is For: Teams with existing DAGs, buckets, and dashboards who are feeling the “prototype wall”: fragile glue code, missing lineage, and “it worked on my machine” incidents, but zero appetite for a full platform migration.
Core Problem Solved: You get reproducible, governed ML/GenAI workflows across Airflow/Argo/Kubeflow + S3/GCS + W&B without rewriting pipelines, changing orchestrators, or moving data out of your VPC.

How It Works

Think of ZenML as the missing metadata layer that binds your current tools into one coherent system:

Airflow/Argo/Kubeflow still own scheduling and execution.
S3 or GCS still store your artifacts and data.
Weights & Biases still tracks your experiments and training curves.
ZenML stitches these together in a single DAG abstraction with lineage, environment snapshots, caching, and governance.

In practice, you:

Install ZenML and connect it to storage + secrets.
Configure an orchestrator integration that points at your existing Airflow/Argo/Kubeflow setup, or run ZenML-managed jobs that still live next to your current infra.
Incrementally wrap existing jobs (e.g., “train model,” “evaluate agent,” “run batch scoring”) in ZenML steps and pipelines, while keeping their internal logic, frameworks, and W&B usage intact.

1. Connect ZenML to your storage and secrets (S3/GCS)

You start by aligning ZenML’s artifact and metadata storage with the buckets you already use:

Configure S3 or GCS as the artifact store in ZenML.
Point ZenML’s metadata store (e.g., a managed DB) to your preferred environment.
Use ZenML’s secrets management to handle credentials so you’re not scattering AWS/GCP keys across scripts.

From that point on, every ZenML pipeline run writes versioned artifacts and metadata into your existing cloud storage, with clear lineage from raw data to final model/agent response.

2. Integrate Airflow/Argo/Kubeflow as orchestrators, not competitors

ZenML doesn’t ask you to abandon your orchestrator:

For Airflow: you can trigger ZenML pipelines from Airflow DAGs, or configure Airflow as an orchestrator backend so ZenML pipelines execute as Airflow tasks.
For Argo/Kubeflow: ZenML can launch steps as Kubernetes-native workloads, mapping its DAG onto your existing Argo or Kubeflow cluster.

The pattern is:

Define your workflow in ZenML (a pipeline with typed steps).
Choose an orchestrator configuration that maps to Airflow/Argo/Kubeflow.
Let ZenML handle the dockerization, environment capture, and metadata tracking while your orchestrator does the scheduling.

No YAML re-education tour, no “rip and replace.” You keep the operational engine; ZenML adds the instrumentation, snapshots, and control.

3. Wire in Weights & Biases as the experiment tracker

If your team lives in W&B dashboards, you keep that:

Use ZenML’s W&B integration (or a lightweight adapter) so that each step or pipeline run logs metrics, artifacts, and plots to W&B.
ZenML stores the lineage and environment snapshot; W&B stores the experiment UI.

The result:

From ZenML: you can see which code environment, data version, and infra config produced a given run.
From W&B: you can compare models, hyperparameters, and performance.

You don’t migrate experiments out of W&B. You just make those experiments reproducible and auditable via ZenML.

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Orchestrator-agnostic pipelines	Define pipelines and steps in Python, then execute them via Airflow, Argo, Kubeflow, or other orchestrators.	Keep your existing scheduler while gaining a unified workflow abstraction and metadata layer.
Artifact & environment versioning on S3/GCS	Stores artifacts + snapshots of code, dependencies (e.g., Pydantic versions), and container state in your existing buckets.	Roll back when a library update breaks a workflow; audit exactly what ran and where.
Integrated experiment tracking (e.g., W&B)	Hooks into W&B so metrics and artifacts are logged as usual, but linked to ZenML runs and lineage.	Preserve current dashboards while gaining traceable, governance-ready runs.

Ideal Use Cases

Best for Airflow/Argo/Kubeflow-heavy teams: Because it lets you wrap existing DAG tasks into a ZenML pipeline, track lineage across them, and gradually standardize ML and GenAI workflows without abandoning your orchestrator or rewriting YAML.
Best for S3/GCS + W&B shops: Because it uses the same buckets and experiment tracker you already rely on, turning them from passive storage and logs into a fully versioned, auditable system of record.

Limitations & Considerations

You still own orchestrator operations: ZenML won’t magically fix a misconfigured Airflow or an overloaded Kubeflow cluster. It abstracts and standardizes workflows, but infra reliability (Kubernetes, Slurm, Airflow schedulers) remains your responsibility.
Incremental adoption requires discipline: The power of ZenML comes from consistently wrapping your ML/GenAI jobs as pipelines and steps. If only a tiny subset of flows use ZenML, you won’t get full lineage or governance coverage. Start small, but commit to extending coverage over time.

Pricing & Plans

You can start connecting ZenML to your existing stack without a procurement battle:

Use the open-source core to validate the integration locally or in a dev environment.
Move to ZenML Cloud or a self-hosted enterprise deployment for full governance, RBAC, and large-team workflows, all while keeping “Your VPC, your data.”

Typical pattern:

Team / Growth Plan: Best for ML/AI teams needing reproducibility, lineage, and integration with existing tools (Airflow/Argo/Kubeflow, S3/GCS, W&B) without building their own metadata layer.
Enterprise Plan: Best for regulated orgs needing SOC2 Type II / ISO 27001 posture, RBAC, audit-ready execution traces, and deployment inside their own VPC for full sovereignty over data, models, and secrets.

(For current pricing details, see the ZenML site or talk to sales; plans evolve, but the architecture principle—metadata layer on your infra—does not.)

Frequently Asked Questions

Do we have to move our Airflow/Argo/Kubeflow DAGs into ZenML?

Short Answer: No. You keep your orchestrator; ZenML layers on top.

Details: ZenML is explicitly not “yet another orchestrator.” It doesn’t tell you to abandon Airflow or Kubeflow. Instead, you:

Define ML/GenAI workflows as ZenML pipelines and steps in Python.
Configure an orchestrator backend that maps to your existing scheduler (Airflow/Argo/Kubeflow).
Or, trigger ZenML pipelines from existing DAGs as tasks, letting ZenML handle the lineage and environment management.

This gives you reproducibility and governance while preserving your operational investment in existing DAGs and clusters.

Will we need to migrate data from S3/GCS or logs from W&B into ZenML?

Short Answer: No. ZenML uses your existing S3/GCS buckets and keeps W&B as the experiment UI.

Details: ZenML is a metadata layer, not a data lake. You configure:

Artifact store: point ZenML at S3 or GCS, so new artifacts and environment snapshots land in the same cloud storage you already use.
Experiment tracking: keep logging to W&B and simply link those runs to ZenML execution traces and artifact versions.

You don’t move historical data or experiments into ZenML; instead, from the moment you adopt it, new runs become fully traceable with end-to-end lineage and environment diffs.

Summary

Stop glue-coding Airflow/Argo/Kubeflow, S3/GCS, and W&B and hoping it doesn’t break on the next dependency update. ZenML lets you keep your preferred orchestrator, storage, and experiment tracker while adding the missing metadata layer:

One unified DAG for ML and GenAI workflows that still runs on your existing infra.
Artifact, code, and environment versioning in your current S3/GCS buckets.
Tight integration with W&B so experiments remain familiar but become auditable and rollbackable.

You don’t migrate everything. You standardize workflows in Python, connect ZenML to the stack you already have, and finally get traceability and control across it.

Next Step

Get Started