How does ZenML keep data and artifacts in our VPC—what exactly gets sent to ZenML Pro (metadata-only) and what stays in our cloud?

The short version: ZenML Pro only ever sees metadata about your ML and GenAI workflows, not the underlying data or model artifacts themselves. Your features, labels, embeddings, models, and logs stay inside your VPC and your storage accounts; ZenML Pro stores encrypted records about what ran, where, and with which versions.

Quick Answer: ZenML Pro is a metadata layer on top of your infrastructure. Data and artifacts live in your cloud; ZenML only stores encrypted metadata like pipeline names, step statuses, metrics, and references to where your artifacts live.

The Quick Overview

• What It Is: A metadata-first control plane for ML and GenAI workflows that runs on top of your existing infrastructure and tools, without pulling data out of your VPC.
• Who It Is For: Teams who want reproducible, auditable pipelines and agent workflows (with full lineage and governance) but must keep data and artifacts inside their own cloud for compliance, sovereignty, or performance.
• Core Problem Solved: You need centralized lineage, tracking, and governance across pipelines and orchestrators, but you cannot afford to send raw data, features, or models out of your environment.

How It Works

Think of ZenML Pro as the “brain” that remembers everything about your workflows, not the “muscle” that moves your data. Orchestrators like Airflow or Kubeflow, compute like Kubernetes or Slurm, and storage like S3, GCS, Azure Blob, or on-prem object stores stay inside your VPC. ZenML Pro just tracks what happened and where things are stored, not the actual payloads.

Under the hood:

1. Inside Your VPC: Pipelines, Data, and Artifacts

   • Your code runs in your own compute (Kubernetes, Slurm, VMs).
   • Data flows between your own storage systems (data lake, feature store, model registry, vector DB).
   • Artifacts like trained models, embeddings, eval reports, and logs are written to your storage backends.

2. Across the Wire: Metadata Only

   • ZenML’s client and server exchange metadata: pipeline definitions, run status, metrics, lineage relationships, and pointers (URIs) to artifacts.
   • No dataset rows, model weights, or embeddings are shipped to ZenML Pro; only IDs and descriptors.
   • All metadata is encrypted in transit and at rest.

3. ZenML Pro: Metadata Control Plane

   • ZenML Pro acts as a central metadata store and dashboard.
   • You get unified lineage, run histories, execution traces, and governance controls across ML and GenAI workflows, without exposing raw data.

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Metadata‑only architecture	Syncs pipeline, run, and artifact metadata to ZenML Pro while keeping data and artifacts in your VPC storage.	Maintain full sovereignty (“your VPC, your data”) without losing observability or control.
Secure service connectors	Generate short‑lived credentials inside your environment to talk to cloud services and data stores.	Remove hard‑coded secrets and keep access control aligned with your security and RBAC policies.
End‑to‑end lineage & tracing	Records how data, code, and artifacts relate (from raw data to final agent response) based on metadata and URIs.	Audit every release and agent workflow without centralizing data or models in a third‑party system.

What Exactly Gets Sent to ZenML Pro?

Let’s make the boundary concrete. When a pipeline or agent workflow runs against ZenML Pro, the following categories of information go to ZenML; everything else stays in your cloud.

1. Pipeline and Step Metadata

Sent to ZenML Pro (metadata-only):

• Pipeline identifiers and names
  • e.g., credit_default_risk_training, llm_agent_eval_loop
• Step identifiers, names, and ordering
  • e.g., load_raw_data, train_xgboost_model, run_langgraph_agent
• Configuration dictionaries stripped of secrets
  • Hyperparameters (learning rate, batch size)
  • Evaluation thresholds and scoring configs
  • Environment tags (e.g., env=prod, team=fraud, region=eu-central-1)
• Structural info about the DAG
  • Which steps depend on which outputs
  • Which artifacts are produced/consumed

Stays in your VPC / cloud:

• The actual pipeline code files and repository (these live where you keep them: Git, internal registry, etc.)
• Any local configuration files that contain secrets or data-specific logic (ZenML treats secrets separately).
• Notebooks, scripts, internal documentation.

ZenML Pro knows that you have a step called train_xgboost_model with certain hyperparameters, not what rows or features it saw when it ran.

2. Run and Execution Status

Sent to ZenML Pro:

• Run identifiers and timestamps
  • e.g., run_id, start/end time, triggered-by (manual, CI, schedule)
• State transitions
  • queued, running, succeeded, failed, canceled
• Basic runtime metrics
  • Duration per step
  • High‑level resource usage summaries if you choose to log them
• Error messages and stack traces (configurable)
  • Enough to debug why a step failed without exposing sensitive batch contents

Stays in your VPC / cloud:

• Underlying orchestrator logs if you keep them local (Kubernetes logs, Slurm logs, Airflow task logs, etc.)
• Any data-bearing logs you write (per-row errors, row-level predictions, raw LLM prompts/responses if you log them to your own systems).

You control how verbose error messages are. If you’re in a highly regulated space, you can ensure stack traces don’t leak PII or sensitive payloads.

3. Artifact Metadata (Not the Artifacts Themselves)

Sent to ZenML Pro:

• Artifact identifiers and types
  • e.g., “model artifact of type sklearn.Estimator,” “embedding_index of type faiss_index”
• Storage references / URIs (but not contents)
  • s3://my-bucket/models/credit_default/2024-03-25/model.pkl
  • gs://my-feature-store/customer_features/v1.parquet
• Versioning metadata
  • Which run produced this artifact
  • Hashes or fingerprints (for deduplication and diffing)
  • Tags/labels (e.g., production_candidate=true, model_family=llama-3)
• Lightweight metrics associated with the artifact
  • Aggregate evaluation metrics (AUC, accuracy, BLEU, win-rate)
  • Validation statistics (e.g., missing value ratios) if logged as scalars

Stays in your VPC / cloud:

• The actual artifact payloads:
  • Model weight files, checkpoints, ONNX exports, TorchScript, etc.
  • Datasets, feature tables, parquet/CSV files.
  • Vector indices and embeddings (e.g., used with LlamaIndex, LangChain, or custom retrieval).
  • Evaluation datasets and detailed per‑example metrics.
• Any files in your object store / file systems referenced by URI.

ZenML effectively remembers where the artifact lives and how it was produced, but never reads or exfiltrates its content.

4. Metrics, Parameters, and Tags

Sent to ZenML Pro:

• Scalar metrics and aggregations you explicitly log
  • Training loss curves (as scalar points)
  • Validation metrics (e.g., accuracy=0.93, roc_auc=0.98)
  • Latency and cost metrics for LLM calls (e.g., avg tokens, p95 latency)
• Run parameters (non-sensitive)
  • Model/hyperparameter settings
  • Feature flags and environment toggles
• Tags and labels for runs, pipelines, artifacts
  • release=2024.04.01, dataset_version=v7, region=eu

Stays in your VPC / cloud:

• Underlying metric source data:
  • Raw predictions, true labels, error distributions at row or example level.
  • Full prompt/response logs for LLMs (if you store those yourself).
• Any sensitive identifiers or PII you explicitly keep out of ZenML metric payloads.

You decide what to log as a numeric or string metric. If it’s sensitive, keep it in your data systems; ZenML just needs enough to track quality and compare runs.

5. Code and Environment Snapshots

A critical part of ZenML’s value is the ability to answer: “What code and environment produced this artifact?” That’s where metadata again does the heavy lifting.

Sent to ZenML Pro:

• References to code versions
  • Git SHA, branch, repo URL (not the repo contents themselves).
  • Optional labels like release_tag=v1.3.2.
• Environment metadata
  • Python version, OS, CPU/GPU info.
  • Dependency versions (e.g., pydantic==2.6.0, torch==2.2.1).
• Container metadata
  • Image name and tag used to run the step (e.g., registry.internal/llm-agent:2024-03-25).
  • Optionally a hash of the image for immutability checks.

Stays in your VPC / cloud:

• The actual containers and images in your registry.
• Your private repositories and code artifacts.
• Any build logs, CI/CD config, or internal deployment scripts.

ZenML lets you diff runs based on this metadata (e.g., “this agent started failing after we bumped langchain from 0.1.17 to 0.1.18”), without cloning your repos or registry contents.

6. Security and Connectivity Information

Sent to ZenML Pro:

• Service connector metadata (but not raw long‑lived secrets):
  • Connector names and types (e.g., “AWS‑S3‑prod”, “GCP‑BigQuery‑dev”).
  • Scopes and roles requested.
  • Expiration timestamps for short‑lived tokens.
• RBAC configuration
  • User, role, and permission records to control who can see what metadata.
  • Audit logs of access to pipeline and run metadata.

Stays in your VPC / cloud:

• Data and model API keys and credentials used to access internal services.
• Long‑lived secrets and keys where your own secret manager stores them (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, etc.).
• Direct network connectivity to your databases, object stores, and internal APIs.

ZenML’s service connectors are designed to avoid the “spread secrets everywhere” pattern. They centralize how short‑lived access is issued without pulling sensitive creds into ZenML’s SaaS.

Ideal Use Cases

• Best for regulated workloads (finance, healthcare, public sector): Because you get SOC2 and ISO 27001‑aligned metadata handling, audit‑grade lineage, and centralized governance while keeping all data and models inside your VPC and existing compliance perimeter.
• Best for multi‑team, multi‑orchestrator ML & GenAI platforms: Because ZenML Pro unifies metadata across Airflow, Kubeflow, Kubernetes, and Slurm without re‑homing data or models into a new platform, so every team keeps their tools and infra but shares one metadata plane.

Limitations & Considerations

• ZenML is not your data lake or model store: By design, ZenML Pro doesn’t host your data or model binaries. You still need robust storage (S3/GCS/Azure/on‑prem) and, if desired, a dedicated model registry. ZenML orchestrates and tracks; it doesn’t replace foundational storage.
• Metadata can still be sensitive: Even if no raw data flows to ZenML, names, tags, and metrics can carry business-sensitive context. Treat metadata with the same governance mindset you apply to logs and dashboards; use RBAC and naming conventions accordingly.

Pricing & Plans

ZenML Pro pricing is structured around the value of a managed metadata layer and control plane, not around hosting your data.

• Pro / Team Plan: Best for product teams and platform squads needing a fully managed ZenML control plane, SOC2 and ISO 27001 compliance, and straightforward integration with existing VPC-based infrastructure—without adding new data egress risks.
• Enterprise Plan: Best for organizations with stricter compliance, custom VPC networking requirements, or advanced RBAC/audit needs, where “your VPC, your data” is non‑negotiable and ZenML is deployed and integrated according to internal security policies.

(For exact pricing details and plan breakdowns, talk to the ZenML team from the signup page.)

Frequently Asked Questions

Does any of my training or inference data ever leave my VPC to ZenML Pro?

Short Answer: No. ZenML Pro only receives metadata; your data remains in your VPC and your storage systems.

Details: The architecture is explicitly designed so that no data flows through ZenML Pro. Pipelines run where they always have—inside your Kubernetes cluster, on Slurm, or in your managed cloud compute. Data moves between your object stores, databases, and vector DBs. ZenML Pro only stores encrypted metadata such as pipeline names, run statuses, metrics, and URIs pointing to where artifacts live. This is the core “metadata layer on top of your infrastructure” design: you get observability and governance, but data never leaves your perimeter.

How does ZenML Pro access my internal services securely without storing my secrets?

Short Answer: Most actions use local authentication in your environment, and ZenML’s service connectors issue short‑lived tokens instead of persisting long‑lived secrets in the SaaS.

Details: When a pipeline step needs to talk to an internal service (S3 bucket, database, feature store, vector DB, etc.), it does so using credentials that live inside your VPC—via your secret manager or node IAM roles. ZenML Pro only keeps metadata about the connector and its scopes; the actual secret material either never leaves your environment or is used to mint short‑lived tokens that expire quickly. This aligns with “your VPC, your data” and avoids turning ZenML into a central secret honeypot. Combined with SOC2 Type II and ISO 27001 compliance and encryption in transit and at rest, this gives you a secure metadata layer without expanding your data attack surface.

Summary

If you’ve avoided external platforms because you cannot let data or models leave your VPC, ZenML Pro is built explicitly around that constraint. It acts as a metadata control plane on top of your existing orchestrators and infrastructure: it tracks pipeline structure, run status, metrics, environment and dependency snapshots, and artifact URIs—but the underlying datasets, model weights, embeddings, and logs stay where they belong in your cloud.

You get full lineage from raw data to final agent response, diffable runs, and governance across ML and GenAI workflows, without routing a single row of data through ZenML’s servers.

Next Step

Get Started