What’s a good approach for giving researchers persistent GPU dev environments (Jupyter/SSH) that don’t get wiped?

Most research teams don’t lose productivity because they lack GPUs. They lose it because their dev environments are fragile: a Jupyter node gets reclaimed, a VM is rebuilt, a home directory disappears, and suddenly someone’s week of debugging vanishes.

The goal is simple: let researchers live in “their” GPU dev environment—Jupyter or SSH—with the expectation that it won’t get wiped, even if the underlying compute changes. That means separating identity and storage from ephemeral GPU nodes, and layering on a sane orchestration model.

Below is a practical approach I’d use as an infra owner, plus how this maps onto VESSL AI if you want a managed path.

---

Start with the right mental model: pets vs. cattle for dev

Persistent GPU dev environments shouldn’t be “pet VMs” you never touch. They should be:

• Ephemeral compute (cattle): GPU machines you’re free to stop, scale, or replace.
• Persistent state (not cattle):
  • User home directories and notebooks
  • Conda/virtualenv environments or containers
  • SSH keys, VS Code settings, small local tools
  • Long-lived tokens/config (within reason)

Most of the “my environment got wiped” horror stories come from mixing these two. When you separate them, you can safely recycle GPUs without trashing people’s work.

---

Core building block 1: Persistent, shared storage

Stop tying a researcher’s work to a single VM disk. Use shared, durable storage and mount it into whatever compute you spin up.

Three patterns work well:

1. Network POSIX storage for “live” work

   • Use NFS, Lustre, BeeGFS, or an equivalent managed file service.
   • Every dev environment mounts the researcher’s home at a stable path, e.g. /home/$USER.
   • Jupyter notebooks, small datasets, and source live here.
   • Backed up or snapshotted on a schedule.

2. Object storage for bulk data and artifacts

   • Datasets, model checkpoints, and logs go to S3/GCS/Azure Blob (or S3-compatible).
   • Access via SDK (boto3, gsutil) or rclone/fuse for convenience.
   • Cheaper, more scalable, and easier to share across experiments.

3. Project- or team-level volumes

   • For a lab or project, provision a shared directory or volume.
   • Mount at /proj/$LAB_NAME alongside user homes.
   • This avoids every researcher keeping a private copy of the same 5 TB dataset.

On VESSL AI, this maps to:

• Cluster Storage for POSIX-like, high-throughput shared files across GPU jobs.
• Object Storage for cheaper, large datasets and artifacts you don’t need at SSD speeds.

This combination is what actually makes a Jupyter environment “persistent” in practice.

---

Core building block 2: Containerized environments, not snowflake VMs

If you rely on mutable system packages installed by hand on a single VM, you’ll always fear losing that machine. Instead:

• Standardize on containers

  • Base images: choose CUDA + driver stack that matches your GPUs (e.g., nvidia/cuda:12.1.1-cudnn-runtime-ubuntu22.04).
  • Layer on PyTorch/JAX/TF, then your typical research libs.
  • Maintain a handful of “blessed” images per team or workload.

• Let researchers extend on top

  • Either:
    • Let them pip install into a writable user directory (e.g., ~/.local) that’s on shared storage.
    • Or give them a way to build personal images (e.g., via a CI pipeline or a “build image” job).

• Track versions

  • Tag images by stack: lab-llm:cuda12.1-py310, lab-vision:cuda11.8-py39.
  • Make it clear what image a given Jupyter/SSH environment used, so they can reproduce it later.

Once compute becomes “start container X + mount my home,” losing a node is no longer a disaster.

On VESSL AI:

• You run Jupyter/SSH sessions as jobs using your chosen container image.
• You can reference the same image in dev and training jobs (vessl run), so your dev environment mirrors your production runs.

---

Core building block 3: A control plane that binds identity, storage, and compute

You need a stable identity for each researcher that:

• Owns their storage mounts.
• Controls their Jupyter/SSH sessions.
• Applies consistent policies (quotas, GPU limits).

This can be:

• Kubernetes + an internal portal
• A homegrown gateway that talks to your IaaS
• Or a multi-cloud control plane like VESSL AI

Key requirements:

1. User-level workspaces

   • Each user gets a workspace with:
     • A persistent home directory.
     • Default CPU/GPU quotas.
     • A history of their running and past sessions.

2. One-click or one-command sessions

   • Start/stop Jupyter with a clear mapping: user → container image → mounted storage.
   • Expose SSH endpoints tied to that same environment.

3. Policy and isolation

   • Enforce per-user and per-team GPU limits.
   • Optionally require certain images (e.g., hardened base) for production data.

This is the difference between “I spun up some Jupyter VMs” and “we have persistent GPU dev environments as a first-class resource.”

On VESSL AI:

• Web Console gives each user or team a visual way to launch Jupyter/SSH on A100/H100/H200/B200/GB200/B300.
• CLI (vessl run) lets advanced users script or template their dev environment starts.
• Identity, storage mounts, and GPU allocation are handled in one place across providers.

---

Session lifecycle: how to avoid “my Jupyter died and everything is gone”

Even with persistent storage and containers, you still need to design how sessions behave.

1. Treat dev sessions as long-lived but disposable

• It’s okay if a Jupyter pod restarts or a GPU node is replaced.
• Because:
  • Home directory is on shared storage.
  • Code is in Git.
  • Datasets and checkpoints are in shared or object storage.

2. Add autosave and idle shutdown

To keep costs sane without surprise data loss:

• Autosave in Jupyter is mandatory.
• Regular snapshots/checkpoints of key notebooks or config to Git or object storage.
• Idle timeout for Jupyter/SSH:
  • After X hours of inactivity, shut down the session.
  • Make sure shutdown is graceful so the filesystem flushes.

On VESSL AI, you can:

• Use job-level policies to auto-stop idle sessions.
• Keep the underlying storage intact so when they start a new session, their environment looks the same.

---

Handling GPUs specifically: quotas, preemption, and reliability

Persistent environments on GPUs are where things get tricky. GPU supply is not infinite, and capacity types matter.

Capacity tiers you should support

You want a spectrum:

1. Spot / best-effort GPU capacity

   • Best for: Casual exploration, batch experiments, non-critical Jupyter.
   • Pros: Lowest cost, good for “I’m trying a new idea.”
   • Cons: Preemptions. The node will disappear at some point.
   • What you must have:
     • Auto-checkpointing of notebooks and models to shared storage.
     • Clear separation so users know: “This environment can be interrupted.”

2. On-Demand with failover

   • Best for: Regular day-to-day dev, production-adjacent work.
   • Pros: Reliable capacity with automatic failover to another provider/region if one fails.
   • Cons: Higher cost than Spot, but users can treat it as their “real” environment.
   • What you must have:
     • If an underlying node dies, the session restarts on a new node with the same mounts.
     • Users barely notice, aside from a brief interruption.

3. Reserved capacity

   • Best for: Critical work, classes, or labs that must have GPUs at a specific time.
   • Pros: Guaranteed H100/H200/B200/GB200/B300 inventory; discounts with commitment.
   • Cons: Requires planning and commitments, but protects you from quota ceilings and waitlists.

On VESSL AI:

• Spot: best-effort, up to 90% savings, preemptible. Great for exploratory dev if you’re okay with interruptions.
• On-Demand: reliable with built-in Auto Failover across providers and regions.
• Reserved: lock in A100/H100/H200/B200/GB200/B300 for 3+ months with discounts and dedicated support.

You can map your policy like:

• Default Jupyter/SSH: On-Demand for active labs.
• Cheap scratch work: Spot sessions.
• Courses or critical research deadlines: Reserved capacity for a given lab or project.

---

Practical design: putting it all together

Here’s a concrete pattern that works well in practice.

For each researcher

1. Create a persistent home

   • A directory like /home/$USER on shared storage.
   • Mounted into any dev session they start (Jupyter, SSH).
   • Regular backups and snapshots.

2. Grant a default GPU profile

   • Example: up to 1×A100 or 1×H100 for dev, with a daily/weekly usage guideline.
   • For labs working with bigger models, allow scale to multiple GPUs when requested.

3. Define a default dev image

   • E.g., lab-default:cuda12.1-py310-pytorch2.2.
   • Pre-configured with:
     • JupyterLab
     • sshd (if you allow SSH)
     • Common libs: numpy, pandas, scikit-learn, PyTorch, Transformers, JAX or TF as needed.

4. Expose simple start mechanisms

   • Web: “Start Jupyter” → choose GPU type (A100/H100/H200/B200/GB200/B300), capacity type (Spot/On-Demand/Reserved), and image.
   • CLI: vessl run --env dev --gpu h100 --type on-demand (or your equivalent).

5. Enforce idle shutdown

   • E.g., 8-hour idle timeout.
   • When session ends:
     • Jupyter stops.
     • GPU is freed.
     • Storage persists.

For each lab or team

1. Create a shared project workspace

   • Shared directory: /proj/$LAB_NAME.
   • Group-level quotas and permissions.

2. Provision capacity

   • For heavy labs: consider Reserved blocks of H100/H200/B200 so you don’t fight quotas.
   • For smaller labs: mix Spot and On-Demand across multiple cloud providers.

3. Standardize images and templates

   • “LLM post-training dev env”
   • “Vision dev env”
   • “RL/Physical AI dev env”

4. Monitor usage

   • GPU hours by user and by lab.
   • Spot vs On-Demand vs Reserved consumption.

On VESSL AI, this becomes:

• One control plane to manage workspaces, Cluster Storage/Object Storage, and GPU capacity modes across providers.
• Researchers use either the Web Console or vessl run CLI to start their persistent-feeling environments without touching cloud consoles.

---

Common failure modes—and how to avoid them

1. “We put everything on a root disk that got deleted.”

Fix:

• Move user work to shared storage.
• Treat node-local disks as scratch only.

2. “Our Jupyter host got preempted and we lost hours of work.”

Fix:

• Don’t run critical dev on Spot without auto-checkpointing.
• Use On-Demand or Reserved for main work.
• Ensure autosave and persistent mounts are configured.

3. “Dependencies drift, and we can’t recreate old environments.”

Fix:

• Containerize.
• Tag images by version.
• Store the image tag in your job/session metadata.

4. “Quota walls and waitlists block labs from actually getting GPUs.”

Fix:

• Use a multi-cloud GPU control plane that can pull from multiple providers.
• For key labs, commit to Reserved capacity for H100/H200/B200/GB200/B300 so they have guaranteed access.

---

How this maps to VESSL AI in practice

If you don’t want to build and operate all of this yourself:

• Stop chasing quotas. Start giving labs stable environments.

  • Access A100/H100/H200/B200/GB200/B300 across multiple providers in one place.
  • No single-cloud quota ceilings or waitlists.

• Make dev environments persistent by design.

  • Mount Cluster Storage into every Jupyter/SSH job for POSIX-style home dirs and projects.
  • Use Object Storage for datasets and checkpoints so experiments are reproducible.

• Pick the right reliability tier per use case.

  • Spot: best-effort, cheapest, with auto-checkpointing and clear expectations.
  • On-Demand: reliable with built-in Auto Failover so sessions survive provider outages.
  • Reserved: guaranteed capacity for critical courses, labs, or production workloads, with discounts for 3+ month commitments.

• Let researchers focus on research, not job wrangling.

  • Start Jupyter or SSH in minutes via Web Console or vessl run.
  • Scale from 1 to 100 GPUs as needed for larger experiments.
  • UC Berkeley, MIT, Stanford, and others already use this model; academic discounts are available for universities and research institutes.

---

Final verdict

A good approach to persistent GPU dev environments is to:

1. Decouple state from compute with shared POSIX storage + object storage.
2. Containerize environments instead of hand-tuning pet VMs.
3. Use a control plane that binds identity, storage, and GPU orchestration across providers.
4. Offer Spot, On-Demand, and Reserved GPU tiers so researchers get both cost efficiency and reliability.
5. Enforce sensible timeouts and autosave so shutting down compute never means losing work.

That’s what turns “fragile Jupyter VMs that sometimes disappear” into “persistent GPU dev environments” that researchers can trust—and that operations teams can actually maintain.

Next Step

Get Started