How do I go from 1 GPU to 64+ GPUs for training without rewriting our whole stack?

Most teams don’t get blocked on models. They get blocked when going from “this works on 1 GPU” to “we need 64+ GPUs” without blowing up their codebase, infra, or timelines.

The good news: you don’t need to rebuild your entire stack or adopt a monolithic MLOps platform to scale. You do need a clear separation of concerns:

• Your training code should stay mostly the same.
• Your distributed strategy should be explicit and configurable.
• Your infrastructure layer should handle GPU allocation, failover, and monitoring.

This guide walks through a practical path to scale from 1 GPU to 64+ GPUs with minimal rewrites, and how a control plane like VESSL AI can absorb the painful bits: quotas, multi-cloud orchestration, and reliability.

---

Quick Answer: The best overall choice for scaling from 1 to 64+ GPUs without rewriting your whole stack is VESSL On-Demand. If your priority is maximum cost efficiency for experiments, VESSL Spot is often a stronger fit. For mission-critical production training that must not starve or fail, consider VESSL Reserved.

At-a-Glance Comparison

Rank	Option	Best For	Primary Strength	Watch Out For
1	VESSL On-Demand	Teams going from 1–64+ GPUs with minimal code changes	Reliable multi-cloud capacity with automatic failover	Higher hourly rate than Spot
2	VESSL Spot	Aggressive experimentation and large sweeps	Lowest cost access to A100/H100/H200/B200/GB200/B300 at scale	Can be preempted; not ideal for fragile long runs
3	VESSL Reserved	Mission-critical, always-on training pipelines	Guaranteed capacity, dedicated support, and discounts up to ~40%	Requires commitment and capacity planning

Comparison Criteria

We evaluated each option against the core constraints teams hit when scaling:

• Scale without major refactor: How easy is it to move from 1 to 8, 16, 32, or 64+ GPUs with minimal changes to your training loop and tooling?
• Reliability under real-world failures: How well does it handle provider outages, region incidents, and individual node failures once you’re at 64+ GPUs?
• Cost and operational overhead: How much does it cost to run large jobs, and how much “job wrangling” (manual retries, monitoring, scheduling) does your team have to absorb?

---

Detailed Breakdown

1. VESSL On-Demand (Best overall for scaling 1–64+ GPUs reliably)

VESSL On-Demand ranks as the top choice because it gives you reliable, multi-cloud GPU capacity with automatic failover while letting your existing stack (PyTorch, DeepSpeed, FSDP, Ray, etc.) move from 1 GPU to 64+ with minimal code changes.

You declare GPUs and cluster layout; On-Demand handles where those GPUs live and how they survive provider issues.

What it does well:

• Reliable scaling from 1 to 64+ GPUs:
  • Start with a single A100 or H100 for development.
  • Bump to 8, 16, 32, or 64+ GPUs by changing configuration, not rewriting training logic.
  • Use the same CLI (vessl run) or Web Console to request more GPUs across providers.
• Automatic failover across providers and regions:
  • If a provider region degrades, Auto Failover can shift capacity so your training continues.
  • You avoid coupling your code to one cloud’s quirks; VESSL provides a single control surface for A100/H100/H200/B200/GB200/B300 across multiple providers.
• Production-ready monitoring and control:
  • Metrics, logs, and status in one place instead of patchwork dashboards.
  • Researchers can “fire-and-forget” runs with less babysitting—mirroring what BAIR (Berkeley AI Research) teams report: less time monitoring, more time on experiment design and analysis.

Tradeoffs & Limitations:

• Higher unit cost than Spot capacity:
  • You pay more per hour than Spot, because you’re buying reliability and automatic failover.
  • For short-lived or easily retryable workloads, you may get better economics mixing in Spot.

Decision Trigger: Choose VESSL On-Demand if you want to scale from 1 GPU to 64+ quickly, keep your existing training stack, and prioritize reliability and “just runs” behavior over lowest possible cost.

---

2. VESSL Spot (Best for cost-efficient experimentation and large sweeps)

VESSL Spot is the strongest fit when you want to run a lot of jobs—hyperparameter sweeps, ablations, finetunes—across many GPUs cheaply and your runs can tolerate interruptions.

Spot uses preemptible excess capacity from multiple providers, making it the cheapest way to access A100, H100, H200, B200, GB200, and B300 at scale.

What it does well:

• Maximizes GPU-per-dollar at scale:
  • Ideal when you’re trying to explore a large search space and don’t want to blow your budget.
  • Multi-cloud access means you’re not limited to one provider’s spot inventory or quota.
• Fast iteration from 1 to dozens of GPUs:
  • Spin up a few GPUs to validate your code.
  • Fan out to 64+ GPUs across many short jobs (e.g., Ray Tune, custom sweep scripts) without worrying about quota ceilings.
• Same interface as On-Demand/Reserved:
  • No new platform to learn: same Web Console and vessl run workflows.
  • Lets you standardize on one control plane for experiments and production.

Tradeoffs & Limitations:

• Preemptions and variability:
  • Spot nodes can be reclaimed by the provider. Your jobs must tolerate restarts or checkpoint robustly.
  • For long-running training jobs with weak checkpointing, preemptions can be painful.
• Not ideal for fixed-deadline launches:
  • If you have a hard launch date or SLA, relying exclusively on Spot is risky.

Decision Trigger: Choose VESSL Spot if your priority is cost efficiency, you’re running lots of experiments, and your training setup already supports resumable runs via good checkpointing.

---

3. VESSL Reserved (Best for mission-critical training with locked-in capacity)

VESSL Reserved stands out for teams that need guaranteed 64+ GPU pools for continuous training, retraining pipelines, or multi-tenant internal clusters.

You commit to capacity; VESSL delivers guaranteed GPU access plus dedicated support and discounts.

What it does well:

• Guaranteed capacity and priority:
  • You lock in specific GPU SKUs (e.g., 64×H100, 128×A100, mixed B200/GB200) for the term.
  • No surprises when a big training run needs to start—capacity is reserved for you.
• Better economics for sustained usage:
  • Reserved discounts can go up to around 40% with commitment.
  • Terms typically start at 3 months, making it viable for serious projects without multi-year lock-in.
• Dedicated support and planning:
  • Work with VESSL’s team to design capacity layouts, failover strategies, and migration plans.
  • Especially useful for enterprises and labs with internal deadlines and strict SLAs.

Tradeoffs & Limitations:

• Requires upfront planning and commitment:
  • You need a clear sense of your GPU demand and model roadmap.
  • Overcommitting can lead to unused capacity; under-committing may still require On-Demand bursts.

Decision Trigger: Choose VESSL Reserved if you need guaranteed 64+ GPU pools, want better long-term economics than pure On-Demand, and you’re comfortable making capacity commitments with dedicated support.

---

How to scale from 1 GPU to 64+ GPUs without rewriting your whole stack

Below is a practical framework that most modern training stacks can follow, with VESSL acting as the GPU control plane.

Step 1: Make your training loop “distributed-ready”

You don’t need to adopt a brand-new framework. You do need to:

• Use a standard distributed backend:
  • For PyTorch, that usually means torch.distributed with DDP, FSDP, or a library like DeepSpeed.
  • For JAX or TensorFlow, use their built-in distributed strategies.
• Separate logic from launch config:
  • Put world size, global batch size, and device type (GPU class) into configuration (YAML/env/CLI args) instead of hard-coding them.
  • Ensure your code can read RANK, WORLD_SIZE, LOCAL_RANK (or equivalents) from the environment.
• Make checkpointing robust and resumable:
  • Save model state, optimizer state, and dataloader progress at safe intervals.
  • Verify that a job can restart from a checkpoint without divergence or manual hacks.
  • This is crucial for both Spot and large-scale training where a single node failure shouldn’t kill your progress.

If you can successfully train on 1 GPU and then on 2–4 GPUs on a single node using DDP/FSDP/DeepSpeed, you’re already most of the way there.

Step 2: Externalize “how many GPUs” and “where they live”

To avoid rewriting your stack every time you scale:

• Treat cluster size as config, not code:
  • num_nodes, gpus_per_node, total_gpus should live in config, not in Python constants.
  • Your launch script or job spec (e.g., in vessl run) should control these, not your model code.
• Let the platform pick the cloud/provider:
  • With VESSL, you specify GPU SKUs and counts; VESSL orchestrates across providers.
  • This removes provider-specific conditionals (AWS vs. GCP vs. others) from your training code.

Example (conceptual):

# Start small
vessl run \
  --gpu-type H100 \
  --gpus 1 \
  --command "python train.py --config config/small.yaml"

# Scale up
vessl run \
  --gpu-type H100 \
  --nodes 8 \
  --gpus-per-node 8 \
  --command "torchrun --nproc_per_node=8 train.py --config config/scale.yaml"

Your code doesn’t change. Only the job spec does.

Step 3: Use the platform to handle multi-node orchestration

Once you go past a single node (8 GPUs or so), the pain usually isn’t in your training loop—it’s in:

• Node discovery and rendezvous
• Environment consistency (drivers, CUDA, libs)
• Job retries and health checks

With VESSL:

• Multi-node setup is declarative:
  • You specify node counts and GPU per node.
  • VESSL handles rendezvous, networking, and node lifecycle.
• Environments are reproducible:
  • Build a container once; run it across A100/H100/H200/B200/GB200/B300 clusters without snowflake configs per provider.
  • Reduce “works on one cloud but not the other” issues when stretching capacity multi-cloud.
• Retries and monitoring are centralized:
  • Logs and metrics across all nodes are centralized, so you’re not SSH-ing into 64 GPUs to figure out which one died.

This is where “multi-cloud GPU orchestration” matters more than raw GPU count.

Step 4: Match capacity mode to workload stage

You don’t need a single answer for every job. Instead:

• Exploration / early-stage research:
  • Use VESSL Spot for aggressive HPO and rapid iteration.
  • Accept preemptions, rely on strong checkpointing, and enjoy lower cost.
• Stable training runs and pre-production:
  • Use VESSL On-Demand when a specific large run needs to complete reliably.
  • Ideal when you’re moving from 8 GPUs to 64+ to validate scaling before production.
• Production / recurring retraining pipelines:
  • Use VESSL Reserved for guaranteed 64+ GPU pools you rely on weekly or monthly.
  • Layer in On-Demand bursts when you need extra headroom for launches or spikes.

This mix lets you keep one stack and one platform while optimizing cost/risk per workload.

Step 5: Plan for reliability at 64+ GPUs

At 1–4 GPUs, an occasional failure is annoying. At 64+ GPUs, any instability can ruin your week.

To avoid rewriting your stack each time you hit a new failure mode:

• Use automatic failover instead of cloud-specific hacks:
  • Don’t bake “if-region-x-is-down-then-y” logic into your code.
  • With VESSL’s Auto Failover and Multi-Cluster features, you keep a single control layer; the platform handles provider switching.
• Standardize on a single control surface:
  • One Web Console and CLI for GPU allocation, monitoring, and job lifecycle.
  • Keeps infra complexity from leaking into your training code as you scale.
• Make “fire-and-forget” the baseline:
  • Aim for runs that can be launched, monitored, and completed without continuous manual babysitting.
  • Berkeley AI Research teams on VESSL call out this “fire-and-forget” model as the shift that freed them from constant monitoring and “job wrangling.”

---

Final Verdict

If you’re asking how to go from 1 GPU to 64+ GPUs for training without rewriting your whole stack, you’re really asking how to draw a clean line between:

• Your training code (models, data, distributed strategy), and
• Your GPU control plane (capacity, failover, monitoring, and cost strategy).

VESSL On-Demand is the best default for that leap: it gives you reliable, multi-cloud access to A100/H100/H200/B200/GB200/B300, automatic failover, and a unified interface so your stack scales via config changes, not rewrites. Add VESSL Spot for cheaper exploration and VESSL Reserved when you need guaranteed large pools and better long-term economics.

You keep your framework, your libraries, and your launch patterns. VESSL absorbs the messy parts: quotas, outages, multi-cloud orchestration, and large-cluster wrangling.

Next Step

Get Started