How do I deploy SambaNova on-prem for data residency—what’s the process to schedule a demo and start an evaluation?

On-prem deployment for data residency starts with one decision: keep your AI workloads physically and operationally inside your own data centers, while still getting frontier-scale performance. With SambaNova, that means bringing SambaRack systems and the full SambaStack inference stack into your environment, then evaluating them against your latency, throughput, and compliance requirements in a structured trial.

Quick Answer: You deploy SambaNova on‑prem for data residency by working with our team to scope your data-center environment, selecting the right SambaRack configuration, and installing the full inference stack behind your firewall. To schedule a demo and start an evaluation, submit the contact form on SambaNova’s site, align on a use case and success metrics with our solution engineers, and run a time‑boxed POC or pilot using your data and workflows.

The Quick Overview

What It Is: An on‑prem, rack‑ready AI inference solution—SambaRack SN40L‑16 and SambaRack SN50—powered by Reconfigurable Dataflow Unit (RDU) chips and the SambaStack inference stack, deployed entirely in your data centers for strict data residency.
Who It Is For: Platform, infra, and security teams that must keep models and data local (financial services, public sector, healthcare, telco, and sovereign AI environments) but still need high‑throughput, multi‑model agentic inference.
Core Problem Solved: Eliminates “one‑model‑per‑node” sprawl and cloud‑only constraints by running bundled frontier-scale models on a single node, within your own racks, with data never leaving your controlled environment.

How It Works

On‑prem SambaNova deployments bring the same stack used in cloud–hosted SambaCloud into your data center: RDU‑based SambaRack hardware, the SambaStack inference layer that supports model bundling and multi‑model workflows, and SambaOrchestrator as the control plane for production operations.

At a high level, the deployment cycle looks like this:

Discovery & Design:
You and SambaNova’s team define workloads (e.g., agentic RAG, reasoning loops, multi‑model pipelines), data residency requirements, and data-center constraints (power, cooling, networking). This drives hardware sizing (SN40L‑16 vs SN50, node counts) and rack layout.
Rack Delivery & Installation:
SambaRack systems are delivered as data‑center‑ready racks. Your facilities team provides power, cooling, and network connectivity; SambaNova works with you (and partners as needed) to bring the rack online and integrate identity, networking, and observability into your environment.
Inference Stack Configuration & Evaluation:
SambaStack and SambaOrchestrator are configured to expose OpenAI‑compatible APIs inside your environment. You port existing applications in minutes, bring your own checkpoints where needed, and run a structured evaluation against your latency, throughput, and compliance targets.

1. Discovery & Design

This phase makes sure your on‑prem deployment matches both your workloads and your compliance envelope:

Workload characterization:
- Agentic workflows (multi‑step, multi‑model toolcalling).
- High‑throughput chat or copilots for internal users.
- Sovereign inference where data must stay inside a specific country or facility.
Data residency and compliance mapping:
- Clarify jurisdictions (e.g., EU, UK, specific national regulations).
- Document data classes (PII, PHI, financial records) and what must remain on‑prem.
- Identify logging and monitoring requirements (what can/cannot be sent off‑box).
Infrastructure constraints:
- Power and cooling budget per rack (e.g., low‑power inference targets ~10 kWh for SN40L‑16).
- Network topology (spine/leaf, L2/L3 boundaries, bandwidth between app tiers and SambaRack).
- Existing observability stack (Prometheus, Grafana, Splunk, etc.) and identity (LDAP, SSO).

Together with SambaNova, you turn these into a proposed deployment BOM: number of racks, choice of SN40L‑16 vs SN50, and a plan for how models will be bundled and served.

2. Rack Delivery & Installation

Once the design is locked:

Hardware arrives as rack‑ready systems:
SambaRack is a complete, standards‑compliant data‑center rack with RDUs, networking, and management infrastructure integrated. Your team treats it as a set of known power/network endpoints to connect.
Data-center integration:
- Connect power and cooling per agreed envelope.
- Connect top‑of‑rack switches to your core network.
- Establish management network access, out‑of‑band if required.
Security and access control:
- Restrict management access to your internal admin networks.
- Integrate authentication/authorization as required by your security policy.
- Define separation between environments (prod vs non‑prod) at the network and control-plane levels.

At this point, you have the physical foundation for on‑prem inference that adheres to your data residency rules—models and data live inside your facility.

3. Inference Stack Configuration & Evaluation

With the rack live, you move into software and evaluation:

SambaStack setup:
- Install and configure the inference stack tuned for RDUs: dataflow execution plus three‑tier memory to keep models and prompts hot.
- Enable model bundling so multiple frontier‑scale models can coexist on the same node and switch quickly mid‑agent loop.
SambaOrchestrator configuration:
- Turn on core control‑plane capabilities:
  Auto Scaling | Load Balancing | Monitoring | Model Management | Cloud Create | Server Management
- Integrate metrics and logs with your observability stack to maintain visibility while keeping data on‑prem.
API exposure and app integration:
- Expose OpenAI‑compatible APIs internally so existing apps can call /chat/completions and related endpoints without code rewrites.
- Port your application in minutes by pointing your existing OpenAI clients at the SambaNova on‑prem endpoint.
Evaluation execution:
- Run your target workloads with real prompts (and, where allowed, real data).
- Measure tokens per second, end‑to‑end latency, and tokens per watt against your SLAs.
- Validate that no data leaves your environment, and confirm logging/telemetry behavior with your security team.

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
On‑Prem SambaRack Deployment	Delivers SN40L‑16 and SN50 racks directly into your data center with integrated RDUs, networking, and management.	Keeps all inference, models, and prompts inside your facility for strict data residency and sovereignty.
Model Bundling on RDUs	Runs and switches between multiple frontier‑scale models on a single node using dataflow execution and three‑tier memory.	Eliminates “one‑model‑per‑node” fragmentation, enabling complex agentic workflows with lower latency and better utilization.
OpenAI‑Compatible APIs (On‑Prem)	Exposes APIs that mirror OpenAI endpoints for chat, completion, and more.	Lets you port existing applications in minutes, minimizing migration risk and engineering effort.
SambaOrchestrator Control Plane	Provides autoscaling, load balancing, monitoring, and model lifecycle management for on‑prem inference.	Gives platform teams data-center-grade control over multi‑model workloads without building orchestration from scratch.
Sovereign AI Alignment	Deploys within your jurisdiction and/or specific partner data centers aligned with regional requirements.	Satisfies regulatory and data residency mandates while retaining open-source model flexibility.
Bring Your Own Checkpoints	Supports integrating your own fine‑tuned models and checkpoints into SambaStack.	Lets you keep proprietary weights on‑prem alongside bundled frontier models, protecting IP and sensitive training data.

Ideal Use Cases

Best for Sovereign and Regulated AI:
Because it keeps models, prompts, and intermediate agent states physically within your data centers—no traffic to external inference endpoints—while still delivering frontier-level performance.
Best for Multi‑Model Agentic Workloads at Scale:
Because SambaStack and RDUs support model bundling and fast switching, so you can run reasoning models, tool routers, and domain specialists together on one node instead of stitching across multiple single‑model servers.

Limitations & Considerations

Data-center Readiness:
On‑prem SambaRack deployments assume you have sufficient power, cooling, and networking capacity. If you’re constrained, work with SambaNova early to right‑size (for example, prioritizing SN40L‑16 for low‑power inference) or to phase deployment across multiple racks.
Operational Ownership:
While SambaOrchestrator simplifies orchestration, your team owns day‑to‑day operations—change management, access control, and integration with existing SRE processes. Plan for runbooks, monitoring dashboards, and clear ownership before going to production.

Pricing & Plans

SambaNova does not publish fixed “self‑serve” pricing for on‑prem deployments because each environment differs in:

Rack counts and configuration (SN40L‑16 vs SN50, node density).
Workload profile (steady‑state vs spiky, interactive vs batch).
Regulatory and support requirements (sovereign AI, 24/7 support, SLAs).

In general, you can expect:

SambaRack SN40L‑16 deployments: Optimized for low‑power inference (average of 10 kWh) and suitable for high‑efficiency, steady workloads and cost‑sensitive environments.
SambaRack SN50 deployments: Designed for fast agentic inference at a fraction of the cost on the largest models, ideal when you are pushing token throughput and complex multi‑model workflows.

Pricing, support tiers, and evaluation terms are finalized during the discovery phase.

Evaluation / POC Engagement: Best for teams needing to validate performance, data residency, and integration with a constrained workload over weeks rather than months.
Production Deployment Agreement: Best for teams ready to standardize on SambaNova for on‑prem and/or sovereign AI, with defined SLAs and long‑term capacity planning.

How to Schedule a Demo and Start an Evaluation

From a practical standpoint, here’s the step‑by‑step process most teams follow:

Submit the Contact Form
- Go to https://sambanova.ai/contact.
- Provide your organization details, role, and specify that you’re interested in “on‑prem deployment for data residency / sovereign AI.”
- Include a short description of your workloads (e.g., “agentic RAG with financial documents, must remain in‑country”) and your approximate timeline.
Initial Scoping Call (30–60 minutes)
- Meet with SambaNova solution engineers and account team.
- Clarify data residency and regulatory constraints, target regions, and internal stakeholders (platform, security, compliance).
- Share high‑level traffic and performance expectations (RPS, prompts per day, current latency).
- Agree on whether to start with a cloud‑based trial (SambaCloud) while on‑prem hardware is being prepared, if that fits your policy.
Evaluation Design & Proposal
- Define 1–3 priority use cases and success metrics (e.g., latency < X ms at P95, tokens/sec per model, cost and power envelopes).
- Align on evaluation duration (commonly 4–8 weeks) and what will be validated: model performance, throughput, agentic workflow behavior, data residency guarantees.
- SambaNova provides a proposal covering hardware configuration, software stack, and evaluation structure.
Procurement & Deployment Planning
- Your team completes internal approvals (security, architecture review, procurement).
- Lock in data-center details: rack locations, power circuits, networking, and access controls.
- Define joint project plan: delivery dates, install windows, and responsible owners on both sides.
Rack Installation & Stack Bring‑Up
- SambaRack is delivered and installed according to the plan.
- SambaStack and SambaOrchestrator are configured and integrated with your network, identity, and monitoring.
- OpenAI‑compatible endpoints are made available to your internal test environments.
Hands‑On Evaluation / POC
- Your engineers port existing OpenAI‑based apps by changing endpoint configuration and keys.
- You run your workloads, monitor performance, and validate data never leaves your environment.
- Jointly review performance and operational metrics at regular checkpoints with SambaNova.
Production Decision & Scale‑Up
- If evaluation criteria are met, you transition to a production agreement.
- Scale hardware and models as needed; formalize SLAs and support processes.
- Optionally, expand to additional data centers or sovereign partners in other regions.

Frequently Asked Questions

Can I guarantee that no customer data leaves my own data center?

Short Answer: Yes—an on‑prem SambaRack deployment keeps inference, models, and prompts within your own data center boundaries.

Details:
In an on‑prem deployment, SambaRack resides in your racks, connected only to your internal networks. SambaStack and SambaOrchestrator are deployed within that environment, and OpenAI‑compatible APIs are exposed as internal endpoints. You control:

Network egress rules (e.g., disallow outbound connections from inference subnets).
Logging and telemetry destinations (on‑prem observability tools).
Identity and access controls for both operators and applications.

This architecture is specifically suited for data residency and sovereign AI requirements where regulators or internal policy forbid sending data to external inference providers.

How long does it take to go from demo to on‑prem evaluation?

Short Answer: Expect weeks for a structured evaluation, rather than the 12–18 month timelines common for traditional AI infrastructure rollouts.

Details:
Timelines vary by organization, but typical stages look like:

Demo & initial scoping: Days to a couple of weeks, depending on stakeholder availability.
Evaluation design & approvals: Often 2–4 weeks as architecture, security, and procurement sign off.
Rack delivery & installation: Physical lead time plus your data-center scheduling; the bring‑up itself is measured in days once the rack is on-site.
Evaluation / POC: Commonly 4–8 weeks of hands‑on testing with your workloads.

Because SambaNova delivers an integrated stack—hardware, inference stack, and orchestration—most of the integration time is spent on your internal processes (approvals, change windows) rather than assembling components.

Summary

Deploying SambaNova on‑prem for data residency is a straightforward path for teams that want frontier‑scale, multi‑model, agentic inference without sending data off‑prem. SambaRack brings RDUs and the full inference stack into your data center; SambaStack and SambaOrchestrator handle model bundling, autoscaling, and monitoring; and OpenAI‑compatible APIs let you port applications in minutes. The deployment workflow—discovery, rack installation, stack configuration, and structured evaluation—is designed to validate performance, compliance, and operational fit in weeks, not years.

Next Step

Get Started