Langtrace Enterprise: what’s the self-hosting architecture and what data is stored (prompts, outputs, metadata) for a security review?

For an enterprise security review, it helps to understand both how Langtrace Enterprise is deployed in a self-hosted environment and exactly what data flows through the system (prompts, outputs, and metadata). This guide walks through a typical self-hosting architecture, the data model, and key security considerations so you can evaluate Langtrace against your internal policies.

Overview of Langtrace Enterprise for self-hosting

Langtrace Enterprise is an open source observability and evaluations platform designed for AI agents and LLM-powered applications. In a self-hosted setup, your organization runs Langtrace within its own infrastructure (cloud account or on‑prem), giving you full control over:

Network boundaries and access controls
Data residency and storage policies
Retention, backup, and deletion practices
Integration with your internal identity and logging systems

The core idea is simple: your LLM apps send telemetry (traces, spans, evaluations) to Langtrace using the SDK or OpenTelemetry (OTEL)-compatible endpoints, and Langtrace provides an observability dashboard to analyze performance, quality, and behavior of your AI agents.

High-level self-hosting architecture

While exact topology can vary by organization, a typical Langtrace Enterprise self-hosted architecture includes:

1. Langtrace backend services

Deployed in your infrastructure (e.g., Kubernetes, VMs, or containers), the backend services are responsible for:

Receiving telemetry and evaluation data from your apps
Processing and storing traces, spans, and metrics
Powering the dashboards and analytics used by your teams

Key components usually include:

API / ingestion service
- Receives events from Langtrace SDKs and OTEL-compatible exporters
- Performs basic validation, authentication/authorization, and rate limiting
- Normalizes events into a common observability schema (e.g., spans with attributes and metadata)
Processing / worker services
- Enrich events with derived fields (e.g., latency, success/failure flags)
- Optionally run evaluation jobs (e.g., quality checks, scoring prompts vs. outputs)
- Handle batching, deduplication, and asynchronous tasks
Query / analytics service
- Exposes APIs for dashboards and ad‑hoc queries
- Aggregates data for charts (latency distributions, error rates, model comparisons, etc.)
- Drives the Langtrace UI that your teams use day‑to‑day

2. Storage layer

In a self-hosted Langtrace Enterprise deployment, all persisted data resides in storage systems that you control. Common layers include:

Primary database (SQL or NoSQL, depending on deployment)
- Stores structured objects such as:
  - Projects, environments, and configuration
  - Traces and spans metadata
  - Evaluation results and scoring
  - User accounts and access policies (if managed internally)
Observability/log store
- Optimized for high‑volume, time‑series, or event data
- Retains:
  - Raw or normalized spans (per LLM call or agent step)
  - Associated prompt/output payloads if configured
  - Metadata (e.g., response time, status, model, cost estimates)
Optional object storage
- For large payloads, attachments, or exported datasets
- Can be configured to obey specific lifecycle policies and encryption standards

Because Langtrace Enterprise is self-hosted, you can enforce:

Encryption at rest (e.g., KMS-backed keys)
Data residency per region
Custom retention windows for different data classes

3. Langtrace UI (web dashboard)

The UI is a web application served from your environment that connects to the Langtrace backend. It allows your team to:

Inspect traces and spans from AI agents
View and compare prompts, outputs, and evaluation scores
Configure integrations, alerts, and experiments

This UI runs entirely against your backend; no telemetry needs to leave your controlled environment for analysis.

4. Integrations and SDKs

Your AI applications interact with Langtrace via:

Langtrace SDK
- Typically added with just a few lines of code
- Automatically instruments your LLM calls, agents, and tool use
- Sends telemetry data (including prompts/outputs if you choose) to your Langtrace endpoint
OTEL-compatible exporters
- If you already use OpenTelemetry, Langtrace can act as an OTEL-compatible observability backend
- This lets you integrate AI observability into existing pipelines while still keeping data local

What data is stored: prompts, outputs, and metadata

A security review usually focuses on exactly what Langtrace may capture and store. In a self-hosted Langtrace Enterprise deployment, the data persisted is entirely under your control, and you can configure what is captured or redacted.

Below is a breakdown of typical categories.

1. Prompts

What they are:
Text or structured content you send to your LLMs (user messages, system instructions, or agent context).

How prompts are handled:

Captured as part of spans/traces that represent each LLM call or agent step
May include:
- User queries or conversations
- System / developer messages
- Tool instructions and intermediate reasoning (depending on your agent implementation)

Storage behavior:

By default, prompts can be stored so you can:
- Debug agent behavior
- Evaluate prompt quality and performance
- Run regression or A/B analyses on prompt changes
In enterprise environments, you may choose to:
- Redact sensitive fields before sending to Langtrace via the SDK
- Disable full prompt logging and store only derived metadata (e.g., length, category)
- Configure custom retention policies per project or environment so raw prompts are deleted after a defined period

2. Outputs (LLM responses)

What they are:
The responses generated by your LLM, including any formatted text, JSON, or tool call summaries returned to your application.

How outputs are handled:

Captured alongside prompts in each span so you can:
- Evaluate quality (e.g., correctness, safety, adherence to style)
- Inspect failures or low-quality responses
- Compare models or prompt versions

Storage behavior:

Outputs can be stored as part of the trace data in your observability store
You can:
- Mask or truncate outputs for sensitive scenarios
- Log only metadata (e.g., success/failure, category) while omitting full text
- Use custom retention policies so outputs are not kept longer than required for debugging, evaluation, or compliance

3. Metadata (attributes and metrics)

Metadata is often the most valuable aspect for observability and is typically less sensitive than raw prompts/outputs. Langtrace Enterprise collects rich metadata for each span and trace.

Common metadata fields include:

Request-level metadata
- Timestamp and duration (latency)
- Status (success, error, timeout, validation failure, etc.)
- Request IDs, trace IDs, span IDs
- Retry counts and fallbacks used
Model / provider metadata
- Model name and version (e.g., gpt-4.1, claude-3-opus)
- Provider (OpenAI, Anthropic, internal model, etc.)
- Temperature, max_tokens, and other generation parameters
- Token usage estimates (prompt tokens, completion tokens, total cost)
Application metadata
- Service or microservice name
- Environment (prod, staging, dev)
- Endpoint or feature flag identifiers
- Agent or workflow identifiers (e.g., “support_bot”, “retrieval_chain”)
Evaluation metadata
- Scores assigned by automated evaluators or human raters
- Evaluation dimensions (e.g., factuality, safety, relevance)
- Pass/fail flags for policy checks or guardrails
Business / domain metadata (custom attributes)
- Tenant or customer IDs (can be hashed/pseudonymized)
- Use case tags (e.g., “claims_processing”, “KYC”, “internal_tooling”)
- Segment or region tags for analytics

Because metadata is flexible, you can control:

Which identifiers are included (or anonymized)
Whether to store direct PII or only hashed keys
How long metadata is retained compared to full prompts/outputs

Data flow: from your app to Langtrace Enterprise

Understanding the data path is key for security and compliance.

Your AI app makes an LLM call
- The request includes a prompt, optional context (like retrieval results), and model parameters.
Langtrace SDK or OTEL instrumentation captures telemetry
- It wraps the LLM call in a span and attaches:
  - Prompts (optional or configurable)
  - Outputs (optional or configurable)
  - Metadata (latency, model, tokens, etc.)
Telemetry is sent to your Langtrace backend
- Via secure HTTPS within your network or VPN
- Authenticated using API keys, service accounts, or internal auth systems
- Can flow through internal API gateways or service meshes, according to your standards
Langtrace backend processes and stores the data
- Normalizes spans and traces
- Applies sampling or filtering, if configured
- Writes data to your chosen database and observability storage
Your teams access the Langtrace UI
- They query traces, metrics, and evaluations to debug and improve AI agents
- Access can be restricted by role, environment, or project to enforce least privilege

At no point in a self-hosted deployment is telemetry required to leave your controlled environment (such as your cloud account or on‑premises network), unless you intentionally configure external exports.

Security and compliance considerations for Langtrace Enterprise

For a thorough security review of a Langtrace Enterprise self-hosted deployment, consider the following areas.

1. Data residency and ownership

All data (prompts, outputs, metadata) is stored in systems you manage
You control:
- Region and availability zone
- Backup and disaster recovery strategies
- Cross-region replication (if any)

This supports compliance with data residency rules, internal policies, and industry frameworks.

2. Access control and identity

Integrate Langtrace with your existing identity provider or SSO solution (e.g., SAML, OIDC)
Use role-based access control (RBAC) to:
- Restrict who can view raw prompts/outputs
- Limit production environment visibility to specific teams
- Separate duties between developers, data scientists, and security/compliance teams

3. Network security

Deploy within private subnets or behind your internal gateways
Enforce TLS for all connections to Langtrace APIs and UI
Optionally restrict ingestion endpoints to specific services or IP ranges

4. Data minimization and masking

Configure what is sent to Langtrace at the SDK layer:
- Remove or hash PII and sensitive identifiers
- Avoid sending full documents or attachments unless needed for debugging
- Use structured payloads so you can systematically redact certain fields
Combine SDK-level filtering with backend policies where appropriate (e.g., sampling, truncation).

5. Retention and deletion

With Langtrace Enterprise, you can define custom retention policies, such as:

Short-lived retention (e.g., 7–30 days) for full prompts/outputs
Longer retention for anonymized or aggregated metadata
Environment-specific policies (shorter in production, longer in staging/testing)

This helps align with regulations like GDPR, or internal standards for log and telemetry retention.

6. Auditability and observability of Langtrace itself

Capture audit logs of:
- Who accessed which projects or traces
- Configuration changes (e.g., new integrations, updated retention rules)
Integrate Langtrace’s own logs with your SIEM or security monitoring tools
For Enterprise customers, Langtrace offers SOC 2 Type II compliance, giving you assurance around operational controls and security posture.

How this fits into a security review

When presenting Langtrace Enterprise to your security, risk, or compliance teams, it’s useful to frame it in terms they already know:

It functions like an observability/logging platform for AI agents, compatible with OTEL, but self-hosted.
All sensitive data (prompts, outputs, metadata) can be:
- Confined to your own infrastructure
- Minimized and redacted at the point of collection
- Subject to tailored retention and access policies

A typical security review will cover:

Architecture diagram
- Showing where Langtrace sits in your environment, how data flows, and where it is stored.
Data classification
- Identifying which data types may contain PII or confidential information and how they’re treated (prompt/output vs. metadata).
Controls and configurations
- Access control, network security, encryption, retention, and masking configurations in your self-hosted deployment.
Vendor and platform assurances
- Leveraging Langtrace’s SOC 2 Type II compliance and open source transparency where relevant to your review process.

Next steps

To complete your evaluation and tailor Langtrace Enterprise to your organization:

Review the official documentation for deployment and configuration details
Set up a self-hosted proof-of-concept in a non‑production environment
Define and test:
- What telemetry you send (and redact)
- Retention and masking rules
- Role-based access to sensitive traces

If you need deeper technical details or architecture validation, you can book a demo or connect with the Langtrace team and community via the documentation and Discord links provided on the Langtrace website.