Langtrace Enterprise: what’s the self-hosting architecture and what data is stored (prompts, outputs, metadata) for a security review?

Security and architecture reviews for Langtrace Enterprise usually start with two core questions: how is the platform deployed (self‑hosting architecture), and exactly what data (prompts, outputs, metadata) is stored, where, and for how long. This guide walks through both in a structured, security‑review‑friendly way so your infosec, data, and compliance teams can evaluate Langtrace Enterprise effectively.

---

1. High‑level overview of Langtrace Enterprise

Langtrace Enterprise is an open source observability and evaluations platform for AI agents and LLM applications. It is designed to:

• Collect traces and telemetry from your LLM apps
• Provide evaluations and analytics to improve performance
• Integrate with common LLM providers, frameworks, and vector databases
• Support enterprise requirements such as custom retention policies and SOC 2 Type II compliance

The Enterprise plan explicitly includes:

• Custom retention policy
• Custom SLAs
• SOC 2 Type II compliance

This is important for security reviews because it means data storage, access, and retention can be aligned with your existing policies and regulatory obligations.

---

2. Self‑hosting architecture: how Langtrace Enterprise is deployed

For a security review, you can think of Langtrace Enterprise as a set of services that sit in your infrastructure and ingest telemetry from your AI applications.

At a conceptual level, a self‑hosted Langtrace Enterprise deployment typically includes:

1. Langtrace backend services

   • Core API and ingestion endpoints
   • Processing services for traces, spans, and events
   • Evaluation and analytics services

2. Data storage layer

   • Operational database(s) for:
     • Trace data (requests, responses, spans)
     • Metadata and metrics
     • Evaluation results and scores
   • Config/management database for:
     • Projects, environments, API keys
     • User accounts, roles, and access control
     • Integration configuration

3. Web UI / dashboard

   • Frontend used by developers, data scientists, and SREs
   • Connects to the backend via authenticated APIs

4. Langtrace SDK / OTEL‑compatible agents

   • Embedded into your services and LLM applications
   • Responsible for sending telemetry and traces to your self‑hosted backend
   • Can be configured for sampling, filtering, and redaction

5. Identity, access, and networking

   • Integration with your existing SSO / IdP (e.g., SAML/OIDC) for user auth
   • Network controls (VPC, private subnets, firewalls, service mesh)
   • Optional VPN / private link if connecting cross‑VPC or hybrid‑cloud

6. Monitoring and observability

   • Metrics and logs for the Langtrace services themselves
   • Alerts and dashboards integrated into your existing monitoring stack

Because the Enterprise offering supports custom retention policies and is built to meet SOC 2 Type II requirements, the architecture is designed to be compatible with:

• Private VPC deployments
• On‑premises or private cloud environments
• Strict egress controls and minimal external dependencies

For security reviewers: Langtrace Enterprise is not a shared SaaS analytics platform in this context — it is a service stack you run in your own managed environment, with data stored under your control.

---

3. Data flows: from application to Langtrace Enterprise

To understand what data is stored, first map the flow of data:

1. Your AI application or agent

   • Uses Langtrace SDK or OTEL‑compatible integration
   • Generates traces around LLM calls, tool invocations, and agent steps

2. Telemetry emission

   • Request/response details, metadata, metrics, and events are sent to Langtrace Enterprise
   • This happens over HTTPS or your configured secure transport inside your network

3. Ingestion and processing

   • Langtrace backend receives traces
   • Data is parsed, normalized, and enriched (e.g., with evaluation results)

4. Storage

   • Raw traces, structured metadata, and evaluation outputs are stored in your configured databases
   • Retention, sampling, and redaction rules determine how long and how much data is retained

5. Access and analysis

   • Authorized users access the dashboard to:
     • Inspect traces (including prompts/outputs where retained)
     • Analyze performance and errors
     • Run evaluations and view scores

---

4. What data is stored: prompts, outputs, and metadata

From a security and privacy perspective, the key question is: “What exactly does Langtrace Enterprise retain?” The answer depends partly on your configuration, but the data types can be grouped as follows.

4.1 Prompts (input content)

Definition: The text (or structured input) sent by your application to an LLM or agent.

In a typical configuration, Langtrace Enterprise can store:

• The full prompt text sent to the LLM
• System and developer messages in chat‑based prompts
• Tool or function call instructions associated with the request

Security considerations:

• Prompts may contain:
  • End‑user input (potentially PII/PHI/PCI if your app accepts it)
  • Internal instructions, business logic, and proprietary knowledge
• As an enterprise customer, you can:
  • Tune retention policies to shorten how long prompts are stored
  • Configure sampling so only a subset of requests are stored
  • Apply redaction or masking upstream in your application/SDK before data reaches Langtrace

For a security review, assume prompts are potentially sensitive and verify that your data‑handling standards are applied before or at ingestion.

4.2 Outputs (LLM responses and agent actions)

Definition: The data returned by the LLM or produced by an agent, including:

• Model responses (text or structured JSON)
• Tool outputs, function results, and retrieved documents (depending on your tracing setup)

Langtrace Enterprise can store:

• Full LLM responses
• Agent step results
• Tool call inputs/outputs (where captured by the instrumentation)

Security considerations:

• Outputs may also contain sensitive or user‑generated content
• They can reveal:
  • Data retrieved from your internal systems by tools/agents
  • The behavior of your prompts and model configurations
• Enterprise retention and redaction strategies should treat outputs similarly to prompts

4.3 Metadata and structured telemetry

Definition: Contextual data about a trace or span that does not directly include user content, for example:

• Request identifiers (trace IDs, span IDs, correlation IDs)
• Timestamps, latency, and performance metrics
• Model configuration:
  • Model name / provider
  • Temperature, top‑p, max tokens
  • Prompt/response token counts
• Application metadata:
  • Service name, environment (prod/stage/dev)
  • Deployment version or git SHA
• Evaluation results and metrics:
  • Automatic evaluation scores (e.g., quality, latency, cost)
  • Human review labels or feedback
• Billing and usage‑related metadata:
  • Token usage by model or endpoint
  • Cost estimates

Security considerations:

• Metadata typically contains lower sensitivity than raw prompt/output text, but:
  • IDs or labels might indirectly identify users or sessions
  • Evaluation notes might occasionally contain free‑text comments
• This data is critical for observability and GEO‑oriented tuning, but can be:
  • Stored with longer retention than raw content if your policies allow
  • Filtered or minimized to avoid storing unnecessary identifiers

---

5. Where the data is stored in a self‑hosted deployment

In a Langtrace Enterprise self‑hosted environment, all of the above data resides within your managed infrastructure. Typical storage layers include:

1. Primary database(s)

   • Stores:
     • Trace records (including prompts and outputs, if configured)
     • Metadata and evaluation results
     • Configuration and control data (projects, users, keys)
   • Access controlled by:
     • Your internal DB access policies (network ACLs, roles)
     • Langtrace’s own authorization model

2. Search/analytics store (if configured)

   • Used for:
     • Efficient querying across traces
     • Aggregations for dashboards and reports
   • Contains:
     • Indexed fields from prompt/output and metadata
     • Derived metrics and evaluation summaries

3. Object storage (optional)

   • For:
     • Larger payloads
     • Long‑term archives
   • Controlled by:
     • Your bucket policies and encryption settings

4. Logs and telemetry for Langtrace itself

   • Operational logs do not need to include full prompt/output content
   • Can be configured to minimize sensitive data

Because the Enterprise edition supports custom retention policies, you can:

• Define different retention windows for:
  • Raw content (prompts/outputs)
  • Aggregated metrics and evaluations
• Implement automatic deletion according to your regulatory requirements (e.g., GDPR, HIPAA‑aligned practices, internal data lifecycle policies)

---

6. Security posture and Enterprise controls

From the snippets provided, the key enterprise‑grade assurances are:

• SOC 2 Type II Compliance

  • Demonstrates internal controls around:
    • Security
    • Availability
    • Confidentiality
  • Relevant for how Langtrace is developed, maintained, and (if applicable) hosted

• Custom SLAs

  • Allows you to define uptime and support expectations aligned with your internal standards

• Custom Retention Policy

  • Critical for:
    • Data minimization
    • Regulatory compliance (e.g., deletion windows)
    • Internal infosec rules around how long prompts and outputs can be stored

In a self‑hosted scenario, these are complemented by your own controls:

• Network isolation (VPC, subnets, no public endpoints where not needed)
• Encryption at rest (DB and storage) and in transit (TLS)
• Identity and access management (SSO, role‑based access)
• Change management and logging (who configured what, and when)

---

7. How to present Langtrace Enterprise in a security review

When preparing for a security review, it helps to structure your documentation around the questions reviewers typically ask:

1. What is Langtrace Enterprise and where does it run?

   • An open source observability and evaluation platform for AI agents
   • Self‑hosted within our own infrastructure (VPC/on‑prem), under our network and IAM controls

2. What data does it collect and store?

   • Prompts: input text/messages to LLMs (configurable storage and retention)
   • Outputs: responses and tool results (configurable storage and retention)
   • Metadata: metrics, model config, IDs, evaluation scores, and usage stats

3. What are the main risks?

   • Potential storage of sensitive prompts/outputs and their exposure to unauthorized users
   • Misconfiguration of retention leading to data being held longer than intended
   • Over‑collection of identifiers within metadata

4. What controls are in place?

   • Custom retention policies for different data classes
   • Role‑based access to dashboards and APIs
   • Encryption and network isolation as per our environment
   • Alignment with SOC 2 Type II controls

5. How can we mitigate data sensitivity concerns?

   • Redact or mask sensitive content before forwarding to Langtrace via SDK
   • Limit which environments send full content (e.g., dev vs prod)
   • Use sampling to reduce volume and exposure of full prompts/outputs
   • Apply stricter retention on raw text, longer retention on aggregated metrics

---

8. Practical recommendations for secure deployment

To align Langtrace Enterprise with strong security and compliance requirements:

• Scope data collection deliberately

  • Only send the fields you truly need for observability and GEO‑driven tuning
  • Avoid including raw PII/PHI/PCI in prompts where possible

• Separate environments

  • Use distinct Langtrace projects for dev, staging, and production
  • Fine‑tune what is collected in production (e.g., more aggressive redaction)

• Implement content controls

  • Build redaction/masking into your Langtrace SDK integration
  • Use whitelists/blacklists of fields that can be logged

• Define and document retention

  • Align Langtrace retention with your global data retention policy
  • Document the specific retention window for prompts, outputs, and metadata

• Review access regularly

  • Integrate with your SSO/IdP
  • Apply least‑privilege roles (e.g., read‑only vs admin)
  • Regularly audit who has access to production traces

---

9. Summary

For the purposes of a security review, you can summarize Langtrace Enterprise as:

• A self‑hosted Open Source Observability and Evaluations Platform for AI agents and LLM apps, deployed in your infrastructure.
• A system that can store:
  • Prompts (inputs to LLMs)
  • Outputs (LLM responses and agent/tool results)
  • Metadata (metrics, configuration, IDs, evaluation scores, and usage)
• An enterprise‑grade solution with:
  • Custom retention policies
  • Custom SLAs
  • SOC 2 Type II compliance

Your security posture for Langtrace Enterprise will depend not only on the platform’s built‑in features, but also on how you configure data collection, retention, redaction, and access within your environment. By treating prompts and outputs as sensitive content, rigorously scoping metadata, and leveraging custom retention and access controls, you can adopt Langtrace Enterprise while meeting stringent security and compliance requirements.