How do we use LaunchDarkly AI Configs to change prompts/models at runtime and roll out changes to a segment first?

Shipping AI changes should feel as safe as flipping a feature flag, not like pushing a new model into production at 2am and hoping for the best. AI Configs in LaunchDarkly give you runtime control over prompts, models, and agents—so you can change behavior after deploy, start with a small segment, and recover instantly if something breaks.

Quick Answer: LaunchDarkly AI Configs let you define prompts, models, and parameters as controlled configurations, evaluated at runtime. You can target specific segments, run progressive rollouts, and roll back instantly—without redeploying code.

The Quick Overview

What It Is: AI Configs are a runtime control layer for AI behavior—your prompts, model choices, parameters, and agent graphs—managed and evaluated via LaunchDarkly instead of hard-coding them into your app.
Who It Is For: Teams shipping AI-powered features (chatbots, copilots, recommendation systems, internal tools) that need to safely test changes, control blast radius, and prove impact in production.
Core Problem Solved: You no longer have to ship a code deploy every time you tweak a prompt or swap a model, and you don’t have to expose new AI behavior to 100% of traffic at once.

How It Works

AI Configs treat AI behavior as configuration, not code. Instead of embedding prompts and model IDs directly into your application logic, you reference an AI Config through a LaunchDarkly SDK. At runtime, LaunchDarkly:

Evaluates which AI Config (or version of a config) each request should use.
Applies targeting rules (segments, attributes, environment).
Supports progressive rollouts and guardrails.
Returns the selected prompt/model/parameters to your app in sub-200ms worldwide—no redeploys required.

Then:

Define AI behavior as a config:
In LaunchDarkly, you create an AI Config that holds:
- Base prompt or system message
- Model and version (e.g., gpt-4.x vs a cheaper model)
- Parameters like temperature, max tokens, top_p
- Optional tool/agent setup or agent graphs
Wire AI Configs into your app at runtime:
In your backend or service, you call LaunchDarkly via one of the 25+ native SDKs. Instead of hard-coding the prompt/model, you:
- Resolve the AI Config for the current user or request
- Use the returned values to call your LLM provider
- Log and observe behavior using LaunchDarkly’s observability and AI Configs’ LLM observability features
Target, roll out, and iterate on changes:
Once your app is wired, you can:
- Edit prompts and models in the LaunchDarkly UI or via API
- Roll changes out to specific segments first (e.g., internal users, beta cohort)
- Use percentage rollouts and guardrails
- Roll back to the last known-good configuration instantly if metrics degrade

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Runtime AI Config evaluation	Resolves prompts, models, and parameters at runtime through LaunchDarkly SDKs.	Change AI behavior after deploy without touching the codebase.
Targeting, segments, and progressive rollouts	Applies rules (segments, attributes, percent rollouts) to decide which config each request gets.	Start with a small segment, control blast radius, and scale up safely.
Guardrails, observability, and rollback	Uses guardrails, online evals, LLM observability, and approvals to monitor and reverse risky changes.	Detect regressions quickly and flip a kill switch or roll back with no redeploys.

Ideal Use Cases

Best for rolling out a new prompt to power users first: Because you can target a specific “beta testers” or “internal-users” segment, monitor quality, and only then expand to everyone.
Best for testing a cheaper or faster model in production: Because you can expose a new model to, say, 5% of traffic, compare quality and latency with online evals, and roll forward or back instantly based on results.

Limitations & Considerations

You still need good prompts and evaluation criteria: AI Configs make changes safe and reversible, but they don’t automatically generate good prompts or metrics. Pair them with clear quality definitions (e.g., “helpfulness,” “factuality,” latency, error rates).
Instrumentation is required for full value: To benefit from guardrails and LLM observability, you need logs/metrics wired into LaunchDarkly’s observability and AI Config views. Plan instrumentation when you integrate, not after an incident.

Pricing & Plans

AI Configs are part of LaunchDarkly’s broader runtime control platform that includes feature flags, experimentation, and observability. Pricing is typically based on factors like:

Number of seats/users
Environments and projects
Evaluation volume (e.g., flag and AI Config evals)
Required enterprise governance (custom roles, audit logs, policies)

Specific packaging for AI Configs can vary by plan.

Growth or Team-level plans: Best for product and engineering teams getting started with AI-powered features and needing safe runtime control for prompts/models.
Enterprise plans: Best for organizations with multiple squads, compliance needs, and high-volume AI traffic that require custom roles, approvals, policies, and detailed audit trails.

For exact pricing and availability of AI Configs in your region and plan, it’s best to talk to LaunchDarkly sales.

Frequently Asked Questions

How do we change prompts or models at runtime using AI Configs?

Short Answer: You define the prompt/model in an AI Config, reference it from your app via an SDK, and then edit the config in LaunchDarkly—changes apply in production in under 200ms worldwide without redeploys.

Details:
Here’s the typical workflow:

Create an AI Config in LaunchDarkly:
- Go to your project/environment.
- Create a new AI Config (often scoped to a feature like support_assistant_config).
- Add:
  - System / base prompt
  - Model ID (e.g., your selected LLM provider/model)
  - Parameters (temperature, max tokens, top_p, etc.)
  - Any optional metadata/tools/agents

Integrate AI Configs with your app:

Use a LaunchDarkly SDK in your backend (Node, Python, Java, .NET, etc.).
At runtime, resolve the AI Config for the current context (user, account, or request).
Use the resolved configuration to call your LLM provider.

Pseudocode pattern (language-agnostic):

// Initialize LaunchDarkly SDK once at startup
ldClient = initLDClient(SDK_KEY)

// For each AI request
context = {
  kind: "user",
  key: userId,
  plan: userPlan,
  betaTester: userInBeta
}

aiConfig = ldClient.aiConfig("support_assistant_config", context)

// aiConfig might contain:
// {
//   prompt: "...",
//   model: "gpt-4.1",
//   temperature: 0.4,
//   maxTokens: 1024
// }

response = callLLM(
  model = aiConfig.model,
  prompt = aiConfig.prompt,
  temperature = aiConfig.temperature,
  maxTokens = aiConfig.maxTokens
)

Change prompts/models without redeploy:
- When you want to iterate:
  - Edit the prompt copy in the AI Config.
  - Swap to a new model or adjust parameters.
  - Optionally create new versions for testing.
- Once you save, LaunchDarkly pushes changes globally in <200ms.
- Every new request now uses the updated config, respecting your targeting rules.
Govern changes with approvals and audit logs:
- Use policies, approvals, and custom roles so not everyone can modify AI behavior in production.
- Every change is recorded with an audit trail: who changed what, when, and why.

This keeps prompts and models out of your deployed code, while still letting PMs, designers, or AI specialists iterate within guardrails—exactly like feature flags.

How do we roll out a new prompt or model to a segment first?

Short Answer: Create a new version of your AI Config, add targeting rules (segment-based and/or percentage rollout), and send only that segment to the new config. Expand gradually once metrics and user feedback look good.

Details:
Think of this like a guarded rollout for AI behavior:

Define or import your segments:
- In LaunchDarkly, create segments such as:
  - internal-users
  - ai-beta-testers
  - high-value-accounts
- Segments can be built from attributes (email domain, account tier, geography, etc.) or imported from your CDP/identity systems.
Create a new AI Config variation:
- In the AI Config for your feature, define:
  - Baseline config: current prompt + model (the “control”).
  - New config: updated prompt and/or new model (the “candidate”).
- You can also leverage AI Experiments/AI Versioning to treat this as a structured experiment.
Add targeting rules to send only a segment to the new config: Example rule structure:
- If user is in segment ai-beta-testers
  → Serve New AI Config (updated prompt / model).
- Else
  → Serve Baseline AI Config.
Or, for a more gradual rollout:
- If user is in segment ai-beta-testers
  → Roll out New AI Config to 10% of that segment (progressive rollout).
- Else
  → Serve baseline.
This lets you:
- Start with internal users only.
- Move to a beta cohort.
- Then expand to 10%, 25%, 50%, and eventually 100% of production.
Observe performance and quality:
- Use AI Configs’ LLM observability to watch:
  - Latency
  - Error rates
  - Quality signals (from online evals or human feedback)
- Optionally wire in:
  - Online evals (LLM-as-a-judge) for quality scores
  - Business metrics (CSAT, resolution rate, click-through, etc.)
Guarded releases and rollbacks:
- Set guardrails: thresholds for latency, errors, or quality scores.
- If the new prompt/model underperforms:
  - Automatically pause or roll back exposure.
  - Or flip the kill switch manually in LaunchDarkly.
- No redeploy. No reverting code. The runtime config just points traffic back to the last healthy configuration.
Scale up once it’s stable:
- Increase the percentage of the segment using the new config.
- Eventually drop the baseline or keep it in reserve as a quick rollback option.

This is the same progressive delivery pattern you already use for features, now applied to AI behavior.

Summary

AI Configs turn prompts, models, and agent behavior into controlled, observable, and reversible runtime configurations. You integrate once with an SDK, then:

Change prompts and models after deploy.
Target specific segments and run progressive rollouts.
Monitor behavior with LLM observability and online evals.
Roll back to a healthy configuration in seconds if something breaks.

You move at AI speed—but you stay in control.

Next Step

Get Started