How do we use LaunchDarkly AI Configs to change prompts/models at runtime and roll out changes to a segment first?

Fast AI changes are only useful if they’re also safe. AI Configs in LaunchDarkly give you both: you can change prompts, models, and parameters at runtime and progressively roll those changes to a specific segment before you expose them to everyone—no redeploys required.

Quick Answer: AI Configs let you define prompts, models, and parameters as runtime-managed configurations, then target those configurations to specific user segments or percentages. You can safely ship new AI behavior to a small audience first, observe impact, and roll forward or back instantly.

The Quick Overview

What It Is: AI Configs are LaunchDarkly’s runtime control surface for AI behavior—a prompt, model, and agent manager that evaluates at request time via SDKs and APIs.
Who It Is For: Engineering, product, and AI/ML teams that need to ship AI features, change them after deploy, and minimize blast radius when something behaves unexpectedly.
Core Problem Solved: AI behavior changes (new prompts, models, tools, or agents) normally require risky releases or ad hoc config changes. AI Configs centralize that behavior behind feature-flag-style controls so you can target, experiment, and recover instantly.

How It Works

You treat AI behavior as a runtime configuration, not as hard-coded strings buried in your services. Your application calls LaunchDarkly to resolve an AI Config; LaunchDarkly evaluates rules (targeting, segments, rollout percentages, guardrails) and returns the right combination of prompt, model, and parameters for that request.

From there, your app uses those values to call your LLM provider or internal model. When you want to change behavior, you update the AI Config in LaunchDarkly (UI, API, CLI, or MCP integration), not the code. Changes propagate globally in under 200ms.

Typical flow:

Define your AI Config:
Create an AI Config with prompts, models, tools/agents, and parameters as versioned, editable values.
Wire it into your app:
Use a LaunchDarkly SDK or API to fetch the AI Config at runtime and pass the returned prompt/model to your LLM call.
Target and roll out safely:
Add rules to roll out a new prompt/model to a specific segment or percentage, monitor impact, and promote or rollback without redeploys.

How to Change Prompts/Models at Runtime

1. Model your AI behavior as a Config

Instead of this:

// Hard-coded prompt & model (risky)
const prompt = `You are a helpful support assistant...`;
const model = "gpt-4.1";

const response = await llm.chat({ model, prompt, messages });

You move to this:

// Pseudocode: Fetch AI Config from LaunchDarkly
const aiConfig = await ldClient.aiConfig("support-assistant", userContext);

// aiConfig might include: prompt, model, temperature, tools, system settings
const response = await llm.chat({
  model: aiConfig.model,
  temperature: aiConfig.temperature,
  messages: [
    { role: "system", content: aiConfig.systemPrompt },
    ...messages,
  ],
});

The behavior now lives in LaunchDarkly:

systemPrompt / userPrompt / templates
model (e.g., gpt-4.1, claude-3.5-sonnet, internal model ID)
parameters (temperature, max tokens, tool definitions)
Optional multi-agent/agent-graph configuration

2. Edit prompts and models in LaunchDarkly

In the LaunchDarkly UI:

Navigate to AI Configs.
Open your config (e.g., support-assistant).
Update:
- Prompt text (e.g., tighten instructions to reduce hallucinations).
- Model (e.g., switch from gpt-4.1-mini to gpt-4.1).
- Parameters (e.g., reduce temperature, adjust context window).
Save the changes.

Those updates apply at runtime:

No new deploy.
No restart of services.
Global propagation in under 200ms.

You can require approvals and use audit logs so changes to prompts/models are governed like any other production release.

How to Roll Out Changes to a Segment First

The key is to treat AI Config changes like a progressive rollout, not a big-bang switch.

Step 1: Create a “baseline” and a “new” configuration

Inside one AI Config, define two variants of behavior:

Baseline: Existing prompt + existing model.
New: Updated prompt and/or new model.

You might structure it as:

version: "baseline" → current prompt, model A
version: "v2-extraction" → new prompt, model B

The AI Config returns the correct version based on targeting rules.

Step 2: Define a segment for early exposure

Create a segment in LaunchDarkly:

Examples:
- beta_customers
- internal_users
- high_volume_merchants
Criteria can be:
- By user attributes (e.g., plan == "enterprise").
- By org/account ID.
- By internal staff emails/domains.
- Manually managed list.

This segment will get your new prompt/model first.

Step 3: Add targeting rules to the AI Config

In the AI Config targeting UI, set up logic like:

If user is in beta_customers segment → use v2-extraction
Else → use baseline

You can also combine with percentage rollouts:

For beta_customers: 20% get v2-extraction, 80% stay on baseline.
For everyone else: 0% on v2-extraction, 100% on baseline.

Outcome:

Only your chosen segment sees the new AI behavior.
You can start tiny (1–5%), observe, and step up the rollout as you gain confidence.

Step 4: Observe behavior and impact

Because AI changes can fail in subtle ways, you pair targeting with observability:

Log which AI Config version was served (baseline vs v2).
Track:
- Error rates (timeouts, provider errors).
- User outcomes (task completion, click-through, deflection).
- Guardrail metrics (blocked outputs, toxicity flags).
Use LaunchDarkly’s LLM observability with AI Configs to inspect:
- Response quality.
- Cost and latency.
- Behavior differences between variants.

This gives you real-time signal to know if the new prompt/model is actually better, worse, or just different.

Step 5: Roll forward or rollback instantly

If things look good:

Increase the rollout percentage for your segment (e.g., 20% → 50% → 100%).
Optionally start exposing v2-extraction to a broader segment or to all users.

If things go sideways:

Flip the configuration back to baseline for the affected rules.
Or use an associated kill switch feature flag to disable the AI feature entirely.

No redeploys, no 2am hotfix.

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Runtime AI Configs	Centralizes prompts, models, and parameters behind a runtime-evaluated config.	Change AI behavior after deploy, without touching code.
Targeting & Segmentation	Targets specific segments or percentages with different AI Config versions.	Roll new prompts/models to a safe audience first.
Guardrails & Governance	Uses approvals, policies, and audit logs for AI changes.	Control who can change AI behavior, and track every edit.
LLM Observability	Surfaces quality, cost, and performance metrics for AI Configs.	See how AI changes behave in production in real time.
AI Experiments & Versioning	Compares multiple AI Config versions (e.g., prompts/models) side-by-side.	Find the best-performing configuration without heavy stats work.
Automated Rollback (Guardian)	Monitors thresholds and can pause or rollback features automatically.	Reduce blast radius when AI behavior regresses.

Ideal Use Cases

Best for AI assistants and copilots:
Because you can ship new reasoning prompts or models to internal users or beta customers first, see how they behave, then expand confidently.
Best for search, recommendations, and content generation:
Because you can trial a more capable (or cheaper) model on a subset of high-value traffic, compare quality and cost, and then standardize on the winning configuration—no redeploys.

Limitations & Considerations

You still need instrumentation:
AI Configs control behavior, but you must emit metrics (success, complaints, cost, latency) and connect them to Config versions to make informed decisions. Use LaunchDarkly’s observability SDKs plus your existing logging/analytics.
Not a replacement for model training or safety frameworks:
AI Configs help you manage prompts, models, and agents in production. You still need upstream model governance (provider selection, safety settings) and downstream content policies.

Pricing & Plans

Details vary by contract, but the pattern is:

AI Configs, feature flags, experimentation, and observability live in the same runtime control platform.
Pricing typically scales with:
- Number of MAUs / flag evaluations.
- Environments and projects.
- Feature sets (experimentation, advanced governance, etc.).

Talk to LaunchDarkly for specifics, but in practice teams layer AI Configs on top of their existing feature management investment so AI and non-AI changes share one release surface.

Core / Growth-style plans: Best for product and engineering teams needing reliable feature flags, basic AI Configs, and progressive rollouts.
Enterprise-style plans: Best for larger orgs needing advanced governance (custom roles, approvals, audit logs), higher-volume usage, AI-focused experimentation, and stricter compliance.

Frequently Asked Questions

How do we safely test a new prompt or model on just one customer segment?

Short Answer: Create multiple versions inside an AI Config, target the “new” version to a defined segment or percentage, and keep everyone else on the baseline until you’re confident.

Details:
Define an AI Config with both baseline and new versions (different prompts/models). Build a segment like beta_customers. In the AI Config rules:

If user in beta_customers → serve new version (optionally with a small percentage rollout).
Else → serve baseline.

Monitor performance and behavior with LLM observability and your app analytics. If metrics look healthy, increase the rollout or expand segment coverage. If not, switch the segment back to baseline in one click.

Can non-engineering teams change prompts and models without breaking production?

Short Answer: Yes—if they’re given the right roles and workflows, they can update prompts/models through the LaunchDarkly UI with approvals and audit logging.

Details:
Because prompts and models live as AI Configs in LaunchDarkly, product managers, AI ops, or content teams can propose changes without editing code. You can:

Use custom roles to control who can modify which AI Configs.
Require approvals for changes in production environments.
Use audit logs to see who changed what and when.

This allows non-engineers to iterate on prompts and models quickly, while engineering retains guardrails and observability to keep production safe.

Summary

To change prompts and models at runtime and roll those changes to a segment first, you:

Move AI behavior into AI Configs instead of hard-coded strings.
Fetch those configs at runtime through LaunchDarkly’s SDKs or API.
Use targeting rules and segments to expose new prompts/models to a small, controlled audience.
Pair rollouts with LLM observability and metrics to validate impact.
Use approvals, policies, and automated rollback to keep changes safe—even when they ship at AI speed.

You end up with one control plane for AI and non-AI releases, and a way to improve AI behavior continuously without redeploys or 2am fire drills.

Next Step

Get Started