Our LLM spend jumped after switching models/providers—how do we pinpoint which prompts, tools, or flows are driving cost?

Most teams only notice their LLM costs when the bill spikes—especially after switching models or providers. If your spend suddenly jumped and you don’t know which prompts, tools, or flows are responsible, you don’t have a finance problem; you have an observability problem.

In this guide, I’ll walk through how to instrument your AI stack so you can pinpoint cost down to the exact prompt, tool, and agentic path—then use Future AGI to keep that visibility continuous as you iterate.

---

Why LLM spend jumps when you switch models/providers

LLMs are probabilistic. When you swap models (e.g., OpenAI → Anthropic, or gpt-4 → gpt-4.1-mini), you’re changing:

• Token economics: Different $/1K input and output token rates, minimum charges, and rounding behavior.
• Response patterns: One model might be more verbose, use more function-calls, or retry more often.
• Tool behavior: Some providers encourage more aggressive tool-calling or return larger tool JSON payloads.
• Agent depth: Slightly different reasoning can result in more “steps” in an agent workflow (more calls, more tools).

If you don’t trace each call and attribute cost to prompts, tools, and flows, all of this collapses into a single line-item: “LLM spend.” That’s not useful for optimization or GEO-focused experimentation.

---

The outcome you actually want: cost per scenario, not cost per month

To manage LLM economics intelligently, you need to move from:

• “We spent $X this month”
  to
• “This specific flow (e.g., ‘email summarization with RAG + classification’) costs $0.013 per run, and this prompt-tool combo added 35% to that cost with no measurable quality gain.”

That requires:

1. Granular traces of every agent run and tool call.
2. Deterministic evaluation so you can balance cost vs. quality.
3. Attribution: which prompt, which tool, which experiment variant, which provider.
4. Monitoring: real-time detection when costs per flow drift up.

This is exactly where Future AGI’s “Datasets → Experiment → Evaluate → Improve → Monitor & Protect” lifecycle becomes a cost-control system, not just an accuracy-control system.

---

Step 1: Instrument your stack so every token has an owner

Before you can answer “which prompts, tools, or flows are driving cost?”, you need traces.

What to capture on each request

Regardless of provider (OpenAI, Anthropic, Bedrock, Gemini, etc.), log:

• Request metadata

  • Flow / feature name (e.g., voice_support_agent, sales_research_agent)
  • Environment: dev / staging / production
  • User or session ID (pseudonymized if needed)
  • Request timestamp

• LLM call details

  • Provider and model (gpt-4.1-mini, claude-3.5-sonnet, etc.)
  • Prompt template ID / version
  • System / user / tool messages
  • Tool/function call names and arguments
  • Input tokens, output tokens
  • Per-call cost (provider rate × token counts)

• Agent context

  • Which tools were available/used
  • Number of steps / depth of reasoning
  • Branch taken in multi-step flows (e.g., “RAG + classification” vs “classification only”)

With Future AGI, you wire this up via SDK-style instrumentation (e.g., pip install traceAI-openai and provider-specific instrumentors) so every call is traced without rewriting your whole stack.

Claim → Mechanism → Outcome

• Claim: You can’t control cost without per-call traces.
• Mechanism: SDK instrumentation records tokens, models, prompts, tools for every request.
• Outcome: You can immediately see which flows and prompts are cost hotspots.

---

Step 2: Aggregate cost by flow, prompt, and tool

Once traces are flowing, the next job is aggregation: turn millions of calls into a few clear cost drivers.

Slice by flow

Group traces by application or flow name, e.g.:

• customer_support_chat
• sales_email_generation
• weekly_report_summarizer
• voice_assistant_triage

Compute for each:

• Total spend per period
• Average cost per run
• 95th percentile cost per run
• Call volume (runs/day or runs/hour)

This immediately answers: “Which flows became more expensive after the model switch?”

Drill down by prompt and prompt version

Within a flow, break down cost by:

• Prompt template ID (e.g., cs_triage_v3)
• Prompt version (e.g., cs_triage_v3.1_long, cs_triage_v3.1_short)
• System vs. user prompt patterns

You’ll often see:

• A verbose “safety” or “style” system message adding thousands of tokens per day.
• Legacy prompts still hitting the old model while new flows use the cheaper model.
• Experiment prompts that were never cleaned up still running in production.

Future AGI’s Prompt Hub + Prompt Playground + traces make this visible: you know which prompt version is active where, and what it costs in real traffic.

Attribute cost to tools

For tool-using agents (RAG, search, APIs), treat tools as first-class cost drivers:

• Track number of tool calls per run.
• Track input/output token usage around tool calls (RAG context is often the silent killer).
• Tag tools: expensive_tool (calls external APIs) vs cheap_tool (local DB).

Patterns you’ll find:

• RAG context windows bloated to 8–16k tokens when 1–2k would maintain quality.
• Multiple redundant tool calls due to missing caching or poor planning.
• Tool-driven loops (e.g., re-querying search) that only show up in traces, not logs.

With Future AGI’s Observe Traces and out-of-the-box dashboards, this becomes “click to expand the run and see exactly where cost spiked.”

---

Step 3: Compare before/after the model or provider switch

To understand why your LLM spend jumped, you need before/after comparisons at the workflow level, not just overall.

Build a representative dataset (scenarios)

Start with Datasets in Future AGI:

• Collect real production examples or generate synthetic datasets that mirror your common use cases (including edge cases).
• Include metadata: flow, complexity label, user segment, language, etc.

This creates a fixed benchmark so you can run “old model vs new model” under identical conditions.

Run Experiments: old vs new configurations

Use Experiment to define:

• Variant A: original provider/model, prompts, tools.
• Variant B: new provider/model, same prompts and tools.
• Variant C: new provider/model, updated prompts (shorter, more constrained).
• Variant D: new model + modified tool strategy (e.g., RAG only for complex queries).

Run the same dataset through all variants using your own API key. For each:

• Record cost per scenario.
• Record latency.
• Evaluate quality with deterministic metrics (see next section).

Future AGI’s no-code experiment setup + “Winner prompt identifier” lets you do this without stitching together custom scripts.

---

Step 4: Evaluate quality so you don’t optimize cost blindly

Cutting tokens is easy; maintaining quality and safety while reducing cost is hard. That’s where Evaluate matters.

Use deterministic evals, not vibes

For each scenario in your dataset, define metrics such as:

• Task success (binary or graded)
• Factual accuracy (for RAG flows)
• Relevance / coverage of key points
• Conciseness / verbosity
• Policy/safety adherence (no PII, no harmful content)

Future AGI provides proprietary eval metrics and supports custom ones. The key is:

• Same scenarios
• Same rubric
• Different variants (prompts/models/tools)

Now you can say: “Variant B is 30% cheaper but loses 5% accuracy; Variant C is 20% cheaper with no quality loss; Variant D is 40% cheaper for easy cases and automatically escalates complex cases to a more expensive model.”

Pin-point root cause of expensive failures

With Error Localizer and traces, you can:

• Identify where quality drops are correlated with:
  • Specific prompts
  • Specific tools
  • Specific models or model families
• See which eval failures have outlier token counts or unusually deep tool chains.

Claim → Mechanism → Outcome:

• Claim: You can reduce cost while maintaining or improving quality.
• Mechanism: Experiments + deterministic evals compare cost/quality for each workflow variant.
• Outcome: You pick a “Winner” that balances spend vs. accuracy safely.

---

Step 5: Improve prompts and flows based on cost + eval feedback

Once you know which prompts and tools are expensive and how they affect quality, use Improve to close the loop.

Optimize prompts for cost

Typical wins:

• Shorter, more focused system messages (cut boilerplate)
• Use explicit output formats (JSON, bullet points) to reduce rambling
• Avoid double-instruction patterns (“Follow these rules…” repeated multiple times)
• Use model-specific guidance tailored to the provider (e.g., better tool hints)

Future AGI can automatically refine your prompt based on evaluation feedback and constraints (e.g., target max output tokens or cost per call), then re-test them in new Experiments.

Optimize tool usage

• Introduce conditional RAG: only call RAG when the query requires context, identified via a classifier or rules.
• Limit retrieval context: fewer, more relevant chunks instead of dumping entire documents.
• Collapse multi-step tools where possible (e.g., a single batch API instead of many small calls).
• Use cheaper models for classification / routing; reserve expensive models for hard reasoning or generation.

Run the improved configuration through the same dataset and evals. If it wins on cost while meeting your quality thresholds, promote it to production.

---

Step 6: Monitor & Protect in production to catch future cost spikes

Fixing cost once is not enough. As prompts, models, and traffic patterns evolve, you need continuous monitoring.

Monitor for cost anomalies and drift

With Monitor & Protect, you can:

• Track Observe Traces volume and cost over time.
• Use Outlier Detection to flag:
  • Unusual spikes in token usage.
  • Flows whose per-run cost drifts above a threshold.
  • Individual runs that are 10–100x more expensive than median.
• Use Watchdog: Automated insights to highlight flows or prompts that suddenly become cost outliers.

Configure alerting integrations (Slack, email, etc.) so your team knows when:

• A new deploy increases cost per run for a specific flow.
• A specific prompt version is producing abnormally long responses.
• A model update from a provider changes token behavior.

Protect safety with minimal latency

Cost and safety are linked. Prompt injection or malicious queries can trigger excessive tool calls or huge context windows.

With Protect and multimodal guardrailing, you can:

• Block or sanitize input/output that attempts prompt injection, leaks sensitive data, or triggers expensive query patterns.
• Enforce privacy, toxicity, and prompt injection guardrails with minimal latency so your voice/real-time agents remain responsive.

This prevents attackers and abusive edge cases from inflating your LLM spend.

---

How Future AGI’s pricing lines up with cost visibility

Future AGI’s plans are built so you can start instrumenting cost without committing to a huge budget:

• Starter (Free)

  • For new teams exploring LLM evals and tracing.
  • 3 seats, one-time $5 credit, $0 monthly credits.
  • Perfect for wiring up traces, running small Experiments, and validating that you can see cost per prompt/flow.

• Growth

  • For small teams and startups running serious workloads.
  • Unlimited seats, one-time $5 credit, $20 monthly credits.
  • Monthly retainer around $50, with higher usage limits.
  • Best when you want continuous Observe Traces, regular experiments, and production Monitor & Protect.

• Enterprise

  • For teams with advanced needs or global scale.
  • Custom seats, credits, and limits.
  • Best if you have multiple products, high-volume traffic, and want deep support plus private Slack guarantees.

You can also use the cost calculator on the site to estimate spend based on your expected trace volume and evaluations.

---

Practical checklist: pinpoint which prompts/tools/flows are driving cost

Use this as a quick implementation guide:

1. Instrument

   • Integrate Future AGI’s tracing SDK with your LLM provider(s).
   • Tag traces with flow name, prompt ID, tool usage, and model.

2. Baseline

   • Collect 1–2 weeks of production traces.
   • Build dashboards for cost per flow, per prompt, per tool, and per model.

3. Detect hotspots

   • Identify top 3 flows by total cost.
   • Within each, find:
     • Prompts with highest average tokens.
     • Tools with high call frequency or large payloads.
     • Outlier runs with extreme token counts.

4. Experiment

   • Build a dataset of representative requests.
   • Create Experiments comparing:
     • Old vs new model/provider.
     • Short vs long prompts.
     • RAG-on-everything vs conditional RAG.

5. Evaluate

   • Apply deterministic evals for accuracy, coverage, and safety.
   • Use Error Localizer to find where expensive runs align with quality failures.

6. Improve

   • Refine prompts with Prompt Hub / Prompt Playground.
   • Adjust tool strategies based on experiment results.
   • Promote the “Winner” configuration.

7. Monitor & Protect

   • Enable anomaly detection and Watchdog.
   • Configure alerts for cost-per-run drift.
   • Use Protect guardrails to block abusive or cost-exploding patterns.

---

Summary

When your LLM spend jumps after switching models or providers, the solution isn’t just “use a cheaper model.” You need end-to-end visibility:

• Traces to see every token, prompt, tool, and path.
• Experiments and deterministic evals to compare configurations.
• A closed loop to improve prompts and tools with measurable outcomes.
• Monitoring and guardrails to prevent future cost surprises.

Future AGI packages this lifecycle—Datasets → Experiment → Evaluate → Improve → Monitor & Protect—so you can deploy agentic workflows with predictable quality and predictable cost.

Next Step

Ready to see exactly which prompts, tools, and flows are driving your LLM bill—and fix them with data, not guesswork?
Get Started