My LLM bill spiked this week—how can I break down token usage and cost by user and endpoint?

When your LLM bill suddenly spikes, the first thing you need is clarity: which users, features, or endpoints are driving the increase in token usage and cost? Without that breakdown, you’re essentially guessing—which makes it hard to optimize, enforce budgets, or have productive conversations with stakeholders.

This guide walks through how to break down token usage and cost by user and endpoint, how to monitor key metrics like latency and accuracy, and how a tracing and observability layer like Langtrace can give you this visibility out of the box.

---

1. Why token-level visibility matters when your LLM bill spikes

A surprise bill usually comes from at least one of the following:

• A new feature or endpoint quietly going viral
• A loop or bug causing repeated calls
• A model change (e.g., moving from a small to a larger, more expensive model)
• A few power users hammering the system
• Prompt bloat increasing prompt_tokens per request

To solve this, you need to:

1. Attribute usage and cost to:
   • Individual users or accounts
   • Endpoints/features
   • Models and systems
2. See trends over time so you can spot anomalies quickly
3. Connect cost to performance (latency and evaluated accuracy), so you’re not just cutting spend blindly

---

2. Instrumenting token usage: what to capture for each request

To break down your LLM bill, you need to log a minimal but consistent set of fields for every request. At a minimum, capture:

Core token and model metrics

• gen_ai.usage.prompt_tokens
• gen_ai.usage.completion_tokens
• gen_ai.usage.total_tokens
• gen_ai.request.model
• gen_ai.system (the provider / environment, e.g., “openai:gpt-4.1”, “vertexai:gemini”)

These are the vital metrics Langtrace dashboards are built around. They let you:

• Distinguish between “expensive prompts” vs “chatty models”
• Compare models and systems by cost and usage
• Detect when a model change increases token consumption

Application and business context

To go beyond raw numbers and understand who and what caused the spike, enrich each trace with:

• User-level identifiers:
  • user_id or account_id
  • tenant_id (for B2B multi-tenant apps)
• Endpoint/feature identifiers:
  • endpoint_name (e.g., summarize_document, chat_assistant, search_rerank)
  • service or module (e.g., web_app, internal_tool)
• Request metadata:
  • request_id (for debugging specific incidents)
  • environment (prod, staging, dev)

With this structure, token usage can be sliced by user, endpoint, or environment in a few clicks.

---

3. Setting up dashboards to break down cost by user and endpoint

Once you’re capturing the right fields, you can create dashboards that show where your spend is going. Langtrace is built for this: it gives you dashboards to track token usage, cost, latency, and evaluated accuracies.

Typical panels you want:

3.1. Cost and token usage by user

• Group by: user_id / account_id
• Metrics:
  • Sum of gen_ai.usage.total_tokens
  • Sum of computed cost (tokens × per-token price, per model)
• Use cases:
  • Identify top 10 most expensive users
  • Detect abuse or misconfigured integrations
  • Design per-user or per-tenant pricing that aligns with actual cost

3.2. Cost and token usage by endpoint

• Group by: endpoint_name
• Metrics:
  • Sum of gen_ai.usage.prompt_tokens
  • Sum of gen_ai.usage.completion_tokens
  • Sum of gen_ai.usage.total_tokens
• Use cases:
  • See which features are driving your bill
  • Evaluate whether expensive endpoints are delivering enough value
  • Decide where to focus optimization (prompt slimming, caching, model downgrades)

3.3. Cost by model and system

• Group by: gen_ai.request.model, gen_ai.system
• Metrics:
  • Token usage per model
  • Cost per model vs baseline
• Use cases:
  • Compare GPT-4-like models vs smaller, cheaper models
  • Spot when a team quietly moved an endpoint to a more expensive model
  • Justify model choices to finance and leadership

---

4. Monitoring latency alongside token usage

Spend isn’t your only vital metric. Latency is just as important for user experience and infrastructure planning.

With Langtrace, you can track:

• Inference latency: e.g., 75 ms for a given endpoint
• Targets and SLAs: e.g., Max 120 ms
• Latency trends vs token volume

You want dashboards that show:

• Latency by endpoint
• Latency by model
• Latency vs total tokens (high token counts often correlate with higher latency)

This helps you avoid “hidden regressions” where a spike in tokens also slows your app down.

---

5. Evaluated accuracy vs cost: are you overpaying?

Cutting cost blindly can damage quality. The better approach:

1. Track evaluated accuracy for your LLM outputs.
2. Compare accuracy vs token usage and cost per endpoint or model.

Example dashboard views:

• Accuracy score (e.g., 303 or a % score) per model
• Token cost: e.g., $6,200 over the last month
• Overlaid with:
  • Token cost change (e.g., +22% vs last week)
  • Latency change (e.g., -16% improvement)

This lets you answer:

• Did we pay more tokens for marginal or no accuracy gain?
• Is there a cheaper model that maintains similar evaluated accuracy?
• Which endpoints have a bad cost-to-accuracy ratio?

---

6. Practical workflow when your LLM bill spikes

Here’s a step-by-step playbook you can follow the next time your bill jumps unexpectedly.

Step 1: Confirm the spike and its time window

• Look at total tokens and cost over the last 7–30 days.
• Identify the exact time when usage diverged from the normal baseline.

Step 2: Break down by model

1. Group by gen_ai.request.model.
2. Compare token usage and cost before vs after the spike.
3. Look for:
   • A model that suddenly starts dominating tokens
   • New models that weren’t in use before

If a new or more expensive model is responsible, review where and why it was introduced.

Step 3: Break down by endpoint

1. Group by endpoint_name.
2. Identify endpoints with:
   • Highest increase in total tokens
   • Highest increase in cost
3. Check whether those endpoints:
   • Recently shipped or changed behavior
   • Use a new chain or orchestration layer (e.g., LangChain, custom agents)

This often reveals a particular feature or API path that’s responsible.

Step 4: Break down by user or account

1. Group by user_id / account_id.
2. Sort by total tokens or cost.
3. Investigate:
   • Outliers with unusually high growth
   • Tenants that integrated your API differently (e.g., loops, retries)
   • Internal test users accidentally generating massive traffic

For B2B products, this data informs plan limits and overage charges.

Step 5: Check prompt vs completion tokens

• Compare gen_ai.usage.prompt_tokens vs gen_ai.usage.completion_tokens:
  • If prompt tokens exploded:
    • Prompts may have become too verbose (long context, many tools, or large documents).
  • If completion tokens exploded:
    • The model may be over-responding (too long answers, missing max_tokens limits).

Tactical next steps:

• Trim prompts (remove irrelevant context, shorten system messages).
• Add or lower max_tokens for specific endpoints.
• Cache or reuse responses for duplicate prompts.

Step 6: Validate impact on latency and accuracy

Before making large changes:

• Check if latency improved or worsened during the spike.
• Check evaluated accuracy:
  • If cost increased but accuracy didn’t, the spike is likely wasteful.
  • If accuracy improved significantly, you can decide whether the extra cost is justified.

---

7. Example metric schema for tracing and dashboards

Below is a concise schema you can use in your observability layer or metrics platform, aligning with Langtrace-style metrics:

Core LLM metrics

• gen_ai.usage.prompt_tokens (int)
• gen_ai.usage.completion_tokens (int)
• gen_ai.usage.total_tokens (int)
• gen_ai.request.model (string)
• gen_ai.system (string / enum)
• latency_ms (float)
• cost_usd (float, derived)

Contextual dimensions

• user_id (string)
• account_id / tenant_id (string)
• endpoint_name (string)
• service / app (string)
• environment (string: prod, staging, etc.)
• request_id (string)

With this schema, any metrics backend (e.g., Prometheus, Datadog, OpenTelemetry-based tools, or Langtrace) can power:

• Per-user cost dashboards
• Per-endpoint token usage charts
• Alerts when:
  • Total cost exceeds a threshold
  • A single user’s usage deviates from baseline
  • Token usage for a model grows too fast

---

8. Using Langtrace to get this out of the box

Langtrace is designed to give you these insights without writing a custom observability framework.

At a high level, you:

1. Initialize Langtrace in your app:

   import langtrace
   
   langtrace.init(api_key="<your_api_key>")
   

2. Wrap or instrument your LLM calls so they automatically emit:

   • gen_ai.usage.* token metrics
   • gen_ai.request.model
   • gen_ai.system
   • Latency and evaluated accuracy (where configured)

3. Use Langtrace’s dashboards to:

   • Track token usage, cost, latency, and evaluated accuracies
   • Slice by user, endpoint, model, and system
   • Monitor your budget (e.g., a $10,000 budget) and detect changes like +22% cost or -16% latency shifts

Because these metrics are standardized, you get consistent, comparable data across models, providers, and code paths.

---

9. Best practices to keep bills predictable going forward

Once you’ve diagnosed the spike, it’s worth hardening your system to avoid future surprises.

1. Set soft and hard budgets

• At the app level: monitor cumulative cost vs a monthly budget (e.g., $10,000).
• At the user or tenant level: implement usage caps or rate limits.

2. Enforce max token limits

• Always specify max_tokens per endpoint.
• Use different limits based on:
  • Endpoint type (generation vs classification)
  • Plan tier (free vs enterprise)

3. Introduce model routing

• Use cheaper models for:
  • Simple classification / extraction tasks
  • Draft answers that a human reviews
• Reserve premium models for:
  • High-value or complex queries
  • Mission-critical endpoints

4. Continuously track evaluations

• Maintain evaluated accuracy metrics per endpoint and model.
• Periodically check whether:
  • A cheaper model now matches your baseline accuracy
  • A prompt change significantly reduces tokens with minimal quality loss

5. Alert on anomalies

• Create alerts when:
  • gen_ai.usage.total_tokens increases by X% week-over-week
  • A particular user or endpoint crosses predefined thresholds
  • Latency or accuracy drops as token usage rises

---

When your LLM bill spikes, the solution is not guesswork—it’s observability. By systematically tracking token usage, cost, latency, and evaluated accuracy, and by tagging each request with user and endpoint metadata, you can precisely identify what caused the spike, fix it quickly, and keep your LLM spend predictable as your product grows.