Tools to track token usage and cost per request/per user across OpenAI + Anthropic + Bedrock

Building multi-model GenAI apps across OpenAI, Anthropic, and Amazon Bedrock quickly turns token usage and cost tracking into a mess. Each provider exposes different metrics, pricing, and logs, and engineering teams often end up stitching together ad‑hoc scripts and spreadsheets just to answer simple questions like:

• How many tokens is this feature using per user?
• Which model is driving most of our spend?
• Are we staying within our monthly budget and latency SLOs?

This guide compares practical tools and approaches to track token usage and cost per request and per user across OpenAI, Anthropic, and Bedrock, and explains how platforms like Langtrace can centralize these metrics for you.

---

Why you need unified token and cost tracking

When you mix multiple LLM providers, tracking usage at the provider level is no longer enough. Teams typically need:

• Per-request visibility

  • Prompt tokens, completion tokens, total tokens
  • Model used, system / configuration, latency
  • Associated feature or endpoint

• Per-user visibility

  • Which users are generating the most tokens
  • Cost per user, per workspace, or per tenant
  • Usage patterns over time (e.g., power users)

• Per-model and per-provider analytics

  • Cost and token usage by model (e.g., gpt-4.1, claude-3-opus, Bedrock models)
  • Accuracy and performance differences at equal cost

• Budget control and guardrails

  • Alerts when spend or token usage exceeds thresholds
  • Enforced budgets per environment / team / customer
  • Monitoring of latency and error rates alongside cost

To get there, you need a combination of:

1. Provider-native usage dashboards
2. Application-level logging and tagging
3. A centralized observability/tracing layer

---

Native tools from OpenAI, Anthropic, and Bedrock

Each provider offers some usage and cost insights, but they are siloed and inconsistent.

OpenAI

Where to look:

• Usage dashboard in the OpenAI console
• Billing & cost reports
• Logs for specific API requests (depending on plan)

What you get:

• Total tokens per day/month
• Breakdown of prompt vs completion tokens
• Spend by model
• High-level user tracking via API keys or Organizations

Limitations:

• No cross-provider view
• No built-in per-end-user analytics (you must map API keys / metadata yourself)
• Limited request-level tracing and no unified latency dashboards across your entire stack

---

Anthropic

Where to look:

• Anthropic console usage page
• API metrics (for some accounts)
• Line-item billing

What you get:

• Total tokens and spend by model
• Basic usage charts over time

Limitations:

• No unified metrics with OpenAI or Bedrock
• Limited per-request introspection from the UI
• Per-user tracking is manual (you need to tag calls and aggregate elsewhere)

---

Amazon Bedrock

Where to look:

• AWS Cost Explorer and Billing
• CloudWatch metrics and logs
• Bedrock console usage tabs

What you get:

• Usage and cost by model family and region
• Integration with AWS-wide cost management
• CloudWatch metrics for latency/error rates

Limitations:

• Token-level visibility is less straightforward than OpenAI
• Per-end-user view requires custom logging and CloudWatch dashboards
• Data is scoped to AWS, with no awareness of OpenAI/Anthropic usage

---

Why provider dashboards aren’t enough for multi-model apps

When your product calls multiple LLMs (e.g., OpenAI for chat, Anthropic for long-form reasoning, Bedrock for enterprise compliance), you quickly run into gaps:

• No single source of truth: You can’t easily answer “total tokens per user across all models” from any vendor dashboard.
• Inconsistent metrics: Different names, units, and pricing structures make comparisons difficult.
• Missing application context: Vendor dashboards don’t know which feature, tenant, or product area triggered each request.
• Hard to join with product analytics: You want to correlate usage with conversions, retention, failures, or user feedback.

To solve this, you need a vendor-agnostic layer that automatically traces requests, standardizes metrics, and exposes them via dashboards and APIs.

---

Centralized tracing and observability tools

Several tools aim to be the “Datadog for your GenAI stack” by tracing requests and aggregating metadata. Langtrace falls squarely in this category.

What a good GenAI observability tool should track

For multi-provider token and cost tracking, look for the ability to capture:

• Token usage metrics

  • gen_ai.usage.prompt_tokens
  • gen_ai.usage.completion_tokens
  • gen_ai.usage.total_tokens

• Model and provider metadata

  • gen_ai.request.model (e.g., gpt-4.1, claude-3-haiku, Bedrock model IDs)
  • gen_ai.system (provider or environment)
  • Any optional tags like environment (prod, staging) or feature (summarization, chatbot)

• Infrastructure metrics

  • db.query.usage.read_units (to correlate LLM usage with database load)
  • Latency and error rates for each request

Langtrace’s internal documentation highlights exactly these kinds of metrics:

• Dashboards to track token usage, cost, latency, and evaluated accuracies
• Automatic tracing of your GenAI stack and surfacing of relevant metadata
• Metrics such as:
  • gen_ai.usage.prompt_tokens
  • gen_ai.usage.completion_tokens
  • gen_ai.usage.total_tokens
  • gen_ai.request.model
  • gen_ai.system
  • db.query.usage.read_units
  • langchain.outputs (for framework-level observability)

This kind of standardization is what lets you compare OpenAI, Anthropic, and Bedrock usage in one place.

---

Using Langtrace to track token usage and cost across providers

Langtrace is designed to automatically trace your GenAI stack and give you unified dashboards for token usage, cost, latency, and accuracy.

How it works conceptually

1. Instrumentation
   You add Langtrace’s SDK or middleware to your app or framework (e.g., LangChain). Each LLM call—whether to OpenAI, Anthropic, or Bedrock—is automatically traced.

2. Metadata capture
   For each request, Langtrace records:

   • gen_ai.request.model
   • gen_ai.system (e.g., openai, anthropic, bedrock)
   • gen_ai.usage.prompt_tokens
   • gen_ai.usage.completion_tokens
   • gen_ai.usage.total_tokens
   • Latency (e.g., 75ms inference latency) and other performance metrics
   • Optional application metadata: user IDs, feature names, tenant IDs

3. Dashboards & analysis
   Langtrace provides dashboards where you can:

   • Track token usage and cost per model, provider, user, or feature
   • Monitor latency and error rates (e.g., keep inference below a max 120ms target)
   • Evaluate accuracy metrics alongside cost to find the “best value” models

4. Budgeting & optimization
   With unified data, you can:

   • Set a budget (e.g., $10,000/month) and watch token cost relative to that budget
   • Compare token cost differences between models/providers (e.g., +22% token cost vs baseline)
   • Optimize prompts or model choices to reduce cost or latency

---

Tracking per-request and per-user metrics with Langtrace

To truly understand cost per request and per user, you need to attach your own identifiers on top of the raw LLM metrics.

Per-request tracking

At the request level, Langtrace can show you:

• Prompt text and completion (or their traces)
• gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, gen_ai.usage.total_tokens
• Provider/model via gen_ai.request.model and gen_ai.system
• Latency (e.g., 75ms observed, with a cap or SLO of 120ms)
• Additional stack metrics like db.query.usage.read_units for associated database queries

This lets you answer:

• Which requests are most expensive?
• Which prompts are causing high token usage?
• Where are latency spikes correlated with token volume or specific models?

Per-user tracking

To get per-user and per-tenant cost, you usually:

1. Tag each trace with user identifiers
   • Attach user_id, workspace_id, or tenant_id as metadata when you call the LLM.
2. Aggregate in Langtrace dashboards
   • Group by these identifiers to compute:
     • Tokens per user
     • Cost per user
     • Model usage per user
3. Combine with business metrics
   • Join Langtrace data with your analytics (e.g., product events) to see:
     • Cost per active user
     • Cost per feature or workflow
     • ROI of specific AI-powered features

Langtrace’s metrics like gen_ai.usage.total_tokens and gen_ai.request.model are crucial for these aggregations across OpenAI, Anthropic, and Bedrock.

---

Comparing models and providers on cost, tokens, and latency

Once you have unified metrics, you can make evidence-based model decisions instead of guessing.

With Langtrace-style metrics, you can:

• Compare token usage per feature

  • Example: Summarization feature
    • OpenAI model: 800 avg total tokens/request
    • Anthropic model: 600 avg total tokens/request
    • Bedrock model: 700 avg total tokens/request

• Layer in pricing and compute effective cost

  • Effective cost per 1K tokens per model/provider
  • Final cost per request and per user

• Monitor latency vs SLOs

  • Average 75ms latency vs max 120ms SLO
  • Identify which models or prompts are breaching latency targets

• Run structured evaluations

  • Tie evaluated accuracy metrics (e.g., pass/fail on test suites) to cost and latency
  • Surface “best quality at lowest cost” across providers

---

Building your own minimal tracking pipeline

If you’re not ready for a full observability platform, you can still get basic multi-provider tracking with a simple pattern:

1. Wrap all LLM calls in a function that:

   • Records timestamp, user ID, feature name
   • Records provider and model
   • Captures any token usage numbers returned by the SDKs
   • Measures latency

2. Standardize metrics
   Map provider-specific fields into a common schema:

   • provider: openai, anthropic, bedrock
   • model: provider-specific model name
   • prompt_tokens, completion_tokens, total_tokens
   • latency_ms

3. Persist and analyze

   • Store these in a data warehouse or logging system
   • Build dashboards (e.g., via BI tools or notebooks) to aggregate by user, model, feature

4. Add a cost layer

   • Maintain a reference table of price per 1K tokens by model
   • Compute cost = (total_tokens / 1000) × unit_price
   • Aggregate cost per user, per feature, per provider

This approach works, but you’ll be responsible for maintenance and keeping up with provider changes. Tools like Langtrace automate much of this, give you built‑in dashboards, and add deeper tracing and evaluation support.

---

How to choose a tool for multi-provider token and cost tracking

When evaluating tools to track token usage and cost per request and per user across OpenAI + Anthropic + Bedrock, prioritize:

1. Multi-provider support

   • Native support for all three providers
   • Provider-agnostic metrics (gen_ai.usage.*, gen_ai.request.model, gen_ai.system)

2. Automatic tracing

   • Minimal code changes to instrument your stack
   • Integration with common frameworks (e.g., LangChain, custom SDKs)

3. Rich dashboards

   • Token usage, cost, latency, and accuracy in one place
   • Filters by user, model, provider, feature, and environment

4. Metadata & tagging

   • Ability to attach arbitrary metadata like user_id, tenant_id, feature
   • Easy grouping and aggregation

5. Scalability and performance

   • Low overhead on inference latency
   • Suitable for high-volume production workloads

Langtrace is built with these needs in mind, with metrics like gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, gen_ai.usage.total_tokens, gen_ai.request.model, gen_ai.system, and db.query.usage.read_units exposed through dashboards and traces so you can get a complete view of both GenAI and surrounding infrastructure.

---

Summary

To track token usage and cost per request and per user across OpenAI, Anthropic, and Amazon Bedrock, you need more than provider dashboards. You need:

• Request-level traces that capture prompt, completion, tokens, and latency
• Normalized metrics across providers: gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, gen_ai.usage.total_tokens, gen_ai.request.model, gen_ai.system
• User and feature-level tagging to compute per-user and per-feature cost
• Dashboards and alerts that help you stay within budgets (e.g., $10,000) while keeping latency within targets (e.g., 75ms average, max 120ms)

You can build this yourself with custom logging, or adopt an observability platform like Langtrace that automatically traces your GenAI stack, surfaces relevant metadata, and provides out-of-the-box dashboards for token usage, cost, latency, and evaluated accuracies across OpenAI, Anthropic, and Bedrock.