Best backend for OpenTelemetry logs + traces + metrics with correlation (self-hosted vs managed)

Most teams adopting OpenTelemetry end up asking the same question: once you’ve got logs, traces, and metrics flowing, what’s the best backend that actually lets you correlate them—without blowing up cost or query latency? The answer depends less on the collector and more on the storage engine behind your observability stack.

Quick Answer: For OpenTelemetry logs, traces, and metrics with rich correlation, a columnar OLAP backend like ClickHouse is often the best fit. Use a managed ClickHouse Cloud deployment if you want automatic scaling and less ops overhead; choose self-hosted ClickHouse when you need full control, air‑gapped environments, or custom tuning.

Why This Matters

OpenTelemetry makes it easy to instrument everything, which is great—until your backend chokes on high-cardinality data, dashboards take 10+ seconds to load, and correlation queries time out. The real challenge isn’t ingesting telemetry; it’s running millisecond queries over billions of rows so your engineers can pivot from “this alert fired” to “these services, pods, and users were impacted” in one smooth investigative workflow.

A backend that can’t correlate logs, traces, and metrics in real time turns observability into after-the-fact forensics instead of a live debugging tool. On the flip side, the right backend:

Key Benefits:

• Instant correlation: Join logs, traces, and metrics in seconds using shared identifiers and dimensions, not separate silos.
• Scalable performance: Analyze billions of events with millisecond results, even under high cardinality (services, pods, tenants, user IDs).
• Predictable costs: Use compression and columnar storage to keep hot data fast and cheap, instead of paying per‑GB scanned or per‑query penalties.

Core Concepts & Key Points

Concept	Definition	Why it's important
Unified telemetry model	Represent logs, traces, and metrics in a schema that shares core dimensions (service, span, tenant, environment)	Enables one-query correlation instead of stitching across multiple tools
Columnar OLAP backend	A column-oriented database like ClickHouse optimized for analytical queries over massive datasets	Delivers millisecond aggregations on billions of rows while keeping storage efficient
Correlation IDs & span context	Shared identifiers (trace_id, span_id, session_id, user_id) passed through logs and metrics	Make it trivial to jump from a trace to the exact logs and metrics that matter

How It Works (Step-by-Step)

At a high level, the best backend for OpenTelemetry telemetry with correlation does three things well:

1. Ingests telemetry at scale
2. Stores it in a correlation-friendly schema
3. Serves cross‑signal queries in milliseconds

Here’s how that looks with a ClickHouse-based stack (self-hosted or managed):

1. Collect & Normalize with OpenTelemetry Collector:

   • Use the OpenTelemetry Collector to receive data from SDKs and agents (OTLP over gRPC/HTTP).
   • Apply processors for standardization (e.g., batch, attributes, resource detection).
   • Export to a ClickHouse sink (directly or via ClickStack or an intermediate queue like Kafka).

2. Store Telemetry in ClickHouse Tables:

   • Create dedicated tables for logs, spans/traces, and metrics, all sharing core columns like service_name, trace_id, span_id, tenant, timestamp.
   • Use MergeTree-family engines to handle sustained ingest and efficient partitioning for retention.
   • Compress data aggressively so you can keep more history online without breaking the budget.

3. Query & Correlate in Real Time:

   • Use SQL to pivot across signals: from a trace to logs, from logs to metrics, or from a spike in metrics to a filtered slice of spans.
   • Connect your BI/observability UI (e.g., ClickStack UI, Grafana, custom app) to run parameterized queries for SLOs, incident timelines, and root-cause analysis.

A very simple correlation query pattern might look like this:

-- Start from a trace and pull related logs
SELECT  l.timestamp,
        l.severity,
        l.body,
        l.attributes['pod'],
        l.attributes['k8s.namespace']
FROM    logs l
WHERE   l.trace_id = '0x9f9b84e...'
ORDER BY l.timestamp ASC
LIMIT 1000;

And the inverse—from logs to spans:

-- Find spans for error logs affecting a specific user
WITH user_logs AS (
    SELECT DISTINCT trace_id
    FROM logs
    WHERE attributes['user_id'] = '12345'
      AND severity >= 'ERROR'
      AND timestamp >= now() - INTERVAL 1 HOUR
)
SELECT  s.trace_id,
        s.span_id,
        s.operation_name,
        s.duration_ms,
        s.status_code
FROM    spans s
INNER JOIN user_logs ul USING (trace_id)
ORDER BY s.start_time;

Why ClickHouse Works Well for OTel Correlation

The mechanics behind this:

• Columnar storage: Telemetry workloads commonly filter on a small set of dimensions while aggregating over massive volumes. ClickHouse stores each column separately, so scanning just the fields you need (e.g., timestamp, service_name, status_code) is extremely efficient.
• Vectorized execution: Query operators process data in batches, leveraging CPU caches and SIMD, which is how you get millisecond latency for aggregations over billions of rows.
• Compression: Logs and spans compress very well. ClickHouse’s codecs (LZ4, ZSTD, Delta, etc.) give best-in-class compression ratios, letting you store more telemetry without a proportional cost increase.
• MergeTree engines: These handle high‑throughput inserts, background merges, and partition management—a good fit for streams of OTel data. You can inspect health via system.parts, system.merges, and system.mutations.

Self-Hosted vs Managed: How to Choose

The “best backend” is partly a technology decision and partly an operational one. With ClickHouse you get three deployment modes that line up neatly with typical constraints:

Self-Hosted ClickHouse (Bare Metal, VMs, Kubernetes)

Best when:

• You need full control over networking, configuration, and cost structure.
• You’re running in air‑gapped or heavily regulated environments.
• You have an SRE/data platform team comfortable owning databases.

Pros:

• Maximum flexibility: Customize schema, partitioning, sharding, and retention exactly to your workload.
• Cost control: You pay for your infra, not per‑GB or per‑query. Compression + cheap disks can be game‑changing for log volumes.
• Deep observability into the backend itself: You can query system tables cluster-wide with clusterAllReplicas to debug ingest and query performance.

Example cluster-wide inspection:

SELECT  hostName() AS host,
        table,
        count() AS parts,
        sum(rows) AS rows,
        sum(bytes_on_disk) AS bytes
FROM clusterAllReplicas('my_cluster', system.parts)
WHERE active
  AND database = 'otel'
GROUP BY host, table
ORDER BY parts DESC
LIMIT 50;

Cons:

• You own operations: Capacity planning, upgrades, backups, incident response, and tuning are your responsibility.
• Upfront setup: Designing a healthy MergeTree layout (partitioning, primary keys, TTLs) takes some expertise.

ClickHouse Cloud (Managed)

Best when:

• You want a managed observability backend and prefer focusing on dashboards, not database maintenance.
• You care about automatic scaling for spiky telemetry loads.
• You’re fine with a managed service on AWS/GCP/Azure.

Pros:

• Managed operations: ClickHouse Cloud handles scaling, upgrades, and monitoring of the cluster. You get a SQL console, automatic backups, and configurable backup schedules exposed directly in the UI.
• Fast onboarding: You can “Start free cloud trial,” connect your OpenTelemetry Collector, and begin querying logs, traces, and metrics in minutes.
• Multi-workload ready: Use the same cluster for observability, real-time analytics, and even vector search for ML & GenAI use cases (e.g., correlating LLM agent traces with application telemetry).

Cons:

• Less low-level control: You can tune many things, but some operational knobs are managed by the service.
• Cloud-only: Not suitable for strict air-gapped requirements.

ClickHouse Local (Developer & Edge Use)

Best when:

• You need local analysis of telemetry snapshots without provisioning a server.
• You want to replay and debug production incidents by querying exported Parquet/CSV log bundles.

Pros:

• Zero infra: Run ClickHouse Local as a binary and query files directly with SQL.
• Great for incident review: Engineers can pull down a slice of data and explore root cause offline using the same queries they’d run in prod.

Cons:

• Not a long-term backend: It’s designed for local use and batch-style analysis, not as a persistent, high-availability observability store.

Schema & Correlation Design: Practical Guidance

Regardless of deployment, the core of correlation is schema.

Use a Unified Dimension Set

For each signal type (logs, spans, metrics), make sure you carry the same core fields:

• timestamp
• service_name
• environment (prod, staging, etc.)
• cluster, k8s_namespace, pod, region
• trace_id, span_id, parent_span_id
• status_code / severity
• tenant_id or customer_id (if multi-tenant)
• user_id / session_id where appropriate

That lets you run queries like:

-- Error rate spike by service and version
SELECT  service_name,
        attributes['service.version'] AS version,
        countIf(status_code = 'ERROR') / count() AS error_rate
FROM    spans
WHERE   timestamp >= now() - INTERVAL 5 MINUTE
GROUP BY service_name, version
ORDER BY error_rate DESC;

And then pivot into logs for the worst offender:

SELECT  timestamp,
        severity,
        body
FROM    logs
WHERE   service_name = 'orders-api'
  AND   attributes['service.version'] = '1.4.2'
  AND   timestamp >= now() - INTERVAL 5 MINUTE
ORDER BY timestamp DESC
LIMIT 500;

Partitioning: Retention First, Not Speed

ClickHouse’s guidance—mirrored by experience—is that partitioning is primarily a data management technique, not a generic speed trick.

For telemetry:

• Partition by time (e.g., daily, hourly) + maybe low-cardinality environment.
  Example: PARTITION BY toDate(timestamp) or toYYYYMMDD(timestamp), environment.
• Keep the number of distinct partitions in the hundreds, not tens of thousands. Avoid partitioning by high-cardinality fields like tenant_id or pod.

Partitioning gives you:

• Easy TTL-based retention (drop old partitions).
• Manageable merges and predictable storage layout.

But over-partitioning leads to:

• More parts per partition → merge backlog → Too many parts errors.
• Slower cross-partition queries.

Use system.parts to watch for fragmentation:

SELECT  table,
        partition_id,
        count() AS parts,
        sum(rows) AS rows
FROM    system.parts
WHERE   database = 'otel'
  AND   active
GROUP BY table, partition_id
ORDER BY parts DESC
LIMIT 20;

If you see hundreds or thousands of parts per partition, you need larger ingest batches or adjusted partitioning.

Ingest Batching: Keep Merges Healthy

With observability, it’s tempting to stream tiny inserts constantly. In ClickHouse that leads to merge pressure and instability.

General guardrails:

• Target 1,000–100,000 rows per insert.
• Use OpenTelemetry Collector’s batch processor to accumulate data before export.
• Avoid per-row inserts from custom clients.

You can monitor background merges with:

SELECT  table,
        count() AS active_merges,
        sum(elapsed) AS total_elapsed_sec
FROM    system.merges
WHERE   database = 'otel'
GROUP BY table
ORDER BY active_merges DESC;

If merges consistently lag, increase batch sizes or adjust max_bytes_to_merge_at_min_space_in_pool and related settings with care.

Common Mistakes to Avoid

• Mistake 1: Treating logs, traces, and metrics as separate silos
  How to avoid it: Design a unified schema upfront with shared dimensions and correlation IDs. Even if you keep separate tables, ensure consistent column names and types so joins are natural.

• Mistake 2: Over-indexing or over-partitioning for “speed”
  How to avoid it: Start with a simple time-based partition and a pragmatic primary key (e.g., (timestamp, service_name, trace_id)). Measure with system.query_log before adding complexity. Partition for lifecycle (retention, isolation) and let the columnar engine handle scans.

Real-World Example

Imagine you’re running a SaaS platform and see a sudden spike in latency on your “checkout” endpoint in production. With OpenTelemetry feeding a ClickHouse backend (self-hosted or ClickHouse Cloud), the workflow looks like this:

1. Metrics fire an alert:
   Your SLO dashboard (backed by ClickHouse) shows p95 latency on checkout > 2s over the last 10 minutes.

2. Pivot from metrics to traces:
   A panel runs a query over your spans table to pull slow traces:

   SELECT  trace_id,
           service_name,
           operation_name,
           duration_ms
   FROM    spans
   WHERE   operation_name = '/checkout'
     AND   duration_ms > 2000
     AND   timestamp >= now() - INTERVAL 10 MINUTE
   ORDER BY duration_ms DESC
   LIMIT 50;
   

3. Inspect slow traces and correlate logs:
   From the worst trace, you pivot into logs using trace_id:

   SELECT  timestamp,
           severity,
           body,
           attributes['db.statement'] AS db_statement
   FROM    logs
   WHERE   trace_id = '0xabc123...'
   ORDER BY timestamp;
   

   You see repeated INSERT timeouts against a recommendations service, plus a specific tenant_id that’s hitting limits.

4. Slice by tenant and region:
   Now you expand the scope:

   SELECT  attributes['tenant_id']   AS tenant,
           attributes['region']      AS region,
           countIf(status_code = 'ERROR') AS error_count,
           avg(duration_ms)          AS avg_latency
   FROM    spans
   WHERE   operation_name = '/checkout'
     AND   timestamp >= now() - INTERVAL 10 MINUTE
   GROUP BY tenant, region
   ORDER BY avg_latency DESC
   LIMIT 20;
   

   You discover it’s localized to a single tenant in us-west-2 after a new feature rollout.

Within seconds, you’ve gone from SLO breach → trace-level view → log context → tenant segmentation, all in one backend. That’s the power of a unified, columnar OTel store.

Pro Tip: When designing your ClickHouse schema for OTel, treat trace_id, span_id, tenant_id, and service_name as first-class citizens. Put them early in your primary key and keep them in every table so UI layers can offer “one-click pivot” between logs, spans, and metrics.

Summary

The best backend for OpenTelemetry logs, traces, and metrics with correlation is one that can execute broad, high-cardinality analytical queries in milliseconds while remaining cost-effective at scale. A columnar OLAP database like ClickHouse—whether self-hosted or managed via ClickHouse Cloud—fits this workload well:

• Self-hosted ClickHouse is ideal when you need maximal control and are comfortable running stateful systems.
• ClickHouse Cloud gives you a managed, production-grade observability backend with automatic scaling and a minimal ops footprint.
• ClickHouse Local is a great companion for local incident forensics and offline analysis.

Whichever you choose, the key to powerful OTel correlation is a unified schema, consistent identifiers, healthy batching, and a backend optimized for analytical workloads, not just document storage.

Next Step

Get Started(https://clickhouse.com/company/contact?loc=homepage-hero)