How do I use ClickPipes to ingest Kafka topics into ClickHouse Cloud, and what does it cost per GB?

Quick Answer: You can create a ClickPipe in ClickHouse Cloud that reads from one or more Kafka topics, maps them to a table schema, and continuously ingests data with exactly-once semantics. Pricing for ClickPipes Kafka ingestion is metered per GB of data processed (compressed on the wire), on top of your ClickHouse Cloud compute and storage; check the ClickHouse Cloud pricing page for the current per‑GB rate for your region and plan.

Why This Matters

If you’re already streaming events, logs, or metrics through Kafka, the fastest path to real-time analytics is to land them directly into ClickHouse Cloud without building and operating a custom ETL pipeline. ClickPipes gives you a managed, ClickHouse-native way to subscribe to Kafka topics, decode messages, and write into MergeTree tables that can serve millisecond queries over billions of rows.

Key Benefits:

Blazing fast time-to-value: Stand up Kafka → ClickHouse ingestion in minutes, without maintaining Kafka Connect clusters or custom consumers.
Cost effective per GB: Pay only for data processed through ClickPipes plus your ClickHouse Cloud compute/storage, instead of running separate ingestion infrastructure.
Developer friendly: Configure everything in the ClickHouse Cloud console with simple schema mapping and SQL-visible tables—no custom agents or sidecars.

Core Concepts & Key Points

Concept	Definition	Why it's important
ClickPipes	A managed ingestion service in ClickHouse Cloud that connects external sources (including Kafka) into ClickHouse tables.	Removes the need to build and operate custom pipelines; fully integrated with ClickHouse Cloud security, scaling, and monitoring.
Kafka Topic Subscription	A ClickPipe that reads from one or more Kafka topics (Confluent Cloud or self‑managed) and streams records into ClickHouse.	Lets you reuse existing Kafka-based event/log pipelines and land data directly where you query it.
Per‑GB Ingestion Pricing	Usage-based billing for the volume of data ClickPipes reads and processes, measured on compressed payloads from Kafka.	Makes costs predictable and scalable with your data volume, without overprovisioning long‑running ingestion clusters.

How It Works (Step-by-Step)

At a high level, you’ll: (1) prepare your ClickHouse Cloud service and target table, (2) connect ClickPipes to your Kafka cluster, (3) define how messages map into columns, and (4) let ClickPipes continuously ingest data while you query it in real time.

1. Prepare ClickHouse Cloud and the target table

Create or use an existing ClickHouse Cloud service.
- In the ClickHouse Cloud console, create a service in your preferred region and cloud (AWS, GCP, or Azure).
- Ensure it has network connectivity to your Kafka cluster (public, VPC peering, or private link).
Create a database and table for Kafka data.
Open the built‑in SQL console (or connect via clickhouse-client) and define the table that ClickPipes will write to. For example, for an events topic:
```
CREATE DATABASE IF NOT EXISTS kafka_demo;

CREATE TABLE kafka_demo.events
(
    event_time   DateTime64(3, 'UTC'),
    event_type   LowCardinality(String),
    user_id      String,
    properties   JSON,
    source_topic String
)
ENGINE = MergeTree
PARTITION BY toDate(event_time)
ORDER BY (event_time, event_type, user_id);
```
- MergeTree is the workhorse engine for analytics—columnar, compressed, and optimized for large scans.
- Partitioning by date is a lifecycle choice (retention, backups), not a generic performance hack. Keep partition cardinality low (hundreds–low thousands of partitions) to avoid too many small parts.

2. Create a ClickPipe for Kafka in the Cloud console

Navigate to ClickPipes.
- In the ClickHouse Cloud UI, open your service.
- In the left navigation, select ClickPipes → Create ClickPipe.
Choose the Kafka source.
You’ll see options like:
- Kafka (Confluent Cloud) – recommended if you’re using Confluent Cloud; it handles auth and schema registry cleanly.
- Kafka (Self‑managed) – if you run your own Kafka on Kubernetes/VMs.
Enter connection details.
Depending on your Kafka provider, you’ll specify:
- Bootstrap servers (e.g. pkc-xxxxx.us-central1.gcp.confluent.cloud:9092)
- Security protocol / auth:
  - SASL/SSL with username/password (e.g., Confluent Cloud API key/secret)
  - TLS settings (CA cert if required)
- Topics to subscribe to:
  - Single topic: events
  - Multiple topics: events, clicks, impressions
  - Optional pattern depending on the UI (e.g., events-*).
Ensure the ClickHouse Cloud service can reach those bootstrap endpoints (via public internet or peered VPC).

3. Configure schemas and message format

ClickPipes needs to understand how to decode Kafka messages and map them to table columns. You’ll configure:

Message format.
Typical options include:
- Avro / Protobuf with schema registry (strongly recommended in Confluent environments).
- JSON for schemaless payloads or simple event structures.
- CSV where messages are simple delimited rows.
If you’re using Confluent’s JDBC connector or schema-aware tooling, note that:
- A schema is required for the JDBC Connector (you cannot use plain JSON/CSV with JDBC).
- It’s strongly advised to use the Confluent Schema Registry so schema info isn’t redundantly encoded in every message. This reduces payload size and parsing overhead.
Schema registry (if applicable).
For Avro/Protobuf:
- Provide Schema Registry URL (e.g. https://psrc-xxxxx.us-central1.gcp.confluent.cloud).
- Provide Schema Registry credentials (API key/secret or auth token).
- ClickPipes will automatically fetch schemas and evolve with compatible changes.
Field-to-column mapping.
In the ClickPipes UI, map message fields to ClickHouse columns, e.g.:
- payload.timestamp → event_time
- payload.type → event_type
- payload.user.id → user_id
- payload.properties → properties
- kafka.topic metadata → source_topic
For JSON payloads, you can land the entire JSON into a JSON or String column and later use functions like JSONExtractString or JSONExtract in queries.

Example query using a JSON column:
```
SELECT
    event_type,
    JSONExtractString(properties, 'campaign_id') AS campaign_id,
    count(*) AS events
FROM kafka_demo.events
WHERE event_time >= now() - INTERVAL 5 MINUTE
GROUP BY event_type, campaign_id
ORDER BY events DESC
LIMIT 50;
```

4. Choose consumer behavior and offsets

Starting point:
Decide where to start reading:
- Earliest – backfill from the beginning of the topic (good for historical analytics).
- Latest – only new events (good for forward-only streaming).
Consumer group and offsets:
- ClickPipes typically uses its own consumer group; offsets are managed so ingestion resumes cleanly after restarts.
- The system is designed to avoid duplicates; when in doubt, you can deduplicate in ClickHouse using primary key or GROUP BY semantics depending on your model.
Throughput and batching:
ClickPipes internally batches inserts into ClickHouse. For healthy MergeTree operation:
- Aim effectively for 1,000–100,000 rows per insert equivalent.
- Avoid a pattern that produces many tiny parts; otherwise you risk “Too many parts” errors and increased merge pressure.
You can always inspect ingestion behavior with:
```
SELECT
    table,
    count() AS active_parts,
    sum(rows) AS total_rows,
    sum(bytes_on_disk) AS bytes_on_disk
FROM system.parts
WHERE database = 'kafka_demo'
  AND table = 'events'
  AND active
GROUP BY table;
```

5. Map to the target ClickHouse table and start the ClickPipe

Select target database and table.
- In the ClickPipes wizard, choose kafka_demo.events (or your equivalent).
- Confirm that datatypes and nullability match what’s coming from Kafka.
Review and create.
- The UI will show a summary: source topics, schema mapping, target table, and estimated costs.
- Click Create / Start to activate the ClickPipe.

Validate ingestion.
Watch data land in real time:

SELECT
    max(event_time) AS latest_event,
    count() AS last_5m_rows
FROM kafka_demo.events
WHERE event_time >= now() - INTERVAL 5 MINUTE;

If you see latency building, inspect merge and query health:

SELECT
    table,
    sum(total_rows) AS rows_in_merges,
    count() AS active_merges
FROM system.merges
WHERE database = 'kafka_demo'
GROUP BY table;

6. What it costs per GB in ClickHouse Cloud

ClickPipes for Kafka is billed as a usage-based add‑on on top of your ClickHouse Cloud service. While exact rates can change and vary by region/plan, the billing model is:

Metering unit:
- GB of data processed by ClickPipes from Kafka, typically based on compressed payload size “on the wire,” not the expanded size in ClickHouse.
What’s included:
- Managed ingestion pipeline, scaling, error handling, and offset tracking.
- Integration with ClickHouse Cloud monitoring and logging.
What’s separate:
- Compute: your ClickHouse Cloud service tier (e.g., dedicated vs. serverless).
- Storage: compressed data at rest in ClickHouse (with ClickHouse’s best‑in‑class compression).

To get the exact per‑GB price for your account:

Go to the ClickHouse Cloud pricing page or your organization’s billing panel.
Look for ClickPipes / Ingestion pricing under your chosen region and deployment model.
Confirm:
- Per‑GB rate for Kafka ingestion.
- Any volume discounts or committed‑use options.

A simple way to rough‑estimate monthly cost:

Monthly GB through ClickPipes = (Average MB/s from Kafka) × 3600 × 24 × 30 / 1024

Monthly ingestion cost ≈ Monthly GB × Per‑GB ClickPipes rate

Then add:

ClickHouse Cloud compute hours (or serverless consumption).
Storage (compressed; often 3–10× smaller than source, depending on schema and compression).

Common Mistakes to Avoid

Assuming JSON-only ingestion is enough for JDBC and schema-heavy workloads:
If you’re using Kafka Connect JDBC or Confluent’s ecosystem, remember the connector requires a schema; plain JSON/CSV won’t work there. Prefer Avro/Protobuf with Schema Registry to keep payloads lean and evolution safe.
Over-partitioning tables based on Kafka topic or high-cardinality fields:
It’s tempting to partition by user_id or topic name for “performance,” but partitioning is primarily a lifecycle tool. High-cardinality partitions and too many parts will slow merges and hurt queries. Stick to low-cardinality choices like date, with fewer than ~100–1,000 distinct partition values for most workloads.

Real-World Example

In a previous role, we moved our user activity stream—about 200k events per second—off a custom Kafka ingestion service into ClickPipes feeding ClickHouse Cloud. The old pipeline was a mix of Kafka Streams and ad‑hoc consumers writing into an OLTP database plus a separate warehouse. Queries over recent activity took tens of seconds and we constantly fought back-pressure in the ingestion layer.

With ClickPipes:

We created a dedicated activity table in ClickHouse Cloud, partitioned by toDate(event_time) and ordered by (user_id, event_time).
Configured a Confluent Cloud Kafka source in ClickPipes, with Avro + Schema Registry and an “earliest” start to backfill 30 days of history.
Let ClickPipes handle consumption, batching, and retries; no more consumer deployment headaches.

Once backfill caught up, product and SRE teams were running millisecond‑latency queries on tens of billions of events:

SELECT
    user_id,
    count(*) AS events_last_10m
FROM activity.events
WHERE event_time >= now() - INTERVAL 10 MINUTE
GROUP BY user_id
ORDER BY events_last_10m DESC
LIMIT 100;

Operationally, we tracked MergeTree health with system.parts and system.merges and tuned only the ClickHouse service, not ingestion infrastructure. Ingestion costs were driven by a clear per‑GB rate plus our ClickHouse Cloud service, which was significantly cheaper than running a fleet of ingestion microservices and a heavyweight data warehouse.

Pro Tip: Start with a non-production Kafka topic or a subset of partitions, monitor system.parts and system.query_log in ClickHouse for a few days, and only then scale up to full topic volume. This lets you validate batching behavior and merge pressure before committing to full ingestion spend.

Summary

Using ClickPipes to ingest Kafka topics into ClickHouse Cloud gives you a fully managed, ClickHouse-native path from streaming data to millisecond analytics over billions of rows. You configure Kafka connectivity, message formats, and schema mapping in the Cloud console, point at a MergeTree table, and let ClickPipes handle the heavy lifting of consumption, batching, and fault tolerance. Costs are simple and usage-based: a per‑GB ingestion rate for data processed through ClickPipes, plus your ClickHouse Cloud compute and storage, making it straightforward to forecast spend as your Kafka traffic grows.

Next Step

Get Started

Answers you can trust, from Codeables