What’s the best way to keep years of telemetry online without my database storage bill exploding?

Most teams hit the same wall: you want years of telemetry online for analytics, auditing, and AI workloads—but the moment you try to keep it all “hot” in your primary database, storage costs spike, queries slow down, and retention becomes a game of compromise. The good news is you don’t need a data lake, a Kafka zoo, and a separate analytics warehouse to fix this. You can keep years of telemetry online in Postgres, control your storage bill, and still serve real-time queries—if you use the right storage primitives.

Quick Answer: The best way to keep years of telemetry online without your database storage bill exploding is to combine time- and key-based partitioning, compression, and tiered storage—so recent data stays in high-performance storage, older data moves to low-cost object storage, and you still query everything from the same Postgres tables.

The Quick Overview

• What It Is: A PostgreSQL-native telemetry stack—TigerData (Postgres + TimescaleDB on Tiger Cloud)—that uses hypertables, compression, and tiered storage to store years of time-series/event data at a fraction of the typical cost.
• Who It Is For: Teams ingesting high-volume metrics, IoT telemetry, logs, trading ticks, or event data that need both real-time queries and long-term history, without building fragile, multi-system pipelines.
• Core Problem Solved: Plain Postgres gets slow and expensive when you try to keep years of high-ingest telemetry online. TigerData restructures how that data is stored so you get predictable performance and “bottomless” history—without a runaway storage bill.

How It Works

The core idea: keep Postgres as your system of record but change how telemetry is stored and managed under the hood. TigerData does this with a set of primitives:

• Hypertables for automatic time + key partitioning
• Hypercore row-columnar storage for write-optimized rows + compressed, analytics-friendly columnar chunks
• Tiered storage that automatically moves older chunks to low-cost object storage while keeping them queryable
• Policies for retention, compression, and rollups (continuous aggregates) so storage stays under control automatically

From your application’s perspective, you still write SQL to a Postgres table. Underneath, TigerData turns that table into a hypertable, manages chunking, compresses historical data, and pushes cold chunks to inexpensive object storage—so you can keep years of telemetry online at a fraction of the cost of keeping it all on high-performance disks.

1. Ingest & Partition (Hypertables):
   You define a hypertable on your telemetry schema. TimescaleDB automatically partitions data by time and (optionally) another dimension, such as device_id, tenant_id, or customer_id. Each time slice becomes a “chunk,” which is independently stored, indexed, and managed.

2. Compress & Optimize (Hypercore Columnstore):
   As chunks age, TigerData applies compression policies. Historical telemetry is converted from row-store to compressed columnar storage, delivering up to 98% compression and speeding up scan-heavy analytics queries. Recent data stays in a write-optimized row layout for real-time inserts and queries.

3. Tier & Scale (Bottomless Storage):
   Older chunks are automatically moved (“tiered”) off expensive high-performance storage into low-cost object storage (S3 on Tiger Cloud). You still query the same hypertable; TimescaleDB and TigerData fetch and process data from both tiers as needed. Storage is billed by actual usage, expanding as data grows and contracting when you prune history—no massive pre-allocated volumes.

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Hypertables & Automatic Partitioning	Replace a single monolithic table with time- and key-based chunks managed by TimescaleDB.	Keeps ingest fast and predictable even with billions of rows; avoids index bloat and vacuum storms.
Hypercore Compression & Columnstore	Converts older chunks into compressed columnar storage, while keeping recent data in row format.	Reduces storage by up to 98% and accelerates analytical queries over large time ranges.
Tiered Storage (“Bottomless” History)	Stores frequently accessed data in high-performance storage and automatically moves older data to low-cost S3, still queryable.	Keeps years of telemetry online without paying high-performance disk prices for cold data.

Ideal Use Cases

• Best for high-volume telemetry (metrics, IoT, logs, ticks): Because it lets you ingest millions of rows per second, keep multi-year history, and still run real-time queries—all from a single Postgres interface.
• Best for teams tired of “glue” (Kafka + Flink + warehouse + lake): Because TigerData replaces fragile, high-maintenance pipelines with native Postgres primitives: hypertables, compression, tiered storage, and built-in lakehouse integration.

Why plain Postgres struggles with years of telemetry

If you’ve tried to keep years of telemetry in standard Postgres, you’ve likely seen:

• Massive tables (hundreds of millions to billions of rows)
• Indexes that balloon until they are larger than the base table
• Vacuum/auto-vacuum contention and unpredictable query latencies
• Manual partitioning and complex retention scripts
• Storage costs that climb linearly with every month of retained data

The underlying reasons:

1. Monolithic heap tables:
   A single big table is a bad fit for write-heavy, time-ordered data. Every insert competes on the same structure; every query has to consider a huge range of pages.

2. Index bloat over time:
   Telemetry workloads often index on timestamp plus one or more keys (device, tenant, etc.). Over time, these indexes grow and fragment, and maintaining them at ingest time gets expensive.

3. No storage lifecycle awareness:
   Postgres doesn’t know that last week’s telemetry is “colder” than this minute’s; it treats all pages and indexes the same. You end up paying the same storage rate for a 5-minute window as for a 5-year window.

4. Coarse-grained retention:
   Dropping data is usually done via DELETE on large ranges, causing massive dead tuples, vacuum load, and extended locks—or via brittle, manual partition rotation.

The result is predictable: to keep performance, you aggressively shorten retention. To keep retention, you live with slow queries and mounting storage bills.

How TigerData changes the calculus

TigerData keeps Postgres and SQL at the center but adds the primitives you need for telemetry scale. Here’s how those features work in practice.

1. Hypertables: time- & key-aware partitioning

You start by modeling your telemetry as you normally would in Postgres:

CREATE TABLE device_readings (
    time        timestamptz       NOT NULL,
    device_id   text              NOT NULL,
    metric      text              NOT NULL,
    value       double precision  NOT NULL,
    metadata    jsonb,
    PRIMARY KEY (time, device_id, metric)
);

Then you convert it into a hypertable:

SELECT create_hypertable(
    'device_readings',
    by_range('time'),
    partitioning_column => 'device_id',
    number_partitions   => 16
);

What this does:

• Splits data into time-based chunks (e.g., daily or weekly), each a physical child of the hypertable.
• Optionally partitions by a second dimension (device_id here) to spread ingest load.
• Keeps the logical illusion of a single table—your queries remain:

SELECT
  device_id,
  time_bucket('1 hour', time) AS hour,
  avg(value) AS avg_value
FROM device_readings
WHERE time >= now() - interval '30 days'
GROUP BY device_id, hour;

Why it matters for storage and cost:

• Chunk-level operations mean you can drop, compress, or move entire chunks instead of row-by-row deletes.
• Indexes are local to chunks, so they don’t grow without bound.
• Queries restricted by time automatically touch only the relevant chunks, dramatically reducing I/O.

2. Compression & Hypercore: shrink history by up to 98%

Once data is older than your “write-hot” window, you can enable compression on the hypertable:

ALTER TABLE device_readings
SET (
  timescaledb.compress = true
);

SELECT add_compression_policy(
  'device_readings',
  compress_after => INTERVAL '7 days'
);

Mechanism:

• Recent chunks (e.g., last 7 days) stay in row-store: ideal for fast inserts and low-latency reads.
• When a chunk ages past compress_after, TimescaleDB:
  • Converts it into Hypercore’s columnar layout.
  • Applies dictionary, run-length, and delta encoding.
  • Stores compressed columns in separate storage structures, still behind the hypertable.

Practical outcomes:

• Storage savings: Compression often yields up to 98% reduction for telemetry—especially for normalized schemas and repeated keys/metrics.
• Analytics speed: Columnar layout is ideal for scans and aggregations. Queries like “7-day avg per metric per device” become faster, not slower, even with more history.
• Transparent access: Your queries don’t change. TimescaleDB decides whether to read compressed or uncompressed chunks.

Note: Compression is chunk-based. You still control which columns to segment, order by, and exclude if you need to tune for specific query patterns.

3. Tiered storage: keep everything online, but stop paying “hot” prices

Compression solves a large part of the storage bill, but at multi-year scale, even compressed data adds up. On Tiger Cloud, TigerData adds tiered storage:

• High-performance storage: For recent and frequently accessed chunks. Optimized for low-latency reads and writes.
• Low-cost object storage (S3): For older, less frequently accessed chunks. Optimized for cost and durability.

From the docs:

Recent, high-velocity data stays in high-performance storage for ultra-fast queries.
Older, less frequently accessed data is automatically moved to cost-efficient object storage but remains queryable and available for building continuous aggregates.

Important: While many systems support data “cooling,” TimescaleDB keeps data queryable from the same hypertable regardless of where it physically lives. You don’t need separate tables, schemas, or connectors.

On Tiger Cloud:

• High-performance storage supports up to 64 TB of compressed data (typically 80–100 TB uncompressed) per service.
• Data is billed on average GB consumption per hour, so you only pay for what you actually use.
• Bottomless storage uses S3 under the hood, with no extra data transfer or compute charges for data in the low-cost tier.

4. Lifecycle policies: automate retention, compression, and aggregates

The goal is a system that manages itself:

• Compression policies to shrink older chunks.
• Retention policies to drop data you no longer need.
• Continuous aggregates to maintain rollups so you can safely drop granular history.

Example: keep 90 days of raw data, compress after 7 days, and maintain hourly rollups for 5 years.

-- Compress chunks older than 7 days
SELECT add_compression_policy(
  'device_readings',
  compress_after => INTERVAL '7 days'
);

-- Keep only 90 days of raw telemetry
SELECT add_retention_policy(
  'device_readings',
  drop_after => INTERVAL '90 days'
);

-- Create an hourly rollup
CREATE MATERIALIZED VIEW device_readings_hourly
WITH (timescaledb.continuous) AS
SELECT
  time_bucket('1 hour', time) AS bucket,
  device_id,
  metric,
  avg(value) AS avg_value,
  min(value) AS min_value,
  max(value) AS max_value
FROM device_readings
GROUP BY bucket, device_id, metric;

-- Keep 5 years of hourly rollups
SELECT add_retention_policy(
  'device_readings_hourly',
  drop_after => INTERVAL '5 years'
);

Why this helps your storage bill:

• Raw data is only kept as long as you truly need it at full granularity.
• Long-term history lives in a compressed, aggregated form that’s orders of magnitude smaller.
• Retention is chunk-based and low-impact; you’re dropping entire compressed chunks, not deleting rows.

5. Independent compute & storage scaling

Another pain point with typical database deployments is pre-allocating large volumes “just in case.” On Tiger Cloud:

• Storage expands and contracts automatically as you add or delete data—no manual resizing.
• Usage-based billing ties cost to actual storage consumption, not guessed capacity.
• Compute scales independently (e.g., larger instances, read replicas) so you can optimize query performance without over-paying for storage, or vice versa.

This matters when you’re retaining years of telemetry: you might have modest query needs most of the time but need bursty compute for specific analytics windows, without wanting to resize storage.

Limitations & Considerations

• Cold data is still online, but access patterns matter:
  While tiered storage keeps cold data queryable from the same hypertable, scanning huge ranges of multi-year data on every dashboard will still incur latency and compute cost. Use time filters, continuous aggregates, and pre-computed rollups for dashboards and frequent queries.

• Schema design still matters:
  Hypertables, compression, and tiering are powerful, but they don’t fix a poor schema. Normalize where appropriate, index for your main query patterns, and use time_bucket, device_id/tenant_id filters, and proper primary keys. TigerData’s primitives amplify a good schema; they can’t fully rescue an unbounded “catch-all JSON” schema.

Pricing & Plans

TigerData’s pricing is designed to be predictable and transparent:

• Storage is metered on average GB consumption per hour.
• High-performance storage covers up to 64 TB compressed (typically 80–100 TB uncompressed) per service.
• Low-cost tiered storage is billed based on the original uncompressed size, with no extra data transfer or compute fees for data in the low-cost tier.
• There are no per-query charges and no separate backup fees; automated backups are included.

Plans (at a high level):

• Performance: Best for teams needing fast ingest and queries on recent telemetry, with growing history and basic HA, and wanting usage-based costs that track storage and compute.
• Scale / Enterprise: Best for organizations running mission-critical telemetry workloads at large scale (multi-TB to multi-PB), needing multi-AZ HA, advanced security (SOC 2, GDPR, HIPAA on Enterprise), VPC peering, and 24/7 support with SLAs.

For precise numbers, regions, and plan details, TigerData’s console and pricing pages itemize storage, compute, and optional features so you can project costs before scaling up.

Frequently Asked Questions

How much telemetry can I realistically keep online with TigerData before costs get out of hand?

Short Answer: With hypertables, compression, and tiered storage, it’s common to keep years of telemetry online while staying within tens of terabytes of compressed high-performance storage—and pushing everything older to very cheap object storage.

Details:
Compression alone can reduce storage usage by up to 98% for well-modeled telemetry. For example:

• 100 TB of raw telemetry
• → ~2–5 TB after compression (workload-dependent)
• → high-performance storage up to 64 TB compressed per service
• → older chunks automatically moved to low-cost S3, with billing based on original uncompressed size but at significantly lower rates than hot storage.

Because Tiger Cloud storage is usage-based and auto-scaling, you don’t have to guess capacity years in advance; you add telemetry and let compression + tiering manage the cost curve.

Do I need Kafka, Flink, and a separate data lake to manage long-term telemetry?

Short Answer: No. You can ingest into Postgres (TigerData), keep years of data online, and integrate with your lakehouse when needed—without building fragile pipelines.

Details:
TigerData’s philosophy is to avoid the “stitched together Kafka, Flink, and custom code” pattern as the default:

• You ingest telemetry directly into Postgres/TimescaleDB.
• Hypertables, compression, tiering, and continuous aggregates handle lifecycle and performance.
• When you do need a lakehouse, TigerData supports native lakehouse integration to ingest from Kafka/S3 and replicate to Iceberg so you can stream into your warehouse or data lake without a bespoke set of scripts and jobs.

For many workloads, Postgres + TigerData is both the system of record and the analytics engine, which means fewer moving parts and fewer places where pipelines break at 3 a.m.

Summary

Keeping years of telemetry online doesn’t have to mean an exploding database storage bill, or a brittle mess of services. The key is to change how telemetry is stored and managed while keeping Postgres and SQL at the center:

• Hypertables to partition by time and key automatically.
• Hypercore compression & columnstore to shrink history by up to 98% and speed up analytics.
• Tiered storage to keep cold data online in low-cost object storage while leaving hot data on high-performance disks.
• Lifecycle policies for compression, retention, and continuous aggregates so the system manages itself.

TigerData packages these primitives into Tiger Cloud, with usage-based billing, independent compute and storage scaling, and production-ready operations (HA, backups, PITR, security controls). The result: you can keep the telemetry you actually need—years of it—without treating your database storage bill as an uncontrollable line item.

Next Step

Get Started