How do I configure tiered storage (SSD + S3) in TigerData Tiger Cloud for long retention?

Long retention on telemetry data only works if you can keep hot data fast and cold data cheap. In Tiger Cloud, that’s exactly what tiered storage gives you: high-performance SSD for recent, high-velocity data, and low-cost S3 object storage for older chunks—without changing your SQL or rewriting your app.

The Quick Overview

• What It Is: Tiger Cloud tiered storage automatically moves older time-series data from high-performance SSD to low-cost, “bottomless” S3, while keeping it queryable through the same Postgres interface.
• Who It Is For: Teams running long-retention telemetry (metrics, events, IoT, tick data) that need sub-second queries on recent data and economical storage for months or years of history.
• Core Problem Solved: Plain Postgres either becomes too expensive (all data on SSD) or too slow / complex (DIY archiving, separate lake). Tiered storage keeps ingest and queries fast while pushing cold data to S3 and letting you keep long retention at predictable cost.

How It Works

Tiger Cloud services use TimescaleDB hypertables and time-based chunking under the hood. Tiered storage adds another layer: chunks age out of the SSD tier and are transparently offloaded to S3-backed storage. You still query them with standard SQL; the system handles where the bytes actually live.

At a high level:

1. Hot data on high-performance SSD:
   New data lands on high-performance storage. This tier is optimized for high ingest and low-latency queries over your active window (typically days to weeks, depending on workload and cost targets).

2. Automatic tiering to S3 “bottomless” storage:
   As chunks age past your configured window and are less frequently accessed, Tiger Cloud automatically tiers them to low-cost object storage (S3). Billing for this tier is based on original uncompressed size, with no extra data transfer or compute fees.

3. Transparent access over the full retention period:
   Queries that touch long-retention data still run through the same Postgres endpoint. You keep a single Tiger Cloud service instead of juggling operational Postgres, Kafka, and a separate data lake.

Note: Tiered storage is configured at the service level and works with your hypertables’ time-based partitioning. You do not change your SQL client, driver, or schema to take advantage of S3 tiering.

Step-by-Step: Configuring Tiered Storage in Tiger Cloud

Because the console UX evolves, think about this in two layers:

• Service-level storage configuration (SSD capacity, tiering behavior)
• Schema-level design (using hypertables so tiering can work effectively)

1. Size your high-performance SSD tier

Your SSD tier is where recent, high-velocity data lives. This is the “fast lane” for:

• Live dashboards
• Alerting and anomaly detection
• Operational queries against the last hours/days/weeks

Tiger Cloud:

• Offers up to ~64 TB of compressed data in the high-performance tier, which typically corresponds to 80–100 TB uncompressed.
• Bills storage on average GB consumption per hour, not provisioned capacity.
• Does not charge extra for backups, data transfer, or compute for tiered data.

To size this tier:

1. Estimate your ingest rate:

   • Events per second (EPS) or metrics per second.
   • Average row size (bytes).
   • Daily volume = EPS * row_size * 86,400.

2. Choose a hot window:

   • Common choices: 7, 14, or 30 days of data in SSD.
   • Multiply daily volume by that hot window to get SSD target usage.

3. Add headroom:

   • Aim to run the SSD tier at 60–80% of its practical capacity to avoid surprise pressure if ingest spikes.

Important: You only pay for what you use, not a fixed SSD allocation. Still, overfilling your hot tier will impact performance; keep a clear margin based on monitoring.

2. Enable tiered storage for your Tiger Cloud service

Within Tiger Console:

1. Create or edit a Tiger Cloud service in your preferred region.
2. In the storage or tiering section, ensure that:
   • High-performance SSD storage is enabled (default).
   • Low-cost S3 tier is enabled for older data (this is what makes storage “bottomless”).
3. Confirm that the service is running with TimescaleDB (all Tiger Cloud telemetry services are) so hypertables are available.

Implementation details may surface in UI as:

• A toggle for “Tier older data to S3” or similar.
• Configuration for SSD target, retention, or tiering policy.

Note: Tiered storage is a Tiger Cloud capability. If you run TimescaleDB on your own hardware or a generic cloud VM, you don’t get managed S3 tiering out-of-the-box; you’d need to build your own archiving pipeline.

3. Use hypertables so data can be tiered efficiently

Tiered storage builds on the TimescaleDB hypertable abstraction:

• Chunks are partitioned by time (and optionally a key).
• Compression and tiering operate at the chunk level.
• As chunks age, they are natural candidates for compression and S3 tiering.

Basic pattern for an ingest table:

CREATE TABLE readings (
    device_id    text        NOT NULL,
    time         timestamptz NOT NULL,
    value        double precision,
    status       text,
    tags         jsonb
);

SELECT create_hypertable(
    'readings',
    'time',
    chunk_time_interval => interval '1 day'
);

Important: Always partition by a time column (time, ts, etc.). Tiered storage depends on time-based chunks aging out of the SSD tier predictably.

4. Configure compression for older chunks (strongly recommended)

Compression is not strictly required for tiered storage, but it is highly recommended:

• TigerData can help you compress data by up to 98%, directly reducing SSD and S3 storage costs.
• Compressed chunks are more efficient to move and scan, especially for analytical queries.

Enable compression on a hypertable:

ALTER TABLE readings
SET (
    timescaledb.compress = true
);

Set a compression policy so that older chunks are automatically compressed, for example after 7 days:

SELECT add_compression_policy(
    'readings',
    INTERVAL '7 days'
);

Now, your table will behave like this:

1. Days 0–7: Uncompressed chunks in SSD → fastest writes and reads.
2. After 7 days: Chunks get compressed (still on SSD).
3. As chunks age further: Tiger Cloud automatically tiers them off to low-cost S3 storage.

Note: Compression and tiering are complementary. Compression shrinks the data; tiered storage chooses which media (SSD vs S3) holds it. Both work together to reduce cost while preserving queryability.

5. Set retention expectations and query windows

You typically want different behaviors for different tables:

• Raw, high-volume telemetry: Long retention, heavily compressed, mostly cold but still queryable.
• Aggregates / rollups: Smaller volume, kept hot for longer to serve operational dashboards.

Example: 2-year raw history, 30-day hot window:

• Keep last 30 days hot on SSD (for frequent queries).
• Push data older than 30 days to S3, but keep full 2-year retention queryable.

Schema pattern:

-- Raw telemetry
CREATE TABLE raw_metrics (
    source_id   text        NOT NULL,
    time        timestamptz NOT NULL,
    metric      text        NOT NULL,
    value       double precision NOT NULL
);

SELECT create_hypertable('raw_metrics', 'time');

ALTER TABLE raw_metrics
SET (timescaledb.compress = true);

SELECT add_compression_policy(
    'raw_metrics',
    INTERVAL '7 days'
);

-- Optionally, prune very old data if you don't truly need infinite retention:
SELECT add_retention_policy(
    'raw_metrics',
    INTERVAL '2 years'
);

Important: Tiered storage focuses on where data lives (SSD vs S3). Retention policies decide if extremely old data is eventually deleted. Use both to match your compliance needs and cost envelope.

6. Monitor storage usage in Tiger Console

Tiger Cloud exposes storage metrics per service:

• Total storage used
• Usage in high-performance tier
• Usage in low-cost tier
• Compression ratios

Use these to:

• Validate that your hot window and compression policies are behaving as expected.
• Tune chunk_time_interval, compression delays, and retention windows.
• Keep the SSD tier within your target usage band.

Note: Because storage is billed on average GB consumption per hour (and not provisioned capacity), you can iterate on policies without incurring the “overprovisioned” tax you’d see in a fixed-volume setup.

How Tiered Storage Behaves in Queries

From the application point of view, tiered storage is invisible:

• You still connect to the same Postgres endpoint.
• You still run the same SQL.
• Query planners are aware of chunk metadata and will push down filters to avoid scanning unnecessary data.

Differences to be aware of:

• Latency: Queries that span very old data (primarily on S3) can be slower than queries that hit only hot SSD chunks. This is expected; use targeted WHERE clauses and indexes.
• Workload design: Use continuous aggregates for always-fresh rollups over long windows so dashboards don’t repeatedly scan raw cold data.

Example: continuous aggregate over 1-minute rollups for a 1-year window:

CREATE MATERIALIZED VIEW metric_1min
WITH (timescaledb.continuous) AS
SELECT
    time_bucket('1 minute', time) AS bucket,
    source_id,
    metric,
    avg(value) AS avg_value,
    max(value) AS max_value
FROM raw_metrics
GROUP BY 1, 2, 3;

SELECT add_continuous_aggregate_policy(
    'metric_1min',
    start_offset => INTERVAL '2 days',
    end_offset   => INTERVAL '1 minute',
    schedule_interval => INTERVAL '1 minute'
);

With this setup:

• Dashboards query metric_1min (mostly hot data).
• Raw history can live compressed on S3 for long retention and occasional forensic analysis.

Limitations & Considerations

• Latency for deep history:

  • Limitation: Queries that scan many months/years of history will be slower, especially if they require full scans of cold data.
  • Workaround: Use continuous aggregates and targeted WHERE clauses; design queries to work on rollups rather than raw points whenever possible.

• Requires hypertable design and policies:

  • Limitation: Tiered storage relies on time-based partitioning and chunk-level policies. A plain, non-hypertable table won’t benefit from it.
  • Workaround: Migrate large telemetry tables to hypertables, enable compression, and add retention and aggregate policies.

• Operational visibility:

  • Be explicit about hot vs cold usage with your stakeholders. Some workloads (like tight SLO analytics) may need a broader hot window on SSD, which has cost implications.

Pricing & Plans

Tiger Cloud’s storage model is built to avoid surprises:

• High-performance storage (SSD):
  • Up to ~64 TB compressed (typically 80–100 TB uncompressed) per service.
  • Billed on average GB consumption per hour.
• Low-cost S3 storage (tiered):
  • Billed based on original uncompressed size of data moved from the high-performance tier.
  • No additional charges for data transfer or compute for tiered data.
• No hidden extras:
  • You don’t pay extra for automated backups.
  • There are no per-query fees.
  • Storage billing is transparent and itemized in Tiger Console.

Plan guidance (names are representative; check the website for exact SKUs):

• Performance / Standard:
  Best for teams that need fast ingest, a moderate hot window, and long retention with tiered storage, but don’t need advanced compliance (HIPAA) yet.

• Scale / Enterprise:
  Best for teams with large multi-petabyte telemetry footprints, strict SLAs, and compliance requirements. Offers larger SSD tiers, stronger support SLAs, SOC 2 Type II reporting, and HIPAA support on Enterprise.

Note: The fastest way to size your storage needs is to sign up for a free Tiger Cloud trial and run a realistic workload. Ingest your actual telemetry, watch storage usage, and tune chunk/retention policies before going to production.

Frequently Asked Questions

Does tiered storage change my SQL or require a new API?

Short Answer: No. You keep the same Postgres SQL interface.

Details: Tiered storage is a Tiger Cloud infrastructure feature, not a new query engine. Your applications still connect over standard Postgres drivers. You still create tables, indexes, and hypertables with SQL. Tiering chooses the physical storage tier for chunks; the logical schema and query surface remain unchanged. That’s the core TigerData philosophy: keep Postgres as the boring, reliable foundation and extend it for telemetry scale.

What happens when I hit the SSD capacity—do I lose data?

Short Answer: No, data is not dropped; older chunks are compressed and tiered to S3.

Details: Because Tiger Cloud storage is billed on actual usage (not fixed volumes), you don’t “hit a hard wall” in the traditional sense. Instead, you:

• Configure policies (compression, retention) and rely on Tiger Cloud’s tiering to push older chunks to low-cost S3.
• Keep all data accessible unless you explicitly add a retention policy that deletes data past a certain age.
• Avoid surprise truncation events or manual archiving scripts.

Operationally, you should still monitor SSD utilization via Tiger Console. If your hot window grows or ingest spikes, you can adjust policies or scale the service to maintain the performance envelope you expect.

Summary

Configuring tiered storage for long retention in Tiger Cloud is mostly about designing your schema and policies correctly:

1. Size your high-performance SSD tier for your hot window.
2. Enable tiering to S3 at the service level.
3. Use hypertables for all high-ingest telemetry tables.
4. Turn on compression and set compression + retention policies that match your business and compliance needs.
5. Use continuous aggregates for operational queries so deep history can safely live on S3.

You get a single, Postgres-native system that ingests at telemetry speed, keeps hot data fast on SSD, and pushes cold data to cheap, bottomless S3—without fragile, high-maintenance ETL pipelines.

Next Step

Get Started