Time series data consists of measurements or observations recorded at specific timestamps, often at high frequency (e.g., CPU usage, IoT sensors).
Use cases include monitoring server metrics (CPU, memory, disk I/O), alerting systems, financial projections, and telemetry from IoT devices.
Time series databases are specialized to handle:
Why Relational Databases Struggle with Time Series Data
Example scenario: 100,000 servers reporting 6 metrics every 60 seconds → up to 600,000 writes per second.
Relational DBs typically max out around 10,000 writes per second without heavy sharding.
Storage inefficiency due to large timestamp and string columns.
Querying billions of rows is slow, often taking minutes instead of milliseconds.
Relational databases are not optimal for time series workloads due to write and read inefficiencies.
Core Techniques Used in Time Series Databases
Technique
Purpose
Description
Append-only storage
Maximize write throughput
Data is written sequentially to disk, avoiding costly random seeks on spinning disks/SSDs.
Log-Structured Merge (LSM) Trees
Organize data for efficient reads and writes
Use write-ahead logs and in-memory memtables to batch, sort, and merge data before flushing to disk. Offline compaction merges segments to optimize storage.
Delta Encoding
Data compression
Store differences between successive timestamps or values to reduce byte size of data.
Delta-of-delta Encoding
Timestamp compression
Store differences of timestamp differences to exploit regular intervals and compress further.
Block-level Metadata
Fast query filtering
Store min/max values, time ranges, and tag information per block to skip irrelevant data during queries.
Bloom Filters
Space-efficient membership tests
Probabilistic data structure to quickly check if a tag or host exists in a block with minimal storage.
Downsampling
Reduce storage and speed up queries on old data
Aggregate older data into coarser granularity (e.g., 10s → 1 min → 1 hour) for efficiency.
Architecture & Data Flow Overview
Incoming data points have a metric name, tags (e.g., host, region), fields/values, and a timestamp.
Data is first written to a write-ahead log (WAL) for durability.
Then stored in an in-memory memtable, where data is sorted and organized by unique time series (metric + tags).
Periodically, memtables are flushed as contiguous segments/files to disk, employing compression (delta encoding, etc.).
Each segment has an index to map series and tags for fast lookups.
Queries filter by time ranges and tags, use indexes and metadata to avoid scanning irrelevant files, decompress relevant blocks, and apply aggregations (e.g., average CPU usage).
Limitations and When to Avoid Time Series Databases
TSDBs excel with a small to moderate number of series (thousands to millions), where queries focus on a few series over time.
High cardinality (very large numbers of unique series) breaks the model:
No implicit time dimension or filtering also reduces TSDB efficiency.
For very high cardinality or non-time-centric data, general-purpose databases like PostgreSQL or Redis may be better.
Practical Examples of Time Series Databases
Database
Description
Use Case Focus
TimescaleDB
PostgreSQL extension optimized for time series data
Integrates time series capabilities with SQL ecosystem
InfluxDB
Dedicated time series database often used beyond metrics/monitoring
Flexible TSDB for various sensor and telemetry data
Summary
TSDBs solve the problems of high-frequency writes, storage efficiency, and fast queries through:
TSDBs are best suited for cases with moderate cardinality and explicit time dimension filtering.
They are widely used in telemetry, monitoring, and IoT data scenarios but have limitations with extremely high cardinality datasets.
