Storage cost is one of the largest line items in any time-series database deployment. Sensor networks generate terabytes per month. Financial tick data accumulates at millions of points per second. Without effective compression, the storage footprint grows linearly with data volume, and query performance degrades as more disk I/O is required to read the same logical data.
TDengine applies compression at multiple levels: per-column encoding, block-level general compression, lossy compression for specific use cases, and transport compression for network traffic. This guide explains each mechanism, the algorithms involved, how to choose among them, and how to monitor compression effectiveness.
1. Two-level compression architecture
1.1 Level 1: Data-type-specific encoding
The first compression pass operates on individual data types before any general-purpose algorithm runs. Because a column stores values of a single type, the encoder can exploit patterns specific to that type.
Timestamp encoding (delta-of-delta): Timestamps in time-series data are usually evenly spaced. Instead of storing each absolute Timestamp (8 bytes), TDengine stores the delta between consecutive timestamps, then the delta of those deltas. For a 1-second interval series, the delta-of-delta is zero, and an entire block of timestamps compresses to nearly nothing. Typical compression ratio: 4:1 to 8:1.
Boolean encoding (bit-packing): A BOOL column stores TRUE/FALSE per row. Bit-packing encodes 8 boolean values per byte instead of using 1 byte per value. Compression ratio: 8:1.
Integer encoding (ZigZag + delta): Integers in sensor data often change by small increments or alternate between a few values. ZigZag encoding maps signed integers to unsigned values (interleaving negatives and positives), and delta encoding then stores the difference from the previous value. Small deltas encode in fewer bytes.
Floating-point encoding (delta-delta + XOR): Floats are harder to compress than integers because IEEE 754 representation scatters value changes across mantissa and exponent bits. The XOR encoding technique stores the XOR difference between consecutive float values. When values change slowly (as in temperature or pressure readings), the XOR difference has many leading and trailing zeros, which compress effectively.
String encoding (dictionary encoding): Tag values and string columns often repeat across rows (location names, device types, status codes). Dictionary encoding builds a lookup table of unique strings and replaces each occurrence with a small integer index. For columns with low cardinality, compression ratios exceed 10:1.
1.2 Level 2: General-purpose block compression
After type-specific encoding produces a stream of bytes, a general-purpose compression algorithm compresses the block further. TDengine supports four compressors, selectable via the compressor configuration parameter:
| Compressor | Speed | Ratio | Best for |
|---|---|---|---|
| LZ4 | Fastest | Moderate | High-throughput ingestion where CPU is the bottleneck |
| ZLIB | Moderate | Good | Balanced workloads, general-purpose use |
| ZSTD | Fast | High | Modern workloads where ratio and speed both matter |
| XZ | Slow | Highest | Archival and cold data where compression time is less important |
The default compressor is ZSTD in recent versions, chosen for its balance of speed and ratio. In practice, the Level 1 encoding already reduces data volume by 4x to 10x. Level 2 adds another 2x to 4x on top, for a combined 10:1 typical ratio.
2. Lossy compression
2.1 TSZ lossy compression algorithm
For time-series data where exact precision is not required, TDengine offers lossy compression through the TSZ algorithm. TSZ extends the lossless delta-delta and XOR encoding by truncating the least significant bits of floating-point values before encoding. Fewer significant bits create more leading and trailing zeros in the XOR difference, improving compression.
The precision is configurable via two parameters:
fPrecision 0.001 # Float precision: values rounded to 3 decimal places
dPrecision 0.0001 # Double precision: values rounded to 4 decimal places
Apply lossy compression to specific columns:
lossyColumn current.0.001 # Apply 0.001 precision to "current" column
2.2 Compression ratios with lossy compression
With lossy compression enabled on float and double columns, compression ratios of 20:1 to 50:1 are achievable. This is particularly relevant for high-frequency sensor data where the difference between 25.12345 and 25.12346 degrees Celsius is sensor noise, not meaningful signal.
2.3 When to use lossy compression
Lossy compression is appropriate when:
- Sensor precision exceeds what the use case requires (a thermometer reporting 6 decimal places when 2 are enough)
- The data is used for trend analysis, not precise financial reconciliation
- Storage cost reduction outweighs the need for exact values
- Data retention periods are long and compression ratio has a compounding effect on total storage
Lossy compression is not appropriate when:
- Data is used for billing or financial calculations
- Regulatory requirements mandate exact data preservation
- The data feeds into downstream systems that expect exact original values
3. Transport compression
3.1 RPC message compression
TDengine compresses RPC messages between cluster nodes when the message size exceeds a configurable threshold. Set compressMsgSize in taos.cfg:
compressMsgSize 1024 # Compress RPC messages larger than 1024 bytes
The default value is -1 (disabled). Enabling compression reduces network bandwidth usage at the cost of CPU for compression and decompression. In cross-datacenter deployments where bandwidth is constrained, transport compression provides a meaningful reduction in network traffic.
3.2 REST API compression
The taosAdapter REST API supports gzip compression for HTTP responses. Clients that include Accept-Encoding: gzip in their request headers receive compressed responses. This is especially valuable for query results that return large datasets.
curl -H "Accept-Encoding: gzip" \
"http://tdengine.example.com:6041/rest/sql/db_name" \
-d "SELECT * FROM meters WHERE ts > NOW - 1h"
3.3 WebSocket compression
The WebSocket interface supports per-message deflate compression via the perMessageDeflate option. When enabled, each WebSocket message is compressed independently, reducing bandwidth for streaming query results and Data Subscription consumers.
4. Algorithm selection guide
4.1 Storage compression
| Scenario | Level 1 encoding | Level 2 compressor | Lossy compression | Expected ratio |
|---|---|---|---|---|
| IoT sensor data (temperature, humidity) | Delta-delta + XOR | ZSTD | Yes (0.01 precision) | 20:1 to 40:1 |
| Industrial telemetry (vibration, current) | Delta + ZigZag | ZSTD | No | 8:1 to 12:1 |
| Financial tick data | Delta-of-delta + XOR | LZ4 | No | 6:1 to 10:1 |
| Archival cold data | Delta-of-delta + XOR | XZ | Optional | 15:1 to 30:1 |
| High-throughput ingestion | Delta-of-delta + ZigZag | LZ4 | No | 5:1 to 8:1 |
| String-heavy data (status, labels) | Dictionary encoding | ZLIB | No | 8:1 to 15:1 |
4.2 Transport compression
| Scenario | Mechanism | Recommended setting |
|---|---|---|
| Cross-region cluster replication | RPC compression | compressMsgSize 1024 |
| REST API for web dashboards | HTTP gzip | Client sends Accept-Encoding: gzip |
| Data Subscription consumers | WebSocket per-message deflate | Enable perMessageDeflate |
| Single-datacenter cluster (10 GbE+) | None required | Default (disabled) |
5. Compression configuration
5.1 Database-level compression settings
Compression is configured per database at creation time or altered later:
CREATE DATABASE sensor_data
COMPRESS 2 <em>-- Enable Two-level Compression</em>
CACHEMODEL 'NONE' <em>-- Cache model</em>
PRECISION 'ms'; <em>-- Timestamp precision</em>
ALTER DATABASE sensor_data COMPRESS 2;
5.2 Column-level compression
Specific columns can be configured with lossy compression:
ALTER STABLE meters MODIFY COLUMN current FLOAT PRECISION 0.01;
ALTER STABLE meters MODIFY COLUMN voltage FLOAT PRECISION 0.1;
5.3 Compressor selection
Choose the Level 2 compressor in taos.cfg:
# General-purpose
compressor ZSTD
# High-throughput ingestion
compressor LZ4
# Maximum compression for archival data
compressor XZ
6. Monitoring compression effectiveness
6.1 Checking compression ratios
TDengine exposes compression statistics through system tables and SHOW commands. Monitor the actual compression ratio achieved:
SELECT db_name, compression_ratio
FROM information_schema.ins_databases;
6.2 Disk usage by database
Compare logical data size with physical storage used:
SELECT db_name, data_size, storage_size,
ROUND(data_size * 1.0 / storage_size, 2) AS compression_ratio
FROM information_schema.ins_storage;
6.3 Transport compression statistics
Monitor RPC compression statistics to verify that transport compression is active and effective:
SHOW DNODES;
The output includes metrics for compressed and uncompressed RPC message counts and byte totals.
7. Compression and query performance
Compression is not only about storage cost. It also affects query performance because compressed data means fewer bytes to read from disk. A query that reads a compressed column block at 10:1 compression touches one-tenth the I/O of an uncompressed read.
However, decompression consumes CPU. The balance depends on the workload:
- I/O-bound workloads (slow disks, cold data, many concurrent queries) benefit more from compression because I/O savings outweigh CPU cost.
- CPU-bound workloads (fast NVMe storage, complex aggregations, few concurrent queries) benefit more from faster compressors (LZ4) or lighter compression settings.
TDengine’s default settings are balanced for typical TSDB workloads: moderate compression with ZSTD and lossless encoding. Tune toward higher compression for archival data and toward faster compression for high-throughput ingestion.
