In industrial IoT, connected vehicles, smart energy, and similar scenarios, large numbers of sensors generate data continuously at millisecond frequencies. Take a factory with 100,000 sensors as an example: data volume can reach millions of records per second, with daily growth exceeding tens of gigabytes. General-purpose relational databases can struggle with write throughput at this scale, and storage cost often becomes a major operational burden. TDengine TSDB addresses this at the storage-engine level, reducing storage consumption to roughly one-tenth of traditional solutions in suitable workloads through four innovations: multi-level compression algorithms, a Columnar Storage engine, Multi-Tier Storage strategy, and data lifecycle management.
Multi-Level Compression Technology
The first level consists of data-type-specific encoding:
- Delta Encoding: Compresses timestamps at ratios from 4:1 to 8:1.
- ZigZag Encoding: Represents signed integers so that small absolute values use less space.
- Delta-Delta Encoding: Compresses floating-point values with gradual changes from 64 bits down to 8 to 16 bits.
- Dictionary Encoding: Achieves compression ratios above 10:1 for enumeration-type string data.
The second level uses general-purpose algorithms: LZ4 (fastest), ZLIB (balanced), ZSTD (recommended default), and XZ (archival).
The third level, TSZ lossy compression, uses a prediction model to achieve 20:1 to 50:1 compression ratios while maintaining over 99% accuracy.
Columnar Storage
Data from the same column is stored contiguously on disk, typically improving compression ratios by 3 to 5 times compared to row-based storage. The key advantages: data homogeneity, larger compression blocks, and higher query efficiency.
Multi-Tier Storage
Hot data resides on memory or SSD (high cost), warm data on SSD (medium cost), and cold data on HDD or object storage (low cost). Since over 90% of data is cold and can be placed on lower-cost media, overall storage costs drop by 60% to 80%.
TTL Data Lifecycle Management
The KEEP parameter automatically cleans up expired data and supports dynamic adjustment. Benefits: minimal operational overhead, immediate space release, and flexible policy configuration.
Example: CREATE DATABASE iot_db KEEP 365; and ALTER DATABASE iot_db KEEP 180;
Storage Cost Comparison
For a scenario with 100,000 sensors reporting once per second (roughly 138 GB per day raw): MySQL or InfluxDB would occupy approximately 138 GB, while TDengine occupies roughly 12 GB (a compression ratio of about 12:1). Annual cloud disk costs come to roughly $7,000 for the traditional approach versus approximately $700 with TDengine. Some users have reported compression ratios exceeding 20:1.
Summary
Choose LZ4 when real-time performance is the priority. Choose ZSTD or XZ when storage cost sensitivity is high. Enable TSZ if some precision loss is acceptable. Combine these with Multi-Tier Storage and TTL for fine-grained management.
