Time-series data is characterized by monotonically increasing timestamps, large data volumes, high write frequency, and time-range-oriented queries. The TDengine storage engine’s core strengths: optimized high-concurrency writes, optimized time-range queries, extreme compression capability (reducing storage space to below 10% of the original data in suitable workloads), and horizontal scaling through the vnode mechanism.
Row and Column Format Encoding
Tuple Encoding is suited to data-dense, non-sparse scenarios. It provides O(1) time complexity access, directly locating data by row number. Key-Value Encoding is suited to sparse data with many NULL values. It uses an offset array index to record the actual position of each row, allocating storage only for data that actually exists.
Three-Layer vnode Storage Architecture
WAL guarantees data durability. All data changes are first appended to the WAL. During system restart, data is recovered by replaying the WAL.
Metadata storage (TDB engine) uses a B+Tree structure, suitable for read-heavy, write-light workloads. It supports range scans and transaction guarantees.
Time-series data storage (TSDB engine) is built on an LSM Tree structure and forms the core of the storage engine.
LSM Tree Structure
The MemTable uses a red-black tree (maintaining data uniqueness and ordering, with O(log n) insertion and lookup) and a SkipList (efficient range queries, multi-level indexing, O(log n) complexity).
Data file groups include: Head files (.head) for data block indexes, Data files (.data) for actual time-series data, SMA files (.sma) for pre-aggregated data, Tomb files (.tomb) for deletion markers, and STT files (.stt) for temporary data during small-file merges.
Compaction uses multi-level merging: Level 0 contains new files produced by MemTable flushes; Levels 1 through N contain progressively larger files. Tiered Compaction organizes by time, while Leveled Compaction organizes by size. During merging, small files are consolidated to reduce file handle overhead, deleted data is cleaned up, data is re-compressed, and indexes are updated.
Data Compression
Delta Encoding targets the timestamp column, converting large integers into small integers to significantly reduce storage space. General-purpose compression supports LZ4 (fast), ZSTD (higher compression ratio), and XZ (extreme compression ratio, suitable for cold data archiving). Compression results: timestamp column compression rates can fall below 5%, numeric columns reach 10% to 30%. In typical scenarios, total storage can drop to below 10% of the original data in suitable workloads.
