Time-series data is everywhere. From IoT sensor readings and server metrics to financial ticks and vehicle telemetry, organizations need databases that can ingest millions of data points per second, store them efficiently, and return query results in milliseconds. A time-series database (TSDB) must handle workloads that look nothing like traditional OLTP or OLAP patterns.
TDengine is a high-performance, distributed time-series database purpose-built for these scenarios. Its architecture reflects a set of design decisions that differ from both general-purpose RDBMS and other TSDB products. This guide walks through the core technology architecture across four layers: data ingestion, distributed storage, query computing, and Stream Processing.
1. Distributed cluster architecture
1.1 Cluster topology
A TDengine cluster is built from dnodes (Data Nodes). Each dnode is a running instance of the taosd process and can host multiple logical node types:
- mnode (Management Node): Handles cluster metadata, user management, and coordination. Each cluster must have at least one mnode, and multiple mnodes form an HA group via Raft consensus.
- vnode (Virtual Node): The core unit for data storage and query execution. Each vnode manages a subset of time-series data and handles writes, queries, and compaction independently. A vgroup (Virtual Group) contains one or more vnodes that replicate the same data shard for high availability.
- qnode (Query Node): Handles Supertable queries that span multiple vnodes. A qnode decomposes a Supertable query into per-vnode sub-queries, distributes them, and merges the results.
- snode (Stream Node): Executes stream processing tasks. At least one snode is required for Stream Processing to function. Multiple snodes can be deployed for load balancing and HA.
A production cluster might look like this:
CREATE DNODE "node1.tdengine.com";
CREATE DNODE "node2.tdengine.com";
CREATE DNODE "node3.tdengine.com";
SHOW DNODES;
1.2 Storage-compute separation
TDengine supports Storage-Compute Separation. vnodes handle storage; qnodes and snodes handle compute. The two layers scale independently. When query load increases, add qnodes. When storage or write throughput becomes the bottleneck, add dnodes with vnodes.
This separation matters because TSDB workloads are often write-heavy at ingestion time but query-heavy during operational hours. Independent scaling avoids over-provisioning one dimension to satisfy the other.
2. Data model: One Table per Device and Supertable
2.1 One Table per Device
The core data modeling principle in TDengine is “One Table per Device.” Each sensor, machine, or data collection point receives its own table. This is not a workaround. It is the fundamental design.
A single device table looks like this:
CREATE TABLE t_001 (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT);
The benefits compound at scale. Because each device owns its table, writes are lock-free and append-only. Data from different devices does not contend for the same write path. This model also maps naturally to the physical world: a wind farm has 500 turbines, each turbine has a table, and nobody queries across turbines as if they were a single entity.
2.2 Supertable and tag system
Managing millions of individual tables by hand is not practical. TDengine introduces the Supertable as a schema template with a tag mechanism for metadata.
A Supertable defines the shared schema (columns) and tag structure (static attributes) for a class of devices:
CREATE STABLE meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT)
TAGS (location BINARY(64), group_id INT);
Subtables inherit the schema and supply their own tag values:
CREATE TABLE t_001 USING meters TAGS ("Building_A", 1);
CREATE TABLE t_002 USING meters TAGS ("Building_B", 2);
Tags are stored separately from time-series data and indexed for fast filtering. This separation means a query like “give me the average current for all meters in group 1” filters by tag metadata first, then touches only the relevant time-series columns, never scanning unrelated data.
2.3 Comparison with other models
| Dimension | TDengine (One Table per Device) | Traditional RDBMS | Other TSDBs (measurement + tags) |
|---|---|---|---|
| Data organization | Per-device table | Shared wide table or partitioned table | Metric-centric measurement with tag sets |
| Write contention | Lock-free per device | Row-level or table-level locks | Varies; some use per-series locks |
| Schema flexibility | Supertable template with tags | Rigid schema or EAV anti-pattern | Tag-based with dynamic columns |
| Query for one device | Single-table scan, fast | Requires WHERE clause filtering | Tag-based filtering |
| Cross-device aggregation | Supertable query via qnode | Complex JOIN or UNION | Measurement + GROUP BY tags |
| High cardinality | Native support; one table per device | Index pressure; wide tables struggle | Index pressure on tag indexes |
3. Storage engine: LSM-based columnar storage
3.1 LSM storage engine
TDengine uses an LSM Storage Engine design, adapted for time-series data. Writes land in a MemTable (in-memory buffer) and are flushed to persistent SSTable files on disk. Because time-series data is append-only and ordered by Timestamp, the compaction process is simpler than in general-purpose LSM implementations: data arrives in time order, so SSTable files are naturally non-overlapping in the time dimension.
3.2 Columnar storage and compression
Data within each SSTable is stored in columnar format. All timestamps for a block are stored together, all current values together, all voltage values together. This has two main effects:
- Compression: Same-type values adjacent in storage compress far better than interleaved rows. TDengine applies type-specific encoding (delta-of-delta for timestamps, ZigZag for integers, XOR for floats), followed by a general-purpose compressor (LZ4, ZLIB, ZSTD, or XZ). Typical compression ratios reach 10:1, and with lossy compression enabled they can exceed 20:1.
- Query performance: A query that only reads
currenttouches only thecurrentcolumn blocks. Other columns stay on disk. This is columnar storage applied to time-series data rather than analytics data, and the principle is the same: read fewer bytes, go faster.
3.3 Tiered storage
On top of columnar compression, TDengine supports tiered storage. Hot data stays on local SSD for fast access. Warm and cold data can migrate to low-cost object storage (S3 or compatible). Data moves between tiers automatically based on the retention policy configured per database.
4. Query engine: virtual table aggregation
4.1 The Supertable query challenge
A Supertable query like SELECT AVG(current) FROM meters WHERE group_id = 1 INTERVAL(1h) must aggregate data across potentially thousands of Subtables. A naive approach that unions all matching tables and then aggregates would be slow and memory-intensive.
4.2 Virtual table query optimization
TDengine handles this with a virtual table approach:
- Tag filtering: The query first evaluates the tag condition (
group_id = 1) against the tag index to identify which Subtables match. - Sub-query decomposition: The qnode decomposes the Supertable query into per-vnode sub-queries, each targeting a batch of matching Subtables.
- Parallel execution: Each vnode executes its sub-query independently, reading only the columns requested from its local storage.
- Result merging: The qnode collects partial aggregates from all vnodes and merges them. For decomposable aggregates (SUM, COUNT, MIN, MAX), the merge is a simple combine step. For non-decomposable aggregates (AVG, STDDEV), the qnode reconstructs the final result from component values.
A typical query pattern:
<em>-- Downsampling with time window</em>
SELECT AVG(current), MAX(voltage)
FROM meters
WHERE group_id = 1
INTERVAL(10m);
<em>-- Cross-device aggregation with tag filter</em>
SELECT location, COUNT(*), AVG(current)
FROM meters
WHERE group_id = 1
PARTITION BY location;
4.3 Performance advantages
Compared with traditional RDBMS that uses indexed WHERE clauses on a shared table, the virtual table approach avoids full-table scans and uses per-device table locality. Compared with other TSDBs that use a metric-centric model, the Supertable query path benefits from tag separation: filtering on static attributes does not scan time-series data.
The columnar format adds a second benefit. A query that reads 3 out of 50 columns touches roughly 6% of the data volume. Row-oriented storage would read all 50 columns regardless.
5. Stream Processing and Data Subscription
5.1 Stream Processing
TDengine includes a built-in Stream Processing engine. Streams are defined as continuous queries over incoming data:
CREATE STREAM avg_current_stream
INTO avg_current_output
AS SELECT AVG(current) AS avg_current
FROM meters
INTERVAL(1m);
The snode evaluates this stream continuously. Each 1-minute window produces a result row written to avg_current_output. The stream definition uses the same SQL syntax as normal queries, so there is no separate stream processing language to learn.
Stream Processing supports multiple trigger types: time windows, session windows, state windows, event windows, count windows, and sliding windows. The engine also handles out-of-order data with WATERMARK and expiration policies.
5.2 Data Subscription
For applications that need to consume data changes in real time, TDengine provides a Data Subscription API (TMQ) modeled after Kafka’s consumer model:
import taos
conn = taos.connect()
consumer = conn.subscribe(
topic="meter_data",
sql="SELECT * FROM meters",
group_id="group_1",
client_id="client_1"
)
while True:
records = consumer.poll(timeout=1000)
for record in records:
print(record)
consumer.commit()
The subscription model supports consumer groups, partition rebalancing, offset management, and auto-commit. A single topic can serve multiple consumer groups, each consuming at its own pace.
6. Performance foundations
6.1 Compression that reduces cost and I/O
The two-level compression model (type-specific encoding followed by general-purpose compression) typically achieves a 10:1 ratio. This directly cuts storage cost by an order of magnitude. It also reduces disk I/O: fewer bytes on disk means fewer bytes to read during query execution. In cloud and edge deployments where storage is metered, compression has a direct impact on total cost.
6.2 Multi-level cache architecture
TDengine uses a Multi-Level Cache with four layers:
- Write Cache: Absorbs incoming writes in memory before they are flushed to disk. High-throughput ingestion stays in RAM as long as possible, and the write acknowledgement returns before the disk flush completes.
- Read Cache: Keeps recently queried data blocks in memory. For dashboard and monitoring workloads that repeatedly query the same recent time ranges, read cache hit rates are high.
- Metadata Cache: Caches table schemas, tag values, and Supertable definitions. Metadata lookups during query planning avoid disk access entirely.
- File System Cache: The OS page cache provides a final caching layer for disk blocks. TDengine’s columnar format means cached blocks are dense with relevant data.
6.3 Stream Processing and Data Subscription performance
The Stream Processing engine runs inside the database process, not as an external compute layer. This eliminates the network hop between the database and a separate stream processor. Data flows from write path to stream evaluation within the same process, which keeps latency low for near-real-time use cases.
7. Application scenarios
7.1 IoT and Industrial IoT
The One Table per Device model maps directly to physical sensor networks. A factory with 10,000 sensors gets 10,000 tables, each storing data independently. Tag-based filtering enables cross-sensor queries such as “average temperature for all sensors in workshop 3.” The columnar compression keeps storage manageable at scale. A typical manufacturing deployment sees 10-15x compression on sensor data.
7.2 Connected cars
Vehicle telemetry involves high write throughput (millions of vehicles reporting every second) and diverse query patterns (single-vehicle diagnostics, fleet-level aggregation, geospatial analysis). The Supertable model with tags for vehicle type, region, and fleet simplifies fleet-wide queries. The built-in Stream Processing enables real-time alerts on speed, geofence violations, or sensor anomalies.
7.3 Energy and power systems
Power grids, wind farms, and solar plants generate high-frequency measurement data. TDengine’s columnar storage keeps per-measurement queries efficient because only the relevant column is read. Tiered storage moves older operational data to low-cost object storage while keeping recent data on SSD for fast access. This aligns with the typical data lifecycle: recent data is queried frequently for real-time operations, while historical data is accessed less often for analysis and compliance.
7.4 FinTech
Tick data, order book snapshots, and transaction logs are all time-series. The append-only write path suits these workloads because financial data is never updated, only appended. Compression ratios on financial time-series are often better than sensor data because values change in small increments, making delta encoding highly effective.
8. Comparison with other database architectures
8.1 vs. TDengine TSDB vs. traditional RDBMS
A traditional RDBMS (MySQL, PostgreSQL) can store time-series data, but the data model and storage engine work against the workload. Timestamp-ordered writes produce random I/O in a B-tree index. Cross-device queries require complex JOINs or UNIONs. Compression is row-level and generic. TDengine’s columnar, append-only, device-per-table design is better matched to the access patterns of time-series data.
8.2 TDengine TSDB vs. other TSDBs
| Dimension | TDengine | InfluxDB | TimescaleDB |
|---|---|---|---|
| Data model | One Table per Device + Supertable | Measurement + tag sets | Hypertable (chunk-based partitioning) |
| Storage engine | LSM with columnar SSTable | TSM (Time-Structured Merge Tree) | PostgreSQL heap with chunk partitioning |
| Compression | Two-level: type-specific + general | Type-specific + Snappy | Native PostgreSQL compression |
| Stream Processing | Built-in (snode) | Via Kapacitor (external) | Via separate tools |
| Data Subscription | Built-in TMQ (Kafka-style) | Via external systems | Via PostgreSQL logical replication |
| Storage-compute separation | Native | Not native | Not native (PostgreSQL extension) |
| Query language | Standard SQL with TSDB extensions | InfluxQL / Flux | Standard SQL |
| High cardinality | Native support | Tag index overhead | Chunk + index overhead |
