In IoT and Industrial IoT scenarios, real-time movement of massive time-series data is a core business requirement. TDengine TSDB includes a message-queue-style Data Subscription mechanism built into the database engine. No additional middleware such as Kafka or RabbitMQ is needed. The system delivers real-time data push with millisecond latency.
A built-in message queue
Many architectures route data through a standalone message queue, creating long data paths and high operations costs. TDengine integrates message queue capability directly into the storage engine, forming a “write once, subscribe immediately” architecture.
Core advantages:
- Simplified architecture: no external message queue cluster to deploy or maintain.
- Zero data copy: data is read directly from the WAL, avoiding duplicate storage.
- Millisecond latency: subscription detection triggers immediately after WAL write.
- SQL preprocessing: filtering and aggregation complete on the database side.
The following table compares TDengine’s built-in subscription with Kafka across several dimensions:
| Dimension | TDengine Data Subscription | Kafka |
|---|---|---|
| Additional components | None required | Independent cluster needed |
| Data storage | Shared with time-series data | Independent log storage |
| Data preprocessing | SQL-based filtering at source | Requires stream processing framework |
| Consumption model | Push-pull hybrid | Pull only |
| Operations cost | Low (managed within TDengine) | High (separate cluster) |
Topic creation
TDengine supports three granularity levels for Topic creation:
- Database level:
CREATE TOPIC topic_db AS DATABASE powersubscribes to all tables in a database. - Supertable level:
CREATE TOPIC topic_meters AS STABLE meterssubscribes to all Subtables under a specific Supertable. - SQL query level: Define a Topic with a SELECT statement that includes filtering and aggregation. All computation runs on the database server side.
Real-time push
TDengine uses a push-pull hybrid model. When data is written, the subscription system immediately detects WAL changes and actively pushes data to consumers. When no new data is available, the connection stays alive through long polling.
The flow: data write, WAL write, subscription detection, active push to consumers.
Consumer group management
Within a single Consumer Group, consumption progress is shared. Each data record is processed by exactly one consumer in the group. Progress is managed centrally by the mnode, persisted to disk, and supports resumption from checkpoints.
A built-in automatic Rebalance mechanism checks Consumer Group status every 2 seconds. The vnode is the minimum unit of partition assignment.
Progress management
Consumption position for each vnode is tracked precisely by WAL version number. The auto.offset parameter controls where consumption begins on first start: earliest, latest, or none.
Kafka-compatible API
TDengine’s Data Subscription API maps cleanly to Kafka concepts:
| TDengine | Kafka Equivalent |
|---|---|
| Topic | Topic |
| Consumer Group | Consumer Group |
| VNode | Partition |
| Commit Offset | Offset |
This alignment reduces the migration learning curve for teams familiar with Kafka.
SQL preprocessing
Filtering and aggregation happen on the database side before data reaches the consumer. You can subscribe to data for a specific device group or only anomalous data, and you can pre-aggregate readings into 5-minute statistical summaries.
Raw data volumes may reach millions of records per second, but the aggregated results pushed to consumers may number only hundreds per second, a reduction of several thousand times in transmission volume.
Typical application scenarios
Real-time monitoring dashboards. Live data feeds push directly from TDengine to dashboard frontends without an intermediate message queue.
Cross-system data synchronization. Subscribe to TDengine data changes and replicate to Elasticsearch, ClickHouse, or other downstream systems.
Alert notification systems. Detect threshold violations through SQL preprocessing in the Topic definition and trigger notifications via SMS, email, or webhook.
Summary
TDengine delivers a one-stop solution from data write to real-time consumption through its built-in message-queue mechanism. Zero-copy reading from the WAL combined with the push-pull hybrid model achieves millisecond data push latency.
