In a typical time-series data processing architecture, data collected from devices first passes through a message queue for buffering and distribution, then downstream systems consume it. This approach is mature, but it comes with clear pain points: added architectural complexity (you need to deploy and maintain a separate message queue cluster such as Kafka or RabbitMQ), higher operational costs, and a longer data pipeline.
TDengine addresses these problems by offering data subscription and consumption interfaces that provide message-queue-like semantics. In many scenarios, a time-series big data platform built on TDengine may be able to remove the external message queue, simplifying application design and reducing operational costs.
The topic: defining what data gets subscribed
The core concept of data subscription is the Topic, which defines the data scope that consumers subscribe to. TDengine supports three topic types:
Query topic
A query topic defines subscription content through a SQL query statement. It works like a continuous query. The syntax is CREATE TOPIC topic_name AS subquery.
A query topic returns only the latest value on each poll, not the full historical dataset. It supports standard SELECT statements with conditional filters and scalar function calculations, but does not support aggregate functions or time-window aggregation.
Example:
CREATE TOPIC power_topic AS SELECT ts, current, voltage
FROM power.meters WHERE voltage > 200;
Supertable topic
A supertable topic subscribes to all data in a specific supertable. The syntax is CREATE TOPIC topic_name [WITH META] AS STABLE stb_name [WHERE condition].
The system does not restrict schema changes, and returns unstructured data. The WITH META option includes table metadata in each message.
Database topic
A database topic covers all data across an entire database. The syntax is CREATE TOPIC topic_name [WITH META] AS DATABASE db_name.
This is suited for scenarios that need global awareness of data changes, such as data synchronization and cross-system data migration.
Consumer model: production-grade consumption
On the consumer side, TDengine provides a complete consumption model:
- Consumer group allows multiple consumers to share consumption progress, with failover and load balancing built in.
- Consumer group isolation keeps different consumer groups advancing independently.
- Multi-topic subscription and message ACK (At Least Once delivery) are both supported.
Client SDKs are available for C, Java, Go, Rust, Python, C#, and other languages. The API design closely follows Kafka conventions, so teams with Kafka experience can migrate with little additional learning.
How it works under the hood
Data subscription is built on WAL (Write-Ahead Log) files. The system automatically creates indexes for WAL files, enabling fast random access. After reading data from the WAL, the same query engine that handles standard queries applies filtering and transformation operations, so the semantics stay consistent.
Data replay
The data replay feature replays data streams in the actual write-time order. It is useful for backtesting algorithmic models, debugging system issues, and testing data pipelines. Currently, only query topic subscriptions support data replay. Supertable topics and database topics do not support it yet.
Management SQL
The following SQL commands are available for managing subscriptions:
SHOW TOPICSSHOW CONSUMERSSHOW SUBSCRIPTIONSDROP TOPICDROP CONSUMER GROUP
When to use data subscription
Common use cases: real-time alerting, data synchronization, stream processing, and multi-line-of-business data distribution.
When a project’s need for a message queue centers on data distribution rather than complex message routing or transaction processing, the built-in subscription feature can replace external message middleware for that part of the architecture.
