Data subscription is a common requirement when building a time-series data processing platform. Traditional architectures often add a message queue such as Kafka between the database and downstream systems, which increases component count, operational work, and failure points. The built-in Data Subscription feature in TDengine TSDB can handle many real-time distribution needs directly at the database layer.
Data subscription core value
TDengine TSDB provides subscription and consumption interfaces similar to those of message queue products. In suitable scenarios, a big data platform built on TDengine can distribute data through the database’s own subscription capabilities instead of adding a separate message queue layer. This reduces the number of components and lowers the operational burden.
Creating a subscription topic
In a time-series database, subscription topics are created through standard SQL statements. A topic is a persistent push channel for query results and is the core mechanism for real-time data distribution.
CREATE TOPIC IF NOT EXISTS topic_meters AS SELECT ts, current, voltage, phase, groupid, location FROM metersThe statement above creates a topic named <strong>topic_meters</strong> that continuously pushes timestamp, current, voltage, phase, group ID, and location information from the <strong>meters</strong> table. When new data is written to the <strong>meters</strong> table, consumers subscribed to this topic receive the data stream in real time.
Consumer configuration details
The consumer concept in a time-series database is similar to that in Kafka: consumers receive data streams by subscribing to topics. The complete configuration parameters are as follows.
Connection parameters:
<strong>td.connect.ip</strong>: server IP address<strong>td.connect.user</strong>: username<strong>td.connect.pass</strong>: password
Consumer group and client identification:
<strong>group.id</strong>: consumer group ID (required, maximum 192 characters; up to 100 consumer groups per topic)<strong>client.id</strong>: client ID (maximum 192 characters)
Offset management:
<strong>auto.offset.reset</strong>: initial offset position for the consumer group, options areearliest,latest, ornone<strong>enable.auto.commit</strong>: whether to enable automatic offset commit (default<strong>true</strong>)<strong>auto.commit.interval.ms</strong>: automatic commit interval in milliseconds (default 5000)
Advanced features:
<strong>enable.replay</strong>: whether to enable data replay (default disabled)
The <strong>group.id</strong> parameter is required and identifies the consumer group. Each topic supports up to 100 consumer groups. The automatic commit mechanism ensures that consumers can resume consumption from the last committed position after an unexpected restart, avoiding data loss or duplication.
Subscription query limitations
The following query restrictions apply when using Data Subscription:
- SELECT only: INSERT, UPDATE, and DELETE operations are not supported
- Raw data only: aggregated results or computed results are not supported
- Ascending time order only: data is pushed in ascending timestamp order
These restrictions mean that the data subscription feature focuses on real-time distribution of raw data. For aggregation needs, you can handle processing on the consumer side or use the database’s stream processing capabilities.
Typical application scenarios
Real-time data synchronization: push collected data to downstream systems in real time, such as monitoring and alerting systems, analytics engines, or data warehouses, ensuring data consistency across systems.
Multi-downstream distribution: use multiple consumer groups to push the same data to different business systems simultaneously, with each group managing its own consumption progress independently.
Data replay: enable replay to re-consume historical data, which is useful for data repair or model retraining scenarios.
Summary
TDengine TSDB Data Subscription offers a lightweight option for real-time data distribution. With built-in topic and consumer group mechanisms, teams can push newly written time-series data to downstream systems without always deploying a standalone message queue. For workloads that need raw data distribution, replay, and multi-consumer delivery, this can simplify the overall architecture.
