TDengine TSDB Data Subscription and Real-Time Synchronization

In IoT and IIoT scenarios, real-time data flow and synchronization are core requirements for operational systems. A time-series database, when designed for time-series data, needs to store large data volumes efficiently and provide flexible data subscription capabilities for real-time data distribution and consumption. This guide explains the data subscription mechanism in TDengine, covering the full practice from topic creation to consumer group management.

1. Core concepts of data subscription

The data subscription feature in TDengine is built on a publish-subscribe (Pub/Sub) model. By decoupling data production from consumption, it achieves efficient, reliable data flow. Understanding the following core concepts is the foundation for mastering data subscription.

1.1 Topic

A topic is the central abstraction of data subscription. It defines the source and scope of data. In TDengine, topics can be created in three ways:

• Database-level topic: Subscribes to all tables in a database. Suitable for scenarios requiring full data monitoring.
• Supertable-level topic: Subscribes to a specific supertable. Suitable for dividing data consumption scope by business module.
• SQL query topic: Created from a custom SQL query. Provides the most flexible data filtering capability. Filters and aggregations execute on the server side, reducing network transfer volume by up to a thousand times.

Once created, a topic is persisted in the database metadata. Consumers subscribe to topics to receive data streams.

1.2 Producer

A producer is an application that writes data to TDengine. Each time a producer writes new data, the database records the change in the WAL (Write-Ahead Log). The subscription system captures and distributes data based on the WAL.

The producer write path and the subscription system are independent. This means:

• Producers do not need to know whether consumers exist.
• The subscription feature has minimal impact on write performance.
• Multiple producers can write to the same data source that a topic references.

1.3 Consumer

A consumer is an application that retrieves data from a topic and processes it. Consumers in TDengine have the following characteristics:

• Independent consumption progress: Each consumer maintains its own consumption position and can resume from a checkpoint after restart.
• Batch pull: Supports configuring the number of records pulled per request, balancing real-time performance and processing efficiency.
• Automatic reconnect: Automatically reconnects on network failures to maintain consumption continuity.

1.4 Consumer group

The consumer group is a key feature of TDengine data subscription. It allows multiple consumers to work together for parallel data processing and load balancing.

Core mechanisms of a consumer group:

• Shared consumption progress: All consumers in the group share the same consumption progress, avoiding duplicate data processing.
• Automatic load balancing: When a consumer joins or leaves the group, the system automatically redistributes data partitions.
• Failover: When one consumer fails, its data partitions are automatically reassigned to other consumers in the group.

Consumer groups are especially useful for high-throughput scenarios, such as real-time analysis of large-scale sensor data and stream processing of log data.

2. Data subscription architecture

The data subscription architecture in TDengine follows a client-server model, making full use of the distributed database’s metadata management and data storage capabilities.

2.1 Architecture components

    +-------------------------------------------------------------+
    |                       Client layer                           |
    |  +-------------+  +-------------+  +---------------------+  |
    |  |  Consumer A  |  |  Consumer B  |  |      Consumer C     |  |
    |  | (Consumer)  |  | (Consumer)  |  |     (Consumer)      |  |
    |  +------+------+  +------+------+  +----------+----------+  |
    +---------+----------------+--------------------+-------------+
              |                |                    |
              +----------------+--------------------+
                               |
                        +------v------+
                        |  Network    |
                        +------+------+
                               |
    +--------------------------+----------------------------------+
    |                       Server layer                           |
    |  +-----------------------+--------------------------------+  |
    |  |                       v                                |  |
    |  |  +--------------------------------------------------+  |  |
    |  |  |              mnode (Management Node)              |  |  |
    |  |  |  - Topic metadata management                      |  |  |
    |  |  |  - Consumer state tracking                        |  |  |
    |  |  |  - Consumer group coordination                     |  |  |
    |  |  +--------------------------------------------------+  |  |
    |  |                       |                                 |  |
    |  |                       v                                 |  |
    |  |  +--------------------------------------------------+  |  |
    |  |  |              vnode (Virtual Node)                 |  |  |
    |  |  |  - WAL data reading                                |  |  |
    |  |  |  - Consumption request processing                  |  |  |
    |  |  |  - Progress commit                                 |  |  |
    |  |  +--------------------------------------------------+  |  |
    |  +----------------------------------------------------------+  |
    +-------------------------------------------------------------+

2.2 mnode management responsibilities

The mnode (Management Node) serves as the coordinator in data subscription:

• Topic management: Maintains topic metadata, including topic type, data source definition, creation time, and more.
• Consumer registration: Records active consumer information, including the consumer group they belong to and the topics they subscribe to.
• Consumer group coordination: Monitors consumer group membership changes and triggers rebalance operations.

2.3 vnode data processing

The vnode (Virtual Node) is the execution unit that handles actual consumption requests:

• WAL reading: Reads data change records from local WAL files.
• Data filtering: Filters data matching the topic definition.
• Progress management: Processes consumption progress commit requests and persists checkpoint positions.

3. Rebalance: automatic load balancing

Rebalance is a core mechanism of data subscription in TDengine. It ensures that data partitions within a consumer group are distributed fairly and efficiently among consumers.

3.1 Trigger conditions

Rebalance is triggered automatically in the following scenarios:

• New consumer joins: When a new member joins the consumer group, data partitions must be redistributed.
• Consumer leaves: When a consumer disconnects or voluntarily exits, its partitions are reassigned to other members.
• Periodic check: The system checks consumer group state every 2 seconds and triggers rebalance if a change is detected.

3.2 Allocation strategy

TDengine uses a vnode-based allocation strategy:

1. Partition granularity: Allocation is performed at the vnode level. Each vnode corresponds to one data partition.
2. Even distribution: Vnodes are distributed as evenly as possible among consumers in the group.
3. Minimal migration: During rebalance, existing allocation relationships are preserved where possible to reduce unnecessary data migration.

3.3 Consumption flow example

<em>-- 1. Create database and supertable</em>
CREATE DATABASE IF NOT EXISTS power;
USE power;

CREATE STABLE IF NOT EXISTS meters (
    ts TIMESTAMP,
    current FLOAT,
    voltage INT,
    phase FLOAT
) TAGS (location BINARY(64), groupId INT);

<em>-- 2. Create topic (based on supertable)</em>
CREATE TOPIC meter_topic AS SELECT * FROM meters;

<em>-- 3. Create consumer group and subscribe to topic</em>
<em>-- Consumer code example (Python)</em>

from taos import Consumer

<em># Configure consumer parameters</em>
conf = {
    "group.id": "meter_consumer_group",  <em># Consumer group ID</em>
    "td.connect.ip": "localhost",
    "td.connect.user": "root",
    "td.connect.pass": "taosdata",
    "td.connect.db": "power",
    "auto.offset.reset": "earliest"  <em># Start from the earliest data</em>
}

<em># Create consumer instance</em>
consumer = Consumer(conf)

<em># Subscribe to topic</em>
consumer.subscribe(["meter_topic"])

<em># Consume data</em>
while True:
    <em># Pull data, timeout 1000ms</em>
    res = consumer.poll(1000)
    if res:
        <em># Process data</em>
        for row in res:
            print(f"Timestamp: {row[0]}, Current: {row[1]}, Voltage: {row[2]}")

        <em># Commit consumption progress</em>
        consumer.commit(res)

<em># Close consumer</em>
consumer.unsubscribe()
consumer.close()

4. Consumption progress management

Consumption progress management is the key mechanism for ensuring data is neither lost nor duplicated. TDengine uses WAL version numbers to precisely track consumption positions.

4.1 Progress storage

• Storage location: Consumption progress is persisted in the mnode.
• Identification method: WAL version numbers identify the consumption position.
• Granularity: Each vnode independently maintains its own consumption progress.

4.2 Consumption start position configuration

Consumers can configure the start position using the auto.offset.reset parameter:

Value	Description
earliest	Start from the earliest available data. Suitable for scenarios requiring full historical data.
latest	Start from the newest data. Suitable for scenarios that only need real-time data.
none	Throw an exception if no committed progress exists. Suitable for scenarios requiring guaranteed progress continuity.

4.3 Manual commit vs. auto commit

TDengine supports two progress commit modes:

Manual commit:

• The consumer explicitly calls the commit() method after processing data.
• Advantage: Precise control over the commit timing, ensuring progress is recorded only after data processing completes.
• Use case: Scenarios requiring high data reliability.

Auto commit:

• The consumer periodically commits progress automatically.
• Advantage: Simplifies code logic and reduces development effort.
• Use case: Scenarios requiring high real-time performance that can tolerate occasional data duplication.

5. Push-pull hybrid data transmission

TDengine uses a push-pull hybrid model for data transmission, balancing real-time performance and resource efficiency.

5.1 Push mode when data is available

When new data is written to a vnode:

• The system actively pushes data to waiting consumers.
• Consumers process data immediately, achieving millisecond-level latency.
• Suitable for high-frequency data write scenarios.

5.2 Poll mode when no data is available

When a vnode has no new data temporarily:

• Consumers enter a poll-wait state.
• Long polling maintains the connection to the server.
• New data triggers an immediate response, avoiding the resource waste of frequent requests.

5.3 Mode advantages

The push-pull hybrid model offers these benefits:

1. Low latency: Data is pushed immediately when available, with no need to wait for a poll cycle.
2. Low resource consumption: The connection is kept alive when no data is present, reducing empty polls.
3. High throughput: Data is pushed in batches, reducing network round trips.
4. Fault tolerance: Automatic reconnect on network errors maintains consumption continuity.

6. Typical application scenarios

The data subscription feature in TDengine is suitable for a variety of real-time data processing scenarios:

6.1 Real-time alerting

Subscribe to key metric data for real-time anomaly detection and alert notification:

• Subscribe to sensor data such as temperature and pressure.
• Evaluate whether data exceeds thresholds in real time.
• Trigger alert notifications (SMS, email, DingTalk, etc.).

6.2 Data synchronization and backup

Synchronize data from TDengine to other systems in real time:

• Sync to a data warehouse for offline analysis.
• Sync to a message queue for consumption by other services.
• Cross-region data backup and disaster recovery.

6.3 Real-time dashboards

Provide real-time data streams for monitoring dashboards:

• Subscribe to operational data from key devices.
• Compute aggregated metrics in real time.
• Push results to front-end display interfaces.

6.4 Stream processing

Combine with stream processing frameworks for real-time analysis:

• Feed subscribed data into frameworks such as Flink and Spark Streaming.
• Compute sliding window metrics in real time.
• Generate real-time business reports.

7. Best practices

When using the data subscription feature in TDengine, the following best practices are recommended:

7.1 Topic design

• Divide topics by business module: Avoid creating overly broad topics to reduce unnecessary data transmission.
• Use SQL topics appropriately: For complex data filtering needs, use SQL query topics instead of client-side filtering.
• Control the number of topics: Too many topics increase metadata management overhead.

7.2 Consumer group configuration

• Set consumer count based on data volume: The number of consumers should match the number of vnodes to avoid resource waste.
• Configure reasonable timeout values: Set consumption timeout and retry policies based on business requirements.
• Monitor consumption lag: Detect insufficient consumption capacity or data processing bottlenecks early.

7.3 Exception handling

• Handle consumption timeouts: Set reasonable poll timeout values to avoid prolonged blocking.
• Implement retry mechanisms: For transient errors, implement exponential backoff retry.
• Log consumption activity: Supports troubleshooting and data auditing.

8. Practical takeaways

TDengine Data Subscription gives developers flexible data flow capabilities. Topic management, consumer groups, and rebalancing enable reliable, high-throughput real-time data consumption. The push-pull hybrid transmission model delivers low latency while keeping system resource consumption in check.

Whether building a real-time alerting system, a data synchronization pipeline, or a stream processing platform, the data subscription feature in TDengine can meet your needs. For more detailed documentation and code examples, visit the official documentation center at docs.tdengine.com.