Stream processing engine architecture
TDengine’s stream processing engine uses a distributed architecture design, with management nodes (mnode) and virtual nodes (vnode) working together to achieve efficient stream task scheduling and execution.
mnode scheduling layer
The management node (mnode) handles global scheduling and management of stream processing tasks. Its functions include:
- Task assignment: Distributes stream tasks to each vnode for execution based on cluster load.
- Status monitoring: Monitors the running status of stream tasks in real time, detects failures, and performs automatic recovery.
- Metadata management: Maintains stream task definitions, configurations, and dependency relationships.
- Checkpoint coordination: Triggers distributed checkpoints to ensure state consistency.
vnode execution layer
The virtual node (vnode) is the actual execution unit for stream tasks. Each vnode independently runs the stream tasks assigned to it. TDengine stream tasks fall into three types:
Source Task
The Source Task reads data from the Write-Ahead Log (WAL) as the input source for stream processing. When data is written into TDengine, the WAL records all changes. The Source Task listens for these change events and pushes newly arrived data to downstream processing nodes.
Agg Task
The Agg Task runs on snode (stream node) and handles complex computation logic, including:
- Time window aggregation (Tumbling Window, Sliding Window, Session Window)
- State computation and maintenance
- Multi-stream Join operations
- User-defined aggregate functions (UDAF)
Sink Task
The Sink Task outputs computation results to target destinations, supporting multiple output methods:
- Write back to a TDengine Supertable or Subtable
- Push to an external message queue
- Call a Webhook endpoint
Key concepts
Stateful stream processing
Unlike traditional stateless computation, TDengine’s stream processing engine supports stateful computation. When handling scenarios such as time window aggregation, deduplication, and pattern matching, the system must maintain intermediate state data. This state data is stored in memory and periodically persisted to remote storage through the checkpoint mechanism, ensuring that processing can continue without data loss after a failure recovery.
Write-Ahead Log (WAL) mechanism
WAL (Write-Ahead Log) is the data source foundation for stream processing. All data written to TDengine is first recorded in the WAL, and the stream processing engine obtains data change events by reading the WAL. This design provides the following advantages:
- Data consistency: Stream processing and the storage layer share the same data source.
- Exactly-once semantics: Through WAL offset management, each data record is processed exactly once.
- Historical data replay: Supports re-consuming data from a specified point in time, enabling data repair and recalculation.
Event-driven processing model
TDengine stream processing uses an event-driven architecture: data write operations trigger the corresponding computation tasks. When new time-series data arrives:
- Data is first written to the vnode’s WAL.
- The Source Task detects the WAL update event.
- Data is extracted and sent to the downstream Agg Task.
- The Agg Task processes the data according to time windows and computation logic.
- Results are output through the Sink Task.
This event-driven model ensures real-time computation results, with latency typically in the millisecond range.
Three time semantics
In stream processing, the concept of time matters. TDengine supports three time semantics:
| Time type | Definition | Use case |
|---|---|---|
| Event Time | The actual time when data was generated, typically reported by the device | Out-of-order data processing, business time analysis |
| Ingestion Time | The time when data arrives at TDengine | Simple scenarios, relaxed ordering requirements |
| Processing Time | The time when data is processed by the stream engine | Low-latency priority scenarios |
Developers can choose the appropriate time semantic based on business requirements, specifying time windows through the INTERVAL and SLIDING clauses in SQL statements.
Time window aggregation
Time window aggregation is a core function of stream processing. TDengine supports multiple window types:
Tumbling Window
A fixed-size, non-overlapping time window, suitable for statistics over fixed periods:
CREATE STREAM stream_current
INTO table avg_current AS
SELECT _wstart, AVG(current) AS avg_current
FROM meters
WHERE voltage > 200
INTERVAL(1m);
This SQL creates a stream task that computes the average current every minute for devices where voltage exceeds 200.
Sliding Window
Windows that can overlap, suitable for scenarios that require smoothed statistics:
CREATE STREAM stream_sliding
INTO table sliding_avg AS
SELECT _wstart, AVG(temperature) AS avg_temp
FROM sensors
INTERVAL(5m) SLIDING(1m);
Session Window
Windows are dynamically partitioned based on data activity, suitable for user behavior analysis and similar scenarios.
Out-of-order data handling
In real-world scenarios, data often arrives out of order due to network latency, unsynchronized device clocks, and other factors. The TDengine stream processing engine provides a out-of-order handling mechanism:
WATERMARK mechanism
WATERMARK allows delayed data to be processed. By setting a watermark, the system waits for a period before triggering window computation, in order to accept late-arriving data:
CREATE STREAM stream_with_watermark
INTO table delayed_avg AS
SELECT _wstart, AVG(value) AS avg_value
FROM device_data
INTERVAL(1m) WATERMARK(30s);
This configuration means the system will wait 30 seconds, accepting late-arriving data before closing the window for computation.
IGNORE EXPIRED
For data that exceeds the WATERMARK threshold, the IGNORE EXPIRED clause can be used to discard it:
CREATE STREAM stream_ignore_expired
INTO table result AS
SELECT _wstart, COUNT(*) AS cnt
FROM events
INTERVAL(1m) WATERMARK(1m) IGNORE EXPIRED;
Checkpoint and fault recovery
The reliability of stream tasks is a key requirement for production applications. TDengine implements a complete fault-tolerance mechanism:
Distributed checkpoint
The system periodically triggers checkpoint operations, persisting stream computation state data (including window states, aggregate intermediate results, etc.) to remote storage (such as S3, HDFS). The checkpoint process uses a distributed snapshot algorithm to ensure:
- State consistency: All task nodes’ states are saved at the same logical point in time.
- Minimal pause: The impact of checkpointing on normal computation is kept to a minimum.
- Incremental backup: Reduces storage and transmission overhead.
Fault recovery
When a vnode fails, the mnode detects the task anomaly and triggers the recovery process:
- Reassign the failed task to a healthy vnode.
- Restore state data from the most recent checkpoint.
- Resume WAL data consumption from the checkpoint position.
- Ensure the accuracy and consistency of computation results.
Flow control and backpressure mechanism
In high-throughput scenarios, mismatched upstream and downstream processing speeds can cause system overload. The TDengine stream processing engine includes built-in flow control and backpressure mechanisms:
Sink Task rate limiting
When the output destination (such as an external database or message queue) has limited processing capacity, the Sink Task output rate can be limited through configuration to avoid overwhelming downstream systems.
Upstream-downstream backpressure
When the downstream Agg Task cannot keep up with the data production rate of the upstream Source Task, the system triggers backpressure:
- The Source Task reduces its data extraction rate.
- Data briefly accumulates in the WAL.
- The system automatically speeds up once downstream processing capacity recovers.
This backpressure mechanism ensures stable operation under various load conditions, preventing memory overflow or data loss.
Application scenarios and practice guidance
The TDengine stream processing engine is suitable for a range of real-time data processing scenarios:
Real-time monitoring and alerting
Real-time analysis of device metrics, triggering alerts when anomalies are detected (e.g., excessive temperature, abnormal current):
CREATE STREAM stream_alert
INTO table alerts AS
SELECT _wstart, device_id, MAX(temperature) AS max_temp
FROM sensors
WHERE temperature > 80
INTERVAL(10s);
Real-time dashboards
Provides second-level updated statistical data for operations monitoring displays, supporting real-time business decisions.
Data cleaning and transformation
Performs real-time cleaning, unit conversion, and format standardization at the point of data ingestion, reducing downstream ETL burden.
Anomaly detection
Combined with machine learning models, enables real-time anomaly detection and predictive maintenance based on stream processing.
Practical takeaways
The TDengine stream processing engine provides real-time data processing inside TDengine. With distributed coordination between mnode and vnode, comprehensive time window aggregation, flexible out-of-order data handling, and reliable checkpoint-based fault tolerance, teams can run the full pipeline from data ingestion to real-time computation to result output entirely within TDengine, without introducing external components such as Flink or Spark Streaming.
The stream processing capability helps simplify system architecture and reduce operational complexity while delivering millisecond-level computation latency and high availability.
