For many time-series applications, storing and querying raw data is not enough. Teams also need real-time aggregation, downsampling, pre-computation, anomaly detection, and low-latency alerting as data arrives. TDengine’s built-in stream processing engine uses SQL to define these workflows inside the database architecture.
Pain points of traditional stream processing
Deploying Kafka + Flink or Spark Streaming introduces four pain points:
- High development cost: Requires writing Java or Scala programs with complex operator chains
- Heavy operational burden: Separate monitoring, tuning, and failover for Kafka and Flink clusters
- Long data paths: Data flows DB → Kafka → Flink → back to DB, resulting in high latency
- Fragmented technology stack: Teams must master multiple technology stacks
When streaming requirements are mainly transformation, aggregation, and anomaly detection on time-series data, a full external stream processing framework may add more complexity than the workload requires.
Core design of the built-in stream processing engine
TDengine’s engine uses SQL to define real-time stream transformations. Data written to a source table is automatically processed and results are pushed to a destination table based on configured trigger modes, supporting low-latency computation under high-throughput writes.
Three key advantages of the SQL-driven approach:
- Low learning curve: SQL familiarity is sufficient
- Declarative definition: Describe what to compute rather than how to compute
- Seamless storage integration: Results write directly into database tables
Three core extension capabilities
1. Extended processing scope: trigger-compute decoupling
The engine decouples the trigger source from the computation source. New data arriving in table A can trigger a computation whose data source is table B.
2. Extended trigger types: multiple windows and filters
Six trigger types: window trigger, scheduled trigger, session window trigger, state window trigger, event window trigger, and count-based window trigger.
A PRE_FILTER mechanism allows filtering data before it enters the computation pipeline.
3. Extended computation scope: cross-table and cross-database
Stream computation can operate across tables and databases, supporting any valid query including aggregation, joins, and scalar computation.
Flexible trigger action configuration
Three output trigger actions: notify only (WebSocket push), compute only (write results to output table), or both.
Key control options
| Parameter | Description |
|---|---|
| WATERMARK | Defines data delay tolerance for handling out-of-order data |
| EXPIRED_TIME | Window expiration time for result retention |
| IGNORE_DISORDER | Whether to ignore out-of-order data |
| DELETE_OUTPUT_TABLE | Whether to auto-clean output table on stream deletion |
| FILL_HISTORY | Whether to backfill computation for pre-existing historical data |
| LOW_LATENCY_CALC | Enable low-latency computation mode |
| PRE_FILTER | Pre-filter condition for trigger data |
| MAX_DELAY | Maximum allowed delay balancing latency and throughput |
WATERMARK and IGNORE_DISORDER are critical for handling out-of-order data, a common problem in industrial scenarios with unstable networks. FILL_HISTORY determines whether pre-existing data should be included in computation upon stream creation.
High-availability architecture: storage-compute separation and load balancing
The architecture features:
- snode (Stream Node): Independent compute nodes for running stream jobs
- Multi-snode load balancing: Automatic task distribution across snodes
- Mutual replicas: Snodes act as replicas of each other; upon failure, other nodes automatically take over
This design allows horizontal scaling of compute capacity independently of storage nodes.
Continuous anomaly detection
The engine supports real-time anomaly detection via the ANOMALY_WINDOW function. Combined with WebSocket notification, anomalies can trigger immediate alerts, providing end-to-end real-time anomaly response.
Comparison with traditional approaches
| Dimension | Built-in stream processing | Kafka + Flink |
|---|---|---|
| Definition method | SQL | Java/Scala programs |
| Deployed components | Database only | Kafka + Flink + DB |
| Operations complexity | Low | High |
| Data path | Inside database | Cross-system |
| Learning cost | SQL only | Requires stream framework knowledge |
| Result storage | Direct table write | Requires writing back to DB |
For Complex Event Processing (CEP), broad multi-stream joins, and heterogeneous source processing, external frameworks can still be the better fit. For IoT and industrial workloads centered on time-series aggregation and transformation, TDengine’s built-in engine can reduce architecture complexity and data movement.
Summary
Stream processing turns stored time-series data into continuously updated results. TDengine’s engine uses SQL as its core interface, with decoupled trigger and computation, multiple window types, and cross-table/cross-database capabilities. Combined with a storage-compute separation architecture for high availability and millisecond-level computation latency, it provides production-grade real-time computation without adding system complexity.
