IoT and Industrial IoT deployments generate large volumes of time-series data from sensors, devices, and machinery. Extracting real-time insights from this data, rather than simply storing it, is where the operational value appears. A common architecture is to export data to external stream processing frameworks like Flink or Spark Streaming for computation. This architecture adds complexity, cost, and latency.
TDengine takes a different approach. Its built-in stream processing engine handles real-time window aggregation, event-driven computation, and continuous queries directly inside the database. This avoids external frameworks or data movement.
Why built-in stream processing?
Traditional architectures that pair a database with an external stream processor create three core problems:
High deployment complexity. Flink and Spark clusters need separate compute resources, ZooKeeper coordination, and state storage backends. Setting up and configuring these components takes careful setup.
Unavoidable data latency. Data must travel from the database to the stream processor, going through network transport and serialization/deserialization at each stage. End-to-end latency typically reaches seconds.
Ongoing operational cost. Stream processing frameworks need dedicated ops teams to handle monitoring, scaling, and recovery. Each additional component multiplies the maintenance burden.
TDengine embeds stream processing logic directly in the database kernel. Computation runs where the data lives, achieving millisecond-level latency with less operational overhead.
Core capabilities
Window aggregation
Window aggregation is the foundation of stream processing. It groups incoming data into bounded time or count windows and computes results over each window.
Time windows use fixed time spans. A common example is computing the average temperature of a device every five minutes. Time windows come in two forms:
- Tumbling windows have no overlap between consecutive windows. Each data point belongs to exactly one window.
- Sliding windows allow overlap. A five-minute window that slides forward by one minute recalculates results every minute using the most recent five minutes of data.
Count windows trigger aggregation when a specified number of rows accumulate. This is useful when data arrives at uneven rates and time-based windows would produce inconsistent batch sizes.
Session windows divide data dynamically based on gaps between data points. When the gap between consecutive events exceeds a threshold, the session closes and a new one begins. This pattern works well for user behavior analysis and equipment run-cycle detection.
Event-driven triggering
TDengine stream tasks fire computation the moment new data is written. There is no polling loop and no idle CPU consumption. The write path and the computation path are integrated, so latency stays at the millisecond level.
Continuous queries
Stream tasks run persistently in the background. They perform incremental computation, meaning each new batch of data updates the results without reprocessing the entire history. Results are written automatically to target tables, where they can be queried like any other data.
Defining stream processing with SQL
TDengine exposes stream processing through standard SQL. There is no separate API or DSL to learn. The core syntax is CREATE STREAM:
CREATE STREAM temp_alert_stream
TRIGGER AT_ONCE
AS SELECT _wstart, _wend, tbname, avg(temperature) AS avg_temp, max(temperature) AS max_temp
FROM factory_sensors
WHERE temperature > 80
PARTITION BY tbname
WINDOW(5m, 1m)
INTO temp_alert_table;
This stream task reads continuously from the factory_sensors Supertable. It uses five-minute windows with one-minute sliding steps, partitions computation by device name (tbname), and filters for rows where temperature exceeds 80. For each window, it computes the average and maximum temperature, then writes the results into temp_alert_table.
A stream task, once created, runs indefinitely. It can be paused, resumed, or dropped at any time.
Typical application scenarios
Real-time anomaly detection
Continuous monitoring of temperature, pressure, and vibration thresholds is one of the most common industrial use cases. A stream task can watch incoming sensor data and fire alerts the moment a value crosses a defined boundary. This shifts the response model from “find the problem in yesterday’s report” to “catch the problem as it happens.”
Predictive maintenance
Beyond simple threshold alerts, stream tasks can compute derived signals that indicate equipment degradation. Calculating the rate of temperature change or tracking frequency drift in vibration data over rolling windows surfaces patterns that precede failure. Maintenance moves from reactive repair to proactive prevention.
Real-time energy statistics
Smart buildings and factories need up-to-the-minute visibility into energy consumption. Stream tasks aggregate meter readings by floor, zone, or usage type, feeding live dashboards and enabling operators to spot anomalies in energy use as they develop.
Comparison with external frameworks
| Dimension | Built-in stream processing | Flink / Spark Streaming |
|---|---|---|
| Deployment | Built into the database | Requires independent cluster |
| End-to-end latency | Millisecond-level | Second-level |
| Operational cost | No separate service | Requires a dedicated operations path |
| Learning curve | Standard SQL | Framework-specific API |
| Data migration | Not required | Must export from database |
| Scalability | Scales with database cluster | Independent scaling |
TDengine’s built-in engine covers many stream processing needs for time-series data. External frameworks are only worth the added complexity when the workload involves cross-system event processing, such as multi-source joins across heterogeneous systems or machine learning inference pipelines.
Hands-on: IoT temperature anomaly detection
Step 1: Create the data table
CREATE STABLE factory_sensors (ts TIMESTAMP, temperature FLOAT, humidity FLOAT)
TAGS (device_id INT, factory_zone NCHAR(50));
This Supertable defines the schema for all sensor devices. Each device becomes a Subtable with its own device_id and factory_zone Tags.
Step 2: Create the stream task
CREATE STREAM temp_monitor
TRIGGER AT_ONCE
WATERMARK 10s
AS SELECT _wstart AS window_start, _wend AS window_end,
tbname AS device,
avg(temperature) AS avg_temp,
max(temperature) AS max_temp,
count(*) AS sample_count
FROM factory_sensors
PARTITION BY tbname
WINDOW(1m, 30s)
WHEN avg_temp > 75 OR max_temp > 90
INTO temp_anomaly_alerts;
The task uses one-minute windows with 30-second slides. The WATERMARK 10s clause allows a 10-second tolerance for out-of-order data. The WHEN clause acts as a post-aggregation filter: only windows where the average temperature exceeds 75 or the peak temperature exceeds 90 are written to the output table.
Step 3: Query alert results
SELECT * FROM temp_anomaly_alerts
WHERE ts > NOW() - 1h
ORDER BY ts DESC;
Alerts appear in temp_anomaly_alerts as soon as each window closes. The table behaves like any other TDengine table and supports full SQL querying, including time-range filters and ordering.
Conclusion
Running computation where the data is stored reduces architectural complexity and eliminates whole categories of latency and operational cost. TDengine’s built-in stream processing engine uses standard SQL, event-driven execution, and millisecond-level latency as its design targets. It serves IoT, Industrial IoT, and smart city scenarios where fast, continuous computation over sensor data is essential. For a complete syntax reference and additional case studies, see the official documentation.
