Collecting and storing time-series data is only the first step. Many applications also need to send selected data to message queues, stream processing systems, alerting tools, or downstream business systems. TDengine data publishing addresses this outflow path by pushing database data to systems such as MQTT, Kafka, and Flink.
Why data publishing matters
A typical IIoT/IoT architecture has four layers:
- Collection layer: Sensors, PLCs generating raw data
- Storage layer: Time-series database for persistence and querying
- Processing layer: Stream processing engines for real-time analysis and event processing
- Application layer: Business systems consuming results for decision-making and automation
Data publishing bridges the storage and processing layers. Instead of leaving time-series data isolated in the database, it makes that data available to downstream consumers that handle alerts, analytics, applications, or stream processing.
Three integration platforms
MQTT integration
MQTT is a lightweight protocol with low bandwidth requirements, low power consumption, and strong network adaptability for IoT device communication. Use cases include:
- Device status push: Real-time device status delivered to mobile terminals for operations staff
- Cross-system data sharing: Publishing to an MQTT Broker for other systems to subscribe
- Alert notifications: Pushing anomaly detection results to alerting platforms
MQTT’s lightweight nature suits industrial field environments with limited network bandwidth and constrained device resources.
Kafka integration
Apache Kafka is a high-throughput distributed message queue widely used in big data collection, log aggregation, and stream processing. Key scenarios:
- Big data analysis pipeline: Writing time-series data to Kafka topics for consumption by Spark or Flink
- Log collection and auditing: Distributing operation logs via Kafka to analytics platforms
- Microservice decoupling: Kafka as middleware for asynchronous data exchange between microservices
Kafka’s high throughput and persistence make it ideal for large-scale data distribution.
Flink integration
Apache Flink is widely used for real-time data analysis and Complex Event Processing (CEP). Typical patterns:
- Real-time anomaly detection: Flink consumes time-series streams, using sliding windows and statistical models
- Complex event processing: The CEP engine identifies complex event patterns from multi-device interactions
- Real-time metric computation: Aggregating time-series data, generating real-time business metrics, writing results back to the database
This end-to-end integration minimizes latency from data collection to analysis and decision-making.
Architectural advantages
Compared to traditional “database + external extraction program” approaches:
- Deep integration, fewer intermediate steps: Built-in publishing pushes data from within the database, reducing intermediate hops and improving efficiency. Traditional approaches require external ETL programs or data pipelines, increasing complexity and latency.
- Unified management, lower operational cost: Publishing rules are configured and managed through the database management UI, without maintaining separate pipeline systems. Rules can be created, modified, started, and stopped through a visual interface.
- Strong real-time performance, preserving data timeliness: Real-time push mode publishes data to downstream systems immediately after writing, maximizing timeliness. This is critical for latency-sensitive scenarios like real-time monitoring and alerting.
Typical application scenarios
IoT data collection and distribution. In smart factories and smart energy deployments, massive numbers of devices continuously generate data. The publishing feature pushes device data to MQTT Brokers for mobile apps and dashboard systems.
Real-time monitoring and alerting. In operations monitoring, the time-series database stores key metrics. Publishing to Flink enables real-time anomaly detection and alert triggering for rapid response.
Big data analysis and mining. Time-series data often needs cross-system correlation with business and user behavior data. Distribution via Kafka to big data platforms enables data fusion and deeper analysis.
Summary
Data publishing and message integration are key capabilities for building modern data processing architectures. Through deep integration with MQTT, Kafka, and Flink, TDengine’s data publishing feature helps users complete the chain from storage to consumption, whether for lightweight IoT distribution, high-throughput big data pipelines, or end-to-end real-time processing. It helps teams move from stored data to usable real-time data flows without building a separate extraction pipeline for every downstream system.
