MQTT in TDengine

As adoption of Industrial IoT (IIoT) technologies increases across manufacturing, energy, and other industries, it can become difficult to adapt traditional data protocols to the new use cases and requirements of the IIoT. For example, devices may be connected over wireless networks where connectivity cannot be guaranteed, and the scale of IIoT deployments may pose difficulties to client/server architectures due to the volume of messages transmitted. In these scenarios, MQTT can be a suitable choice for ensuring communication between devices and the data historian or other platform responsible for storing collected data.

What Is MQTT?

MQTT is a lightweight messaging protocol developed to meet the needs of IoT applications — in particular, constrained devices and unreliable networks with low bandwidth and high latency. It facilitates communication between connected devices, sensors, and back-end systems in a reliable and efficient manner with a lower overhead compared with other protocols.

A key feature of MQTT is its publish-subscribe (pub-sub) messaging pattern, in which clients called publishers publish messages to a central broker, and other clients called subscribers subscribe to specific topics to receive those messages. This model decouples data-generating devices from data-ingesting systems, enabling efficient and scalable communication without requiring a direct connection between them. Devices can publish or subscribe to topics without knowledge of other devices’ network configurations, and a single message published on a topic can be received by multiple subscribers, enhancing the flexibility of the overall system while reducing network bandwidth consumption.

MQTT was made public in 2010 with version 3.1 and became an OASIS standard with version 3.1.1 in 2014. The latest version, MQTT 5.0, was initially released in 2018 and greatly enhances the scalability of the protocol, ensuring that brokers can handle IIoT deployments of any size. In addition to standard MQTT, extensions such as Sparkplug can further simplify and enhance IIoT systems.

How MQTT Modernizes Industrial Data Infrastructure

At present, industrial data infrastructure often includes a wide variety of proprietary protocols, necessitating a data historian that can communicate with PLCs over many different interfaces. However, modern PLCs are quickly standardizing around open protocols such as MQTT and OPC-UA, meaning that the PLC-to-historian data flow will be greatly simplified in greenfield deployments going forward. This will enable enterprises that select PLCs built for the IIoT to forgo traditional historians entirely and connect their MQTT brokers directly to a next generation data historian, greatly reducing their total cost of operation (TCO).

In existing deployments, MQTT facilitates the integration of legacy industrial systems with modern IoT platforms, providing a pathway for the digital transformation of industrial operations. Its support for secure communication via TLS, along with various levels of quality of service (QoS), ensures that sensitive industrial data is transmitted securely and reliably. These characteristics allow for real-time monitoring and control of industrial processes and support advanced use cases such as predictive maintenance and analytics. By leveraging MQTT, industries can create more responsive, adaptive, and interconnected systems that can easily scale with the addition of new devices and technologies.

How TDengine Supports MQTT

As a next generation data historian, TDengine can ingest and store data from multiple MQTT sources as well as other modern protocols like OPC and traditional historians such as PI System and Wonderware. A major advantage of using TDengine as a unified platform for MQTT data is that ETL can be performed on the platform side for multiple brokers and sites. This ensures good data governance even when different systems are in use, centralizing all data with its full context in TDengine. Analytics, visualization, and other tools can then be integrated with TDengine and work with data from all systems without conflicts, convoluted filtering, or manual workloads.

TDengine provides a zero-code connector for MQTT that TDengine Enterprise and TDengine Cloud customers can use at no extra cost. The connector enables real-time ingestion of data from MQTT 3.1, 3.11, and 5.0 sources into the unified TDengine platform. To begin ingesting MQTT data, create a database in TDengine to store the data, configure the MQTT server endpoint, and specify a QoS configuration for your topics.

After connecting to the MQTT broker, you can upload a sample of your MQTT data into the MQTT Payload field and begin the extract, transform, and load (ETL) process.

1. Parse: You use JSON or a regular expression (regex) to specify the MQTT data that you want to ingest. For example, the expression (?<y>[0-9]{4})-(?<m>[0-9]{2})-(?<d>[0-9]{2}) ingests timestamp data in YYYY-MM-DD format.
2. Extract: You can select Split to extract multiple columns from your data using a delimiter. For example, if you have hierarchical data such as California.LosAngeles, you can specify a period (.) as the delimiter and 2 as the number of columns to extract this data into separate state (California) and city (LosAngeles) columns. For more complex extraction, select Regex and input a regular expression.
3. Filter: You can filter out data based on user-specified conditions. Only data that meets the conditions is ingested into TDengine. For example, you can input voltage > 220 to ingest only data whose voltage is greater than 220. In addition to comparisons, Boolean and string conditions are also supported.
4. Transform: You can transform your data into each column of the specified supertable. For example, select Mapping and the name of the desired column to map a key in your MQTT data to a column in TDengine. Note that key names that match existing column names are mapped automatically. Other transformation options include value, generator, join, format, sum, and expression.

After you have configured ETL for your data, click Add, and TDengine will begin ingesting data according to your specifications. For detailed instructions on setting up MQTT as a data source, see the official documentation.