Ingesting Data from MQTT

Last month we announced the new Data Sources  — a robust and easy-to-use data transfer interface within the TDengine data historian, and we gave a demo on how to Ingest Data from a PI System. In this blog post we show a demo of how to use TDengine Cloud to ingest data directly from a MQTT broker. This capability to configure data interfaces to ingest data is available in both TDengine Cloud and TDengine Enterprise.

Setup is as easy as downloading the data collection agent, installing it on the network where the MQTT broker is, and providing the agent with the endpoint where you would like to send the data. From there, everything is configured from the TDengine Data Sources page — just configure the Server Endpoint and parse the MQTT payload and watch the data flow into TDengine.

Setting up the MQTT Demo with TDengine Cloud

Overall, the demo took the following steps:

1. Create a Linux VM (Ubuntu 20.04 was used)
2. Install Docker
3. Install and connect to Hive MQ (docker)
4. Install and configure TaosX-Agent
5. Setup MQTT Data Source in TDengine Cloud
6. Send data to the MQTT Broker and view the results in TDengine

Create the Linux VM and SSH into it

Created a Ubuntu 20.04 Linux VM on Microsoft Azure, downloaded the pem and connected via SSH (example connection below)

ssh -i ~/.ssh/linux-vm_key.pem azureuser@20.118.128.81

Install Docker

Once logged into the VM, setup docker community edition

sudo apt update
sudo apt install software-properties-common curl apt-transport-https ca-certificates -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker-archive-keyring.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt install docker-ce docker-ce-cli containerd.io uidmap -y
sudo usermod -aG docker $USER
newgrp docker

Install HiveMQ

With Docker up and running, pull and start a HiveMQ community edition container

docker run --name hivemq-ce -d -p 1883:1883 hivemq/hivemq-ce

Connect to HiveMQ

Prior to connecting to HiveMQ, open up port 1883 inbound in the firewall for the Linux VM

Using an MQTT client tool, setup the connection to the linux VM. Here the connection using MQTTX is shown.

Install TaosX-Agent

Created a new connection agent in TDengine cloud and copied the configuration into the agent.toml file

scp -i ~/.ssh/linux-vm_key.pem ~/code/taosx-mqtt/taosx-agent-1.2.1-linux-x64.tar.gz azureuser@20.118.128.81:taosx-agent-1.2.3-linux-x64.tar.gz
tar -xzf taosx-agent-1.2.3-linux-x64.tar.gz
cd taosx-agent-1.2.3-linux-x64
sh install.sh
cd /etc/taos/
sudo vi agent.toml
sudo systemctl start taosx-agent

Create a Database in TDengine Cloud

For this demo, used default values (retain data for 10 years, ms timestamp precision)

Data Source Setup in TDengine Cloud

Configure the datasource to read MQTT data using the agent and database created earlier.

Send Data to the MQTT Broker

View the Results in TDengine

Tables are automatically created based on the MQTT configuration.