Last month we announced the new Data Sources — a robust and easy-to-use data transfer interface within TDengine, and we gave a demo on how to Ingest Data from a PI System. In this blog post we show a demo of how to use TDengine Cloud to ingest data directly from a MQTT broker. This capability to configure data interfaces to ingest data is available in both TDengine Cloud and TDengine Enterprise.
Setup is as easy as downloading the data collection agent, installing it on the network where the MQTT broker is, and providing the agent with the endpoint where you would like to send the data. From there, everything is configured from the TDengine Data Sources page — just configure the Server Endpoint and parse the MQTT payload and watch the data flow into TDengine.
Setting up the MQTT Demo with TDengine Cloud
Overall, the demo took the following steps:
- Create a Linux VM (Ubuntu 20.04 was used)
- Install Docker
- Install and connect to Hive MQ (docker)
- Install and configure TaosX-Agent
- Setup MQTT Data Source in TDengine Cloud
- Send data to the MQTT Broker and view the results in TDengine
Create the Linux VM and SSH into it
Created a Ubuntu 20.04 Linux VM on Microsoft Azure, downloaded the pem and connected via SSH (example connection below)
ssh -i ~/.ssh/linux-vm_key.pem email@example.com
Once logged into the VM, setup docker community edition
sudo apt update
sudo apt install software-properties-common curl apt-transport-https ca-certificates -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker-archive-keyring.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt install docker-ce docker-ce-cli containerd.io uidmap -y
sudo usermod -aG docker $USER
With Docker up and running, pull and start a HiveMQ community edition container
docker run --name hivemq-ce -d -p 1883:1883 hivemq/hivemq-ce
Connect to HiveMQ
Prior to connecting to HiveMQ, open up port 1883 inbound in the firewall for the Linux VM
Using an MQTT client tool, setup the connection to the linux VM. Here the connection using MQTTX is shown.
Created a new connection agent in TDengine cloud and copied the configuration into the agent.toml file
scp -i ~/.ssh/linux-vm_key.pem ~/code/taosx-mqtt/taosx-agent-1.2.1-linux-x64.tar.gz firstname.lastname@example.org:taosx-agent-1.2.3-linux-x64.tar.gz tar -xzf taosx-agent-1.2.3-linux-x64.tar.gz cd taosx-agent-1.2.3-linux-x64 sh install.sh cd /etc/taos/ sudo vi agent.toml sudo systemctl start taosx-agent
Create a Database in TDengine Cloud
For this demo, used default values (retain data for 10 years, ms timestamp precision)
Data Source Setup in TDengine Cloud
Configure the datasource to read MQTT data using the agent and database created earlier.
Send Data to the MQTT Broker
View the Results in TDengine
Tables are automatically created based on the MQTT configuration.