TDengine is an open-source, purpose-built for IoT, high-performance, scalable time-series platform with support for standard SQL. It’s open-source version includes high-availability cluster features. While TDengine is a carrier-grade IoT time-series platform, it is very accessible to developers starting out in IoT. You can get started in minutes using a Docker container and below we will show you how.
If you haven’t already, you can download and install Docker Desktop at https://www.docker.com/products/docker-desktop.
Once you have installed Docker Desktop you can follow the steps below to run TDengine. Note that the example below is on macOS Big Sur Version 11.5.1 running on a MacBook Pro (Retina, Mid 2014, 2.5 GHz Quad-Core Intel Core i7)
Start Docker Desktop and then launch a terminal window.
In the terminal window type:
docker pull tdengine/tdengineYou should then see output like the following:
Using default tag: latest latest: Pulling from tdengine/tdengine 2f94e549220a: Pull complete b58cc93d3f3a: Pull complete 4bdb72dcb02d: Pull complete befab8ecc0f3: Pull complete f7b08f2ae8dc: Pull complete 11730065df0e: Pull complete 1f27396f6efc: Pull complete fe556ec02776: Pull complete Digest: sha256:fcea8ba46bbe8ee25cd8fc26bc6a8b78a4875d80ce618e7118372d4729451cb9 Status: Downloaded newer image for tdengine/tdengine:latest
You should then see the image in your Docker Desktop window.
Run the following command. Note that –name is used to name the running container, the -v option synchronizes the files inside the host and container so that data is not lost even after the container is deleted.
docker run -d --name "myiotapp" -v ~/work/taos/log:/var/log/taos-v ~/work/taos/data:/var/lib/taos -p 6030-6041:6030-6041 -p 6030-6041:6030-6041/udp tdengine/tdengineYou will see output that looks like the following which is the container id:
53c14d4967ab593497fcebe9a044cc5951eb316195477478d30f43dd42b32b79
Now you should see the container running in Docker Desktop.
You can also run a command in the terminal to see the container running:
docker psYou should output that looks as follows:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
53c14d4967ab tdengine/tdengine "/tini -- /usr/bin/e…" 46 minutes ago Up 46 minutes 0.0.0.0:6030-6041->6030-6041/tcp, 0.0.0.0:6030-6041->6030-6041/udp, 6042-6049/tcp iotprototype
Now you can connect to TDengine in the Docker container using the command line. Click on the CLI button and this will start a terminal window to connect to the container.
Essentially this is the same as typing the following in a terminal window. The long string is the container id, which is the output of the command which was used to run the Docker container. Since we named the container you could also replace the container id with the name which in our case is “iotprototype”.
docker exec -it 53c14d4967ab593497fcebe9a044cc5951eb316195477478d30f43dd42b32b79 /bin/shNow in the shell you can type “taos” to enter the Taos shell which gives you the ability to run ad hoc queries in TDengine.
# taos
Welcome to the TDengine shell from Linux, Client Version:2.4.0.10
Copyright (c) 2020 by TAOS Data, Inc. All rights reserved.
taos> show databases;
name | created_time | ntables | vgroups | replica | quorum | days | keep | cache(MB) | blocks | minrows | maxrows | wallevel | fsync | comp | cachelast | precision | update | status |
====================================================================================================================================================================================================================================================================================
log | 2022-03-01 22:58:11.145 | 11 | 1 | 1 | 1 | 10 | 30 | 1 | 3 | 100 | 4096 | 1 | 3000 | 2 | 0 | us | 0 | ready |
Query OK, 1 row(s) in set (0.002499s)
From your host machine, you can also access the RESTful interface of TDengine on port 6041. For example you can run the following in any host terminal window.
curl -u root:taosdata -d 'show databases' 127.0.0.1:6041/rest/sqlYou should output like the following:
{"status":"succ","head":["name","created_time","ntables","vgroups","replica","quorum","days","keep","cache(MB)","blocks","minrows","maxrows","wallevel","fsync","comp","cachelast","precision","update","status"],"column_meta":[["name",8,32],["created_time",9,8],["ntables",4,4],["vgroups",4,4],["replica",3,2],["quorum",3,2],["days",3,2],["keep",8,24],["cache(MB)",4,4],["blocks",4,4],["minrows",4,4],["maxrows",4,4],["wallevel",2,1],["fsync",4,4],["comp",2,1],["cachelast",2,1],["precision",8,3],["update",2,1],["status",8,10]],"data":[["log","2022-03-01 22:58:11.145",11,1,1,1,10,"30",1,3,100,4096,1,3000,2,0,"us",0,"ready"]],"rows":1}
TDengine supplies a tool called “taosdemo” which creates 100 million records in a matter of minutes in the database. It does this by creating a “supertable” and tables for simulated sensors with 1 table per sensor. These are 2 novel concepts in TDengine that allow extremely high performance and scalability. You can read more about supertables in TDengine documentation.
# taosdemoYou should see output like the following:
taosBenchmark is simulating data generated by power equipment monitoring...
host: localhost
port: 6030
telnet_tcp_port: 6046
user: root
password: taosdata
configDir: /etc/taos
resultFile: ./output.txt
thread num of insert data: 8
thread num of create table: 8
number of records per req: 30000
random prepare data size: 10000
chinese: no
database count: 1
database[0]:
database[0] name: test
drop: yes
precision: ms
super table count: 1
super table[0]:
stbName: meters
interface: taosc
autoCreateTable: no
childTblExists: no
childTblCount: 10000
childTblLimit: 0
childTblOffset: 0
childTblPrefix: d
escapeCharacter: no
dataSource: random
insertRows: 10000
interlaceRows: 0
disorderRange: 1000
disorderRatio: 0
timeStampStep: 1
startTimestamp: 1500000000000
sampleFile:
useSampleTs: no
tagsFile:
partialColCount: 0
columnCount: 3
column[0]:float column[1]:int column[2]:float
tagCount: 2
tag[0]:int tag[1]:binary(16)
Press enter key to continue or Ctrl-C to stop
Hit the “Enter” key. You will see output as follows:
[03/02 19:32:25.862158] INFO: create database test success!
[03/02 19:32:25.864828] INFO: stable meters does not exist
[03/02 19:32:25.868539] INFO: create stable meters success!
[03/02 19:32:25.878214] INFO: Estimate memory usage: 21.91MB
Press enter key to continue or Ctrl-C to stop
Hit the “Enter” key again. You will see output as follows:
[03/02 19:33:54.092017] INFO: start creating 10000 table(s) with 8 thread(s) [03/02 19:34:06.254369] INFO: Spent 12.1620 seconds to create 10000 table(s) with 8 thread(s), already exist 0 table(s), actual 10000 table(s) pre created, 0 table(s) will be auto created [03/02 19:34:36.264972] INFO: thread[2] has currently inserted rows: 1410000, affected rows: 1410000 [03/02 19:34:36.267614] INFO: thread[4] has currently inserted rows: 1440000, affected rows: 1440000 [03/02 19:34:36.276656] INFO: thread[5] has currently inserted rows: 1470000, affected rows: 1470000 [03/02 19:34:36.294200] INFO: thread[1] has currently inserted rows: 1540000, affected rows: 1540000 [03/02 19:34:36.294271] INFO: thread[3] has currently inserted rows: 1510000, affected rows: 1510000 [03/02 19:34:36.296037] INFO: thread[7] has currently inserted rows: 1500000, affected rows: 1500000 [03/02 19:34:36.328749] INFO: thread[6] has currently inserted rows: 1480000, affected rows: 1480000 [03/02 19:34:36.349476] INFO: thread[0] has currently inserted rows: 1470000, affected rows: 1470000 ....... [03/02 19:38:19.059402] INFO: thread[6] completed total inserted rows: 12500000, total affected rows: 12500000. 50122.71 records/second [03/02 19:38:26.623064] INFO: thread[7] completed total inserted rows: 12500000, total affected rows: 12500000. 48664.23 records/second [03/02 19:38:26.629404] INFO: Spent 260.3635 seconds to insert rows: 100000000, affected rows: 100000000 with 8 thread(s) into test 384078.45 records/second [03/02 19:38:26.629493] INFO: insert delay, min: 16.81ms, avg: 183.65ms, p90: 101.45ms, p95: 127.37ms, p99: 5698.26ms, max: 18446744073709552.00ms
It takes just 5 min to insert 100 million rows on a Docker container running on a MacBook Pro so you can imagine the performance on a server in production. You can now check the database to see how fast TDengine is. Notice how standard SQL is supported and extended functions are provided.
taos> use test;
Database changed.
taos> show stables;
name | created_time | columns | tags | tables |
============================================================================================
meters | 2022-03-02 19:32:25.865 | 4 | 2 | 10000 |
Query OK, 1 row(s) in set (0.002087s)
taos> select count(*) from meters;
count(*) |
========================
100000000 |
Query OK, 1 row(s) in set (0.427062s)
taos> describe meters;
Field | Type | Length | Note |
=================================================================================
ts | TIMESTAMP | 8 | |
current | FLOAT | 4 | |
voltage | INT | 4 | |
phase | FLOAT | 4 | |
groupid | INT | 4 | TAG |
location | BINARY | 16 | TAG |
Query OK, 6 row(s) in set (0.000373s)
taos> select distinct(location) from meters limit 10;
location |
===================
0cFUFXT44cSpMOyO |
0hGD9EwzTNhJ54NN |
0rmS6tfwKcMyqZw9 |
2M7NiPqfqJlyy3qV |
7vtHw1y74n5jYR6i |
BnFDHWSmdDnQJeQu |
DfTul6MV6bWfRsci |
GdVZGOltAKyxtqkX |
H6QrluSyJl0DgPPO |
HBL1wyVt4lU9vFkU |
Query OK, 10 row(s) in set (0.012356s)
taos> select count(*) from meters where location='0cFUFXT44cSpMOyO';
count(*) |
========================
10000 |
Query OK, 1 row(s) in set (0.013081s)
taos> select avg(voltage), avg(current) from meters where location='0cFUFXT44cSpMOyO';
avg(voltage) | avg(current) |
========================================================
214.998700000 | 9.799952337 |
Query OK, 1 row(s) in set (0.012560s)
Hopefully by now you have some idea of how easy it is to install and run TDengine and how blazingly fast it is. In a future blog we will show you how to insert your own data very easily from a ubiquitous file format like a CSV file or migrate from an existing time-series database (TSDB) like InfluxDB or Timescale DB.
