Start prototyping your IoT application in minutes on TDengine

TDengine is an open-source, purpose-built for IoT, high-performance, scalable time-series platform with support for standard SQL. It’s open-source version includes high-availability cluster features. While TDengine is a carrier-grade IoT time-series platform, it is very accessible to developers starting out in IoT. You can get started in minutes using a Docker container and below we will show you how.

If you haven’t already, you can download and install Docker Desktop at https://www.docker.com/products/docker-desktop.

Once you have installed Docker Desktop you can follow the steps below to run TDengine. Note that the example below is on macOS Big Sur Version 11.5.1 running on a MacBook Pro (Retina, Mid 2014, 2.5 GHz Quad-Core Intel Core i7)

Start Docker Desktop and then launch a terminal window.

In the terminal window type:

docker pull tdengine/tdengine

You should then see output like the following:

Using default tag: latest
latest: Pulling from tdengine/tdengine
2f94e549220a: Pull complete 
b58cc93d3f3a: Pull complete 
4bdb72dcb02d: Pull complete 
befab8ecc0f3: Pull complete 
f7b08f2ae8dc: Pull complete 
11730065df0e: Pull complete 
1f27396f6efc: Pull complete 
fe556ec02776: Pull complete 
Digest: sha256:fcea8ba46bbe8ee25cd8fc26bc6a8b78a4875d80ce618e7118372d4729451cb9
Status: Downloaded newer image for tdengine/tdengine:latest

You should then see the image in your Docker Desktop window.

Figure 1: TDengine image downloaded to Docker Desktop

Run the following command. Note that –name is used to name the running container, the -v option synchronizes the files inside the host and container so that data is not lost even after the container is deleted.

docker run -d --name "myiotapp" -v ~/work/taos/log:/var/log/taos-v ~/work/taos/data:/var/lib/taos  -p 6030-6041:6030-6041 -p 6030-6041:6030-6041/udp tdengine/tdengine

You will see output that looks like the following which is the container id:

53c14d4967ab593497fcebe9a044cc5951eb316195477478d30f43dd42b32b79

Now you should see the container running in Docker Desktop.

Figure 2: TDengine container running in Docker Desktop

You can also run a command in the terminal to see the container running:

docker ps

You should output that looks as follows:

CONTAINER ID   IMAGE                COMMAND                 CREATED          STATUS          PORTS                                                                                       
                                                            NAMES
53c14d4967ab   tdengine/tdengine   "/tini -- /usr/bin/e…"   46 minutes ago   Up 46 minutes   0.0.0.0:6030-6041->6030-6041/tcp, 0.0.0.0:6030-6041->6030-6041/udp, 6042-6049/tcp   iotprototype

Now you can connect to TDengine in the Docker container using the command line. Click on the CLI button and this will start a terminal window to connect to the container.

Figure 3: Connect to TDengine in Docker container

Essentially this is the same as typing the following in a terminal window. The long string is the container id, which is the output of the command which was used to run the Docker container. Since we named the container you could also replace the container id with the name which in our case is “iotprototype”.

docker exec -it 53c14d4967ab593497fcebe9a044cc5951eb316195477478d30f43dd42b32b79 /bin/sh

Now in the shell you can type “taos” to enter the Taos shell which gives you the ability to run ad hoc queries in TDengine.

# taos

Welcome to the TDengine shell from Linux, Client Version:2.4.0.10
Copyright (c) 2020 by TAOS Data, Inc. All rights reserved.

taos> show databases;
              name              |      created_time       |   ntables   |   vgroups   | replica | quorum |  days  |           keep           |  cache(MB)  |   blocks    |   minrows   |   maxrows   | wallevel |    fsync    | comp | cachelast | precision | update |   status   |
====================================================================================================================================================================================================================================================================================
 log                            | 2022-03-01 22:58:11.145 |          11 |           1 |       1 |      1 |     10 | 30                       |           1 |           3 |         100 |        4096 |        1 |        3000 |    2 |         0 | us        |      0 | ready      |
Query OK, 1 row(s) in set (0.002499s)

From your host machine, you can also access the RESTful interface of TDengine on port 6041. For example you can run the following in any host terminal window.

curl -u root:taosdata -d 'show databases' 127.0.0.1:6041/rest/sql

You should output like the following:

{"status":"succ","head":["name","created_time","ntables","vgroups","replica","quorum","days","keep","cache(MB)","blocks","minrows","maxrows","wallevel","fsync","comp","cachelast","precision","update","status"],"column_meta":[["name",8,32],["created_time",9,8],["ntables",4,4],["vgroups",4,4],["replica",3,2],["quorum",3,2],["days",3,2],["keep",8,24],["cache(MB)",4,4],["blocks",4,4],["minrows",4,4],["maxrows",4,4],["wallevel",2,1],["fsync",4,4],["comp",2,1],["cachelast",2,1],["precision",8,3],["update",2,1],["status",8,10]],"data":[["log","2022-03-01 22:58:11.145",11,1,1,1,10,"30",1,3,100,4096,1,3000,2,0,"us",0,"ready"]],"rows":1}

TDengine supplies a tool called “taosdemo” which creates 100 million records in a matter of minutes in the database. It does this by creating a “supertable” and tables for simulated sensors with 1 table per sensor. These are 2 novel concepts in TDengine that allow extremely high performance and scalability. You can read more about supertables in TDengine documentation.

# taosdemo

You should see output like the following:

taosBenchmark is simulating data generated by power equipment monitoring...

host:                       localhost
port:                       6030
telnet_tcp_port:            6046
user:                       root
password:                   taosdata
configDir:                  /etc/taos
resultFile:                 ./output.txt
thread num of insert data:  8
thread num of create table: 8
number of records per req:  30000
random prepare data size:   10000
chinese:                    no
database count:             1
database[0]:
  database[0] name:      test
  drop:                  yes
  precision:             ms
  super table count:     1
  super table[0]:
      stbName:           meters
      interface:         taosc
      autoCreateTable:   no
      childTblExists:    no
      childTblCount:     10000
      childTblLimit:     0
      childTblOffset:    0
      childTblPrefix:    d
      escapeCharacter:   no
      dataSource:        random
      insertRows:        10000
      interlaceRows:     0
      disorderRange:     1000
      disorderRatio:     0
      timeStampStep:     1
      startTimestamp:    1500000000000
      sampleFile:        
      useSampleTs:       no
      tagsFile:          
      partialColCount:   0
      columnCount:       3
        column[0]:float column[1]:int column[2]:float 
      tagCount:            2
        tag[0]:int tag[1]:binary(16) 

         Press enter key to continue or Ctrl-C to stop

Hit the “Enter” key. You will see output as follows:

[03/02 19:32:25.862158] INFO: create database test success!
[03/02 19:32:25.864828] INFO: stable meters does not exist
[03/02 19:32:25.868539] INFO: create stable meters success!
[03/02 19:32:25.878214] INFO: Estimate memory usage: 21.91MB
         Press enter key to continue or Ctrl-C to stop

Hit the “Enter” key again. You will see output as follows:

[03/02 19:33:54.092017] INFO: start creating 10000 table(s) with 8 thread(s)
[03/02 19:34:06.254369] INFO: Spent 12.1620 seconds to create 10000 table(s) with 8 thread(s), already exist 0 table(s), actual 10000 table(s) pre created, 0 table(s) will be auto created

[03/02 19:34:36.264972] INFO: thread[2] has currently inserted rows: 1410000, affected rows: 1410000
[03/02 19:34:36.267614] INFO: thread[4] has currently inserted rows: 1440000, affected rows: 1440000
[03/02 19:34:36.276656] INFO: thread[5] has currently inserted rows: 1470000, affected rows: 1470000
[03/02 19:34:36.294200] INFO: thread[1] has currently inserted rows: 1540000, affected rows: 1540000
[03/02 19:34:36.294271] INFO: thread[3] has currently inserted rows: 1510000, affected rows: 1510000
[03/02 19:34:36.296037] INFO: thread[7] has currently inserted rows: 1500000, affected rows: 1500000
[03/02 19:34:36.328749] INFO: thread[6] has currently inserted rows: 1480000, affected rows: 1480000
[03/02 19:34:36.349476] INFO: thread[0] has currently inserted rows: 1470000, affected rows: 1470000
.......
[03/02 19:38:19.059402] INFO: thread[6] completed total inserted rows: 12500000, total affected rows: 12500000. 50122.71 records/second
[03/02 19:38:26.623064] INFO: thread[7] completed total inserted rows: 12500000, total affected rows: 12500000. 48664.23 records/second
[03/02 19:38:26.629404] INFO: Spent 260.3635 seconds to insert rows: 100000000, affected rows: 100000000 with 8 thread(s) into test 384078.45 records/second

[03/02 19:38:26.629493] INFO: insert delay, min: 16.81ms, avg: 183.65ms, p90: 101.45ms, p95: 127.37ms, p99: 5698.26ms, max: 18446744073709552.00ms

It takes just 5 min to insert 100 million rows on a Docker container running on a MacBook Pro so you can imagine the performance on a server in production. You can now check the database to see how fast TDengine is. Notice how standard SQL is supported and extended functions are provided.

taos> use test;
Database changed.

taos> show stables;
              name              |      created_time       | columns |  tags  |   tables    |
============================================================================================
 meters                         | 2022-03-02 19:32:25.865 |       4 |      2 |       10000 |
Query OK, 1 row(s) in set (0.002087s)

taos> select count(*) from meters;
       count(*)        |
========================
             100000000 |
Query OK, 1 row(s) in set (0.427062s)

taos> describe meters;
             Field              |         Type         |   Length    |   Note   |
=================================================================================
 ts                             | TIMESTAMP            |           8 |          |
 current                        | FLOAT                |           4 |          |
 voltage                        | INT                  |           4 |          |
 phase                          | FLOAT                |           4 |          |
 groupid                        | INT                  |           4 | TAG      |
 location                       | BINARY               |          16 | TAG      |
Query OK, 6 row(s) in set (0.000373s)

taos> select distinct(location) from meters limit 10;
     location     |
===================
 0cFUFXT44cSpMOyO |
 0hGD9EwzTNhJ54NN |
 0rmS6tfwKcMyqZw9 |
 2M7NiPqfqJlyy3qV |
 7vtHw1y74n5jYR6i |
 BnFDHWSmdDnQJeQu |
 DfTul6MV6bWfRsci |
 GdVZGOltAKyxtqkX |
 H6QrluSyJl0DgPPO |
 HBL1wyVt4lU9vFkU |
Query OK, 10 row(s) in set (0.012356s)

taos> select count(*) from meters where location='0cFUFXT44cSpMOyO';
       count(*)        |
========================
                 10000 |
Query OK, 1 row(s) in set (0.013081s)

taos> select avg(voltage), avg(current) from meters where location='0cFUFXT44cSpMOyO';
       avg(voltage)        |       avg(current)        |
========================================================
             214.998700000 |               9.799952337 |
Query OK, 1 row(s) in set (0.012560s)

Hopefully by now you have some idea of how easy it is to install and run TDengine and how blazingly fast it is. In a future blog we will show you how to insert your own data very easily from a ubiquitous file format like a CSV file or migrate from an existing time-series database like InfluxDB or Timescale DB.