High-performance time-series database

By making full use of the characteristics of time series big data, the TDengine team designed a novel storage engine, which greatly improves the data ingestion and query speed, as well as the data compression ratio. With a single core, it can process at least 20,000 requests per second, insert millions of data points, and read more than 10 million data points in one second. It is at least 10 times faster than general databases, and data compression ratio is at least 5 times higher.

Compared with the popular InfluxDB, using the test framework, data sets and test scenarios provided by InfluxDB itself (https://www.influxdata.com/blog/influxdb-outperforms-graphite-in-time-series-data-metrics-benchmark/), the ingestion throughput is at least 2 times higher, and when the number of devices exceeds 1000, the ingestion throughput is 5.2 times. For query scenarios, a few have the same speed as InfluxDB, while the majority are several times faster, and some are up to 40 times faster. For data compression, both TDengine and InfluxDB have very good performance, but TDengine’s storage size is only 70% of InfluxDB’s.

Figure 1 shows the time, in seconds, taken to insert all the data points. It’s very clear that TDengine outperforms InfluxDB as the number of devices increase. When the number of devices is 1000, InfluxDB takes 5.2 times longer to get all the data inserted. For a complete ingestion comparison test with InfluxDB, including detailed steps and results, see Writing Performance Comparison of TDengine and InfluxDB.

Figure 2 shows the time, in seconds, taken to fetch all data points for a certain number of devices. When the number of devices is 16, InfluxDB takes 8.78 times longer. When the number of devices is 128, InfluxDB takes 9.37 times longer time than TDengine. When the number of devices is 256, InfluxDB is not able to respond. For a complete query comparison test with InfluxDB, including detailed steps and results, see Query Performance Comparison Test Report: TDengine vs. InfluxDB.

We tried to increase the number of devices to 10000, 1000000 or higher, but InfluxDB was too slow, so we lost our patience and gave up. But you are welcome to follow the steps in the testing report, change the configuration, run the tests and get the result.

For the comparison test reports with other databases such as MongoDB, ClickHouse, OpenTSDB, etc. please check: TDengine Performance Comparison-en.pdf

What makes TDengine stand out among time-series databases? It is the novel design of the storage structure and novel concepts such as “one table per data collection point” and “super table”. For details, please check TDengine system design.