LOS GATOS, Calif., Dec. 14, 2021 — In order to construct a more complete Internet-of-Things big data ecology, and to support the migration from other time-series data platforms to TDengine in a simple and high efficient manner, TAOS Data released taosAdapter.
TDengine is a big data platform specially designed and optimized by TAOS Data for the Internet of Things, Internet of Vehicles, Industrial Internet, IT Operation and Maintenance, etc. In addition to the core time-series database functions that are more than 10 times faster, it also provides functions such as caching, data subscription, and streaming computing to minimize the complexity of R&D and Ops. The core code, including cluster functions, is all open-source.
Since TDengine announced open-source in July 2019, it has received very positive feedback on GitHub, the world’s largest code hosting platform. More than 17,400 people starred and more than 4,100 people forked code. There are an increasing amount of users are beginning to adopt TDengine.
taosAdapter is a TDengine’s companion tool and is a bridge/adapter between TDengine cluster and application. It provides an easy-to-use and efficient way to ingest data from data collecting agents (like Telegraf, StatsD, collectd) directly. It also provides InfluxDB/OpenTSDB compatible data ingestion interface to allow InfluxDB/OpenTSDB applications to immigrate to TDengine seamlessly. It is part of the TDengine server from TDengine v2.3.0.0.
The core starting point of taosAdapter is to solve the pain points of users and reduce migration costs.
In fields such as the Internet-of-Things and operation and maintenance monitoring, some users are still using traditional solutions or older products to solve time-series data processing problems, such as OpenTSDB. OpenTSDB is also an open-source distributed time-series database. It does not have its own storage engine and its related functions are completely based on HBase. Because of the early generation time, many operation monitoring projects chose this system.
Taking SF Technology as an example, they adopted OpenTSDB+HBase as the storage solution for the full monitoring data of the big data monitoring platform. However, as the amount of data accessed by the platform becomes larger and larger, they have encountered many pain points, such as heavy system dependence, high cost, and unsatisfactory performance.
In particular:
- OpenTSDB has heavy dependence and poor stability: The big data monitoring platform is the underlying infrastructure, and it also relies on big data components such as Kafka, Spark, and HBase in terms of data storage. Thus, data processing will take very long, and the longer the data link, the greater the challenge to ensure the reliability of the system. If there is a problem with the monitoring system itself, there will be no way to find and locate the problem with the business system based on it.
- High cost of use: The amount of written monitoring data is very large, and to trace historical problems, they need to store the full amount of monitoring data for more than half a year. Data storage costs remain high.
- Performance cannot meet requirements: OpenTSDB, as a full monitoring data storage solution, basically meets the requirements in terms of writing performance, but cannot meet the requirements in terms of daily large-span and high-frequency queries.
To solve these pain points, the engineers of SF Technology at that time believed that it was necessary to upgrade the full monitoring data storage solution. They investigated several time-series database products and eventually decided to choose TDengine. Afterward, they modified the system based on TDengine. After the transformation is completed, the TDengine cluster can easily handle the full monitoring and data writing, and it is currently operating stably.
The improvement brought by this transformation is very eye-catching: the number of physical servers has been reduced from 21 to 3, and the daily storage space required is only about 1/10 of OpenTSDB+HBase under the same conditions, which greatly reduces the hardware cost. In terms of query performance, in the case of using the pre-calculated function, the query p99 is within 0.7 seconds, which has been able to meet most of the daily query requirements. In the case of a large-span (6 months) non-pre-calculated query, the first query spent about 10 seconds, the time consumption of subsequent similar queries will be greatly reduced (2-3s).
RisingStar IoT platform has also encountered similar problems. They used OpenTSDB to store time-series data before, which can meet the needs in function; but due to the complexity of OpenTSDB architecture and excessive size, it has brought troubles to development and testing, installation and deployment, and operation and maintenance management. These problems have become more serious with the growing development of the business scale. After the same investigation, they also chose TDengine. In the process of upgrading, they need to retain historical data, so they need to migrate historical data from OpenTSDB to TDengine. To this end, they also specially developed a data migration tool and conducted detailed tests.
The needs of users are the driving force for product evolution. TDengine’s R&D team began to think about this question: Since many users have this kind of migration needs, can they officially give a unified solution? taosAdapter is their answer. taosAdapter mainly has the following functions:
- RESTful interface
- Compatible with InfluxDB v1 write interface
- Compatible with OpenTSDB JSON and Telnet format writing
- Seamlessly connect Telegraf, collectd, StatsD, Icinga, TCollector
It is compatible with OpenTSDB’s Telnet/JSON writing protocol. For operation and maintenance monitoring services, users can directly push data collected by collectd and StatsD to TDengine through taosAdapter. After the data can be normally written to TDengine, Grafana can be adjusted and adapted to visualize the data written to TDengine. TDengine also provides connection plug-ins for Grafana. If you want to migrate historical data, TAOS Data has also developed a data synchronization tool DataX plug-in, which can help users automatically write data into TDengine. Users do not need to modify a single line of code, they only need to modify a few configurations for seamless migration.
You may explore more about taosAdapter according the document.
In the future, taosAdapter will continue to improve and support the migration from more platforms to TDengine.
About TDengine
TDengine is a popular open-source data platform purpose-built for time-series data. With over 17,000 stars on GitHub and hundreds of new installations daily, TDengine is used in more than 50 countries worldwide. The company is headquartered in Los Gatos, CA and has raised $70M in venture capital.
Press Contact
For press or business inquiries, contact us.
