ECARX faced two problems in their secure SuperCloud: large amounts of data storage was being consumed and real-time queries of vehicle status had high latency. After due diligence and development, they ended up using TDengine as the foundation of their system. TDengine is responsible for the writing, storage and real-time queries of real-time vehicle data. This article describes the specific difficulties encountered by the R&D team in their use of Apache HBase and why they chose TDengine over OpenTSDB. The article also discusses the process they went through and the deployment of TDengine.
Project Introduction
ECARX has developed Cloud-based solutions which use AI and various sensor technologies such as cameras, radar, and sonar to ensure the safety of drivers and passengers. The ultimate aim is to develop autonomous driving capabilities. The core data that is collected in autonomous driving is the background telemetry data as well as data on the status of the equipment in vehicles. In order to achieve autonomous driving capability, there needs to be precise and robust data collection from the onboard equipment and this needs to be combined with data from high-resolution maps.
TDengine plays an important role in the ECARX SuperCloud data center with responsibility for the writing, storage, and instantaneous queries, of real-time vehicle data.
Database Selection
The prior architecture consisted of Kafka + Flink + HBase. As our business grew we found that the Key-Value storage model of HBase was not suitable for our use cases. For structured data, the Key-Value storage model leads to a particularly large disk footprint and high latency in real-time queries of the latest vehicle status. These were the two core problems that we needed to solve urgently.
We looked into various solutions and realized that what we needed was a time-series database (TSDB). The core data that we needed to store was time-series data. We looked at the various time series databases on the market and decided to take a closer look at InfluxDB, TDengine, and OpenTSDB.
Initially, we were leaning towards OpenTSDB because it is based on HBase and it would have been an easier deployment. However, we realized that OpenTSDB would not solve our performance and compression ratio problems since these are intrinsic to HBase. When we looked at InfluxDB, we realized that the open-source version only allows a single node and this would not give us the performance, high availability, and scalability that we needed in a production environment. Finally, after comparing various dimensions, we decided that TDengine was the optimal choice for us – an open-source database engine, with performance, high availability, and scalability built into the open-source version and with SQL support.
TDengine is, not coincidentally, highly aligned with our current business use-cases. We say “not coincidentally” because TDengine is designed specifically for IoT time-series big data. We were especially thrilled with the novel concept of the “super table” and it felt tailor-made for our use-cases. We assigned a subtable to each vehicle to receive the data generated by the IHU device. The IHU is the first-generation vehicle computing platform product released by ECARX in the second quarter of 2017. It is an infotainment device with various functions including navigation, multimedia entertainment, and vehicle information.
The new and optimized architecture consists of Kafka + Flink + TDengine. The data streaming into Flink consists of structured data stored in JSON and unstructured data such as pictures and videos. If the data is structured JSON data, it is written to TDengine as follows: Kafka—>Flink—>TDengine. If it is unstructured data, it is stored directly on S3, and then the file paths are written to TDengine as follows:
S3—>Kafka—> Flink—>TDengine.
Deployment and Results
We chose to deploy on a three node cluster in single-copy mode. Each node is configured with 8 Cores, 16 GB RAM and a 500G hard drive. There are 3 super tables and 276571 tables in the current environment.
The structure of the super table is as follows:
Since our JSON is rather large, we choose to use protobuf (Google’s open-source, language neutral data interchange format) to compress it and then write it to TDengine. This allows us to limit the JSON structured data to 1500 bytes. Data can be deserialized as needed from TDengine.
The current largest super table has stored more than 30 billion pieces of data with each row being 2362 bytes. Roughly estimated, the total data volume should be about 68T since the beginning of the project. But TDengine’s compression ratios allow the actual storage on disk to be only 1.4 TB. Typically we would generate and use 15 TB every 20 days but now we no longer have to worry about running out of storage and worrying about the cost of storage. The resource usage with TDengine is nearly a hundred times lower than it was with our prior architecture.
Additionally the high query latency issue that we had been experiencing will be solved when we upgrade to the newest release of TDengine. The “last” function of TDengine can return the latest status of a device in milliseconds. Since the version we are currently using is still relatively old 2.0.18, there is no cache for the last function in this version. The most commonly used SQL for querying the real-time location of a vehicle is as follows, all of which return results at the millisecond level:
Conclusion
As we have seen above, the major pain points that we were experiencing i.e. extremely high storage consumption and very high query latency for real-time vehicle status have been solved by TDengine and resource consumption is nearly 2 orders of magnitude lower. TDengine has helped us reduce costs while increasing efficiency and performance. In addition, TDengine will also form the foundation for our future projects which include equipment monitoring and statistics. In terms of our deployment of TDengine, we received very timely and comprehensive support from TDengine’s support staff which shortened our timelines. As ECARX grows and scales, we hope to partner with TDengine to develop an industry-standard model for autonomous driving.