To ensure safety and reliability in coal-mine production scenarios, Shandong Energy Group built an integrated safety-production management and control platform. This data-lake project spans 75 mines and more than 1,100 industrial systems across 23 categories, aiming to achieve real-time data acquisition, comprehensive monitoring, in-depth analysis, and practical application. By unlocking the value of data, the platform seeks to improve both the safety and efficiency of coal-mine production processes. After conducting system research and comparing available products, Shandong Energy decided to select an appropriate solution in the field of time-series databases to ensure efficient storage, fast querying, and high reliability.
During the evaluation process, Shandong Energy found that many single-node time-series databases on the market suffer from several issues, including:
- A failure in the database node can render the entire system unavailable, causing losses to the company and its customers.
- Server storage failures can lead to data loss, potentially resulting in irreversible asset damage.
- Limited scalability. As business grows, database performance requirements increase, and a flexible scaling mechanism is essential for improving delivery efficiency.
Based on these factors, to achieve stable and efficient data processing, the platform’s technical team emphasized that the chosen time-series database must support a cluster architecture, multi-replica data storage, high compression ratios, and flexible expansion. TDengine meets all of these requirements. Its distributed cluster architecture and multi-replica mechanism ensure high data reliability and fault tolerance, while its flexible horizontal scaling enables the system to easily handle growing data volumes and evolving business demands. Together, these capabilities provide the integrated management platform with stable and efficient data-processing performance.
Evolution of the Data Architecture
Throughout the project, Shandong Energy encountered a wide range of data-integration scenarios. Because time-series data integration requires a high degree of timeliness and stability, the efficiency of the integration solution became one of the key factors in selecting a time-series database. Previously, the company experimented with the following architecture:
However, this architecture faced many issues, including:
- High server resource consumption and elevated deployment costs.
- Poor timeliness of time-series data, with noticeable latency that made it unsuitable for scenarios requiring rapid data processing.
- Low overall system availability and high maintenance complexity, as the long data pipeline involved many nodes, each requiring stable operation.
Later, by leveraging TDengine’s built-in data subscription capabilities, the project was able to significantly reduce the use of middleware, improve overall system availability, and further enhance data distribution efficiency.
In practical application, the production units’ acquisition systems first collect data into their local TDengine databases. The IoT platform then synchronizes the data from the local TDengine instances to the group-level TDengine cluster. After being processed by the group’s data-governance system, the industrial data is made available to upper-layer applications as a reliable data service.
This project successfully enabled the acquisition, storage, and analytical use of industrial data in the coal sector, providing dependable data support for production processes and further improving both operational efficiency and management capability.
Issues Encountered During Implementation and Their Solutions
In Shandong Energy’s application-integration scenario, TDengine was initially used solely as a database for storing and querying time-series data. As the business expanded, the need for alarm-related functionality emerged. To meet this requirement, the project team initially built a separate alarm service. However, because certain messaging middleware could not persist data, the alarm service had to remain online continuously. This not only introduced the risk of losing alarms but also made upgrades and maintenance more complicated.
Later, the team decided to leverage the stream-processing capabilities available in the newer version of TDengine, shifting part of the alarm logic into stream processing. This improvement simplified the architecture, enabling some alarms to be generated directly at the database level and greatly reducing the development burden on the service layer.
The deployment and implementation of TDengine proceeded relatively smoothly, with only two key issues encountered. Thanks to the close collaboration between Shandong Energy’s project team and TDengine’s technical experts, both issues were successfully resolved.
Data writing issue: According to the technical specifications of the integrated management and control platform, high write performance is critical for the time-series database. In the early stages, Shandong Energy used TDengine in a real-time write mode—writing each data point to TDengine as soon as it arrived. They found that the write performance did not meet expectations. After consulting with TDengine’s technical experts, they switched to a batch-write mode, grouping multiple data points into a single large insert statement before writing. This change significantly improved write performance and met the platform’s requirements.
During the process of integrating TDengine into the overall data architecture and application stack, the project encountered an issue where the TDengine client was unable to connect after a version upgrade. TDengine’s technical experts recommended switching to the new JDBC-RESTful connection method. After testing, this approach successfully resolved the issue. While meeting performance requirements, the service layer also became more resilient to client-side connection changes caused by database version upgrades, further improving stability in the production environment.
Experience Sharing on Data Migration
During the data-architecture upgrade, data migration became a critical step. Ensuring data security and integrity through careful planning and optimized migration strategies is essential for a smooth transition. In the first phase of the integrated management platform’s data-lake project, data migration progressed in an orderly manner to fully leverage the features of the new version of TDengine. To ensure that the old version of TDengine could continue providing services normally while maintaining data consistency between the old and new databases, the project adopted the following steps:
A data-writing service was created to support dual-write operations, allowing data to be written simultaneously to both the old and new versions of the TDengine database.
Retrieve a segment of historical time-series data from both the new and old database versions, then validate and process the data before comparing the results. After multiple rounds of sampling and verification, the results consistently showed that data consistency could be ensured.
Enable the data tasks that write to the new version of the database, monitor their execution, and check whether the application-layer data remains consistent. After running for a period of time and confirming consistency, stop the data tasks for the old version, then stop the dual-write process, and finally shut down the old TDengine instance.
Following these steps, the project successfully completed the data switchover while ensuring uninterrupted operation of the live system. In the first phase of the integrated management platform project, TDengine ran smoothly, with a daily data increase of 250 million records. The product’s stability and performance fully met the expected design targets. Additional systems built on TDengine—such as the hierarchical industrial data early-warning system—were also successfully implemented.
Today, Shandong Energy uses TDengine as the core time-series data storage engine for its IoT platform, building a cloud-coordinated architecture for time-series data storage and transmission. This setup meets the time-series data needs of individual production units while also supporting group-level analytical applications. TDengine instances are deployed both at the production-unit level and at the group level, ensuring that each side’s requirements are met with optimal performance.
Conclusion and Outlook
Looking ahead, Shandong Energy plans to continue validating and exploring the use of TDengine in production-management and centralized-monitoring systems across industries such as power and chemicals. They also intend to deepen communication and collaboration with TDengine’s technical experts as new application scenarios emerge.
Reflecting on the project, Shao Guopeng, Assistant Director, Product Development Center, Industrial Internet Division, noted: “Throughout the use and integration of TDengine, the technical experts from TDengine provided timely solutions to many issues through various communication channels. Their support greatly contributed to the progress of the project. In the future, we hope to engage more deeply with real customer scenarios and bring together customers with similar needs for joint discussions. By sharing challenges and best practices from real-world implementations, we can further co-create a business ecosystem around time-series databases.”
Li Zhijun, Director, Data Team, Industrial Internet Division, also shared his expectations for TDengine’s future development: “To expand application scenarios in the AI domain, TDengine can further enhance its integration with AI technologies and strengthen capabilities in data governance, data-warehouse construction, data modeling, and data services, thereby unlocking more value from time-series data. At the same time, improving TDengine’s openness is equally important—this includes more open data formats, diversified integration models, flexible migration methods, and efficient data-distribution mechanisms. This will give customers more options tailored to their needs and allow TDengine to better integrate with existing system architectures. In terms of system stability, TDengine can draw on the experience of well-known international industrial databases to further reinforce its robustness and reliability, enabling it to better meet the demands of industrial environments.”
About Shandong Energy Group
Shandong Energy Group is one of the world’s largest coal and energy companies, with annual operating revenue of approximately USD 122 billion and a top-100 position in the Fortune Global 500. The group operates an extensive portfolio of coal mines, power generation assets, coal-chemical plants, new energy materials projects, and logistics operations across multiple regions in China. It is recognized as the third-largest coal producer in the country and a leader in “intelligent mining,” with nine national-level intelligent mine demonstration sites and more than 100 intelligent working faces in operation. This combination of massive production scale and aggressive digitalization makes Shandong Energy a flagship reference for large-scale industrial data and automation projects.
