TDengine TSDB in Shale Gas: From Data Silos to Unified Operations

In shale gas production operations, automated data collection was already in place across gas wells, stations, compressors, and LNG/CNG facilities. But earlier SIS, BPCS, SCADA, and third-party vendor systems had been built independently, making it difficult to create a unified data view. That fragmentation limited the command center’s ability to support production situational awareness, emergency response, and well drainage decisions.

To support visual production operations, fine-grained gas well management, and intelligent production monitoring, the project team built an intelligent shale gas management system based on a cloud-edge-device architecture and TDengine TSDB. The system gives the command center real-time status, historical trends, and anomaly information for key production objects such as wells, stations, and compressors, creating a single data foundation for dispatch, emergency response, and refined drainage control.

Multiple systems, no unified production view

Automated data collection was already broadly deployed across gas wells, stations, compressors, LNG/CNG facilities, and other equipment. Over time, however, the operation zone had built several SIS, BPCS, SCADA, and third-party vendor systems. Different system architectures, data standards, and maintenance owners made unified management and control difficult. The main problems were:

• No unified data view. Production, safety, equipment, and environmental data lived in separate systems. When the command center needed to check individual well status, station operations, equipment anomalies, or environmental monitoring data, staff had to switch between and compare information across systems, making it hard to form a unified picture quickly.
• Legacy real-time databases were expensive to scale and maintain. Earlier refined drainage operations relied on traditional real-time databases. As the number of connected devices, data points, and historical records grew, the pressure on system expansion, long-term storage, and ongoing maintenance increased steadily, making it harder to meet the demands of intelligent operations.
• Weak support for private cloud and cross-platform deployment. Existing systems had limited ability to deploy flexibly in private cloud and hybrid cloud environments, and offered poor compatibility with diverse CPU architectures and operating systems. This constrained platform expansion and long-term reliability.
• Complex system operations and maintenance coordination. Multiple systems were built and maintained by different vendors. The chain of interface integration, issue coordination, and fault diagnosis was long. On-site problems were hard to pinpoint quickly, and the complexity of unified operations kept growing.

Selecting a Time-Series Database for the digital upgrade

The database selection process focused on whether the system could match how shale gas production actually works: continuously ingesting high-frequency data from gas wells, stations, compressors, and other equipment, while letting the command center query historical trends quickly by individual well, station, equipment type, and time range. The system also needed to work in private cloud environments, support diverse infrastructure, and scale without creating new data silos.

After evaluation, TDengine TSDB was selected as the core Time-Series Database for the intelligent shale gas management system. TDengine TSDB is an open-source, high-performance, distributed time-series database. Its architecture fits this production scenario well: it supports real-time collection, storage, and monitoring for oil and gas production data and equipment status, while providing a unified data foundation for predictive alerts, refined drainage analysis, and data-driven decision-making through analytics and machine learning. This helps improve production management efficiency, reduce system construction and maintenance costs, and make production data more usable.

Key advantages include:

• Data organized by equipment. TDengine TSDB’s “one table per device” model, combined with Supertables and columnar storage, maps naturally to data organized by gas well, station, compressor, and other equipment types. This reduces cross-system data extraction and duplicate modeling.
• Reliable production system operation. Cluster deployment with multi-replica support meets the continuity requirements of production monitoring and centralized dispatch.
• Lower system integration and migration costs. Standard SQL reduces the effort of adapting the application layer. Compatibility with diverse operating systems and CPU platforms also supports cloud deployment and reliability requirements.

Cloud-edge-device: connecting the field to the command center

The intelligent production operations management system uses a three-tier cloud-edge-device architecture. Smart IoT handles unified data collection from field devices, edge aggregation, and cloud analysis, creating a complete data channel from the production site to command-center decisions.

The system runs on TDengine TSDB as its unified data foundation. Combined with low-code development and model-driven technology, it supports a production command center and technical decision center for visual production operations, fine-grained gas well management, and intelligent production monitoring.

Perception layer: complete data collection from the production site

At the perception layer, the project built a data collection system covering all key production objects, including gas wells, process equipment, station equipment, and environmental monitoring points. The design follows the principle of “one file per well, one data source per device.” Pressure, temperature, flow, liquid level, valve status, equipment operating status, and environmental monitoring data generated by field devices enter the time-series data pipeline through unified access, providing the data foundation for centralized monitoring, dispatch, and refined drainage analysis.

For data modeling, TDengine TSDB’s Supertable and Subtable capabilities were used to abstract similar equipment into unified data models, with individual wells and devices managed as specific data objects. This accommodates differences across equipment types and measurement point structures, and enables efficient querying and analysis by well, station, equipment type, and operation zone.

The perception layer covers the following data sources:

• Gas well monitoring: Wired pressure/temperature sensors, vortex flow meters, and static pressure level gauges collect real-time wellhead pressure, temperature, gas production, and downhole liquid level data. These feed gas well status assessment and refined drainage decisions.
• Process control equipment: Smart control valves, foam-drainage/gas-lift equipment, and related devices enable remote control and automated execution of well startup/shutdown, flow regulation, and production enhancement processes, supporting refined production management.
• Station and supporting equipment: PLC control systems, RTU remote terminals, and compressor monitoring systems handle process logic control at gathering stations, data collection and transmission from remote well sites, and status monitoring of key power equipment, ensuring stable station operations.
• Environmental monitoring: Data from groundwater and other environmental monitoring points provides continuous visibility into relevant environmental indicators for compliance management.

Edge layer: reliable data aggregation from stations to the center

At the edge layer, HUB collectors deployed at each gathering station and operation zone handle unified access, protocol conversion, cleaning, compression, and edge computing for data from gas wells, stations, compressors, and industrial control equipment. The processed data is written to TDengine TSDB. Once ingested, the data is made available through queries and subscriptions to the IoT business sharing center, IoT data sharing center, and upper-layer applications.

The edge layer covers the following capabilities:

• HUB collectors: Deployed at gathering stations and operation zones, supporting Modbus, OPC UA, IEC-104, and other industrial protocols. They handle unified collection, protocol conversion, cleaning, compression, and edge computing, with store-and-forward capability to ensure data integrity during network interruptions.
• Data ingestion: Real-time streaming data is ingested via MQTT, while traditional industrial control equipment is connected through Modbus TCP and similar protocols. In continuous collection scenarios, TDengine TSDB can support writes at millions of data points per second.
• Data storage: Using the “one table per device” model, each well and each piece of equipment has its own time-series table. Supertables (STables) unify the data structure and tag information for equipment of the same type. Raw data is stored at second-level granularity, with support for more than 10 years of historical data retention.
• Data processing: TDengine TSDB’s Stream Processing, time-window aggregation, and downsampling capabilities compute key metrics such as gas production rates and equipment efficiency in real time, delivering ready-to-use data for command-center dashboards, report analysis, and anomaly assessment.

Unified data modeling with Supertables

For the data model design, TDengine TSDB’s Supertable capability was used to unify the data structure of similar equipment types. Field types are standardized as TIMESTAMP, INT, BOOLEAN, DOUBLE, FLOAT, and JSON based on the characteristics of field-collected data. This reduces duplicate modeling effort when onboarding different equipment and simplifies query and analysis by equipment, measurement point, and time range.

CREATE STABLE IF NOT EXISTS ts_kv(
  ts TIMESTAMP,
  bool_v bool,
  str_v varchar(4000),
  long_v bigint,
  dbl_v DOUBLE,
  json_v varchar(8000)
) TAGS (key int, entity_id varchar(36));

Multi-well, multi-metric queries in under 100ms

In practice, the command center frequently needs to view historical changes across multiple wells and multiple metrics at once. In one query spanning 3 devices, 6 metrics, 15 days of historical data, and a 10-minute data interval, covering instantaneous gas flow, valve position, pressure, and similar indicators, TDengine TSDB returned results within 100ms. Compared with the previous system, field personnel no longer wait for query results when reviewing trends, comparing operating conditions, or supporting dispatch decisions.

Upgrades and optimizations during stable operation

Version upgrade: from 3.1 to 3.3.6.35

During the project, the TDengine support team helped upgrade TDengine TSDB and optimize the connection approach, further improving system stability. The main upgrades included:

1. Upgrading TDengine TSDB from version 3.1 to enterprise version 3.3.6.35, improving system stability;
2. Upgrading the taos-jdbc driver to 3.7.8, improving driver compatibility and connectivity;
3. Switching the connection method to WebSocket to further improve query performance.

Operations tuning: query performance, slow SQL, and memory optimization

During project construction and operation, several operations and performance tuning practices were adopted based on TDengine TSDB’s characteristics.

1. Using the cache for faster latest-data queriesWhen retrieving the latest data for each device, the system uses TDengine TSDB’s built-in cache for queries, which improves response speed.select last_row(*) from ts_kv group by tbname;
2. Investigating slow SQL through system tablesFor daily operations, performance_schema.perf_queries is used to identify long-running SQL statements and pinpoint queries that affect system performance. For example:# Query the top 10 longest-running SQL statements exceeding 10 seconds select sql,exec_usec/1000/1000,`user` from performance_schema.perf_queries where (sql like '%select %' or sql like '%SELECT%' )and exec_usec >10000000 order by exec_usec desc limit 10;
3. Optimizing taosd and taosAdapter memory usageTDengine TSDB provides a memory allocator configuration script, set_taos_malloc.sh, for tuning memory allocation strategy. This script lets users switch between different memory allocator modes to improve TDengine TSDB performance and stability.If taosd or taosAdapter memory usage is high, use this script to switch memory allocation to tcmalloc, jemalloc, or other modes. For general optimization, modes 1 and 3 are recommended as a starting point.-m <mode> Specify the memory allocator mode. 0: glibc default allocator 1: tcmalloc optimization 2: tcmalloc with custom checks, for memory leak detection 3: jemalloc optimization 4: jemalloc with custom checks, for memory leak detection
4. Professional services to support project deliveryDuring version upgrades, connection adjustments, and troubleshooting, the TDengine support team responded quickly and helped identify issues, ensuring that key milestones in project launch and ongoing operation were met.

Next steps: extending the data foundation with TDengine Historian

TDengine Historian extends the time-series data foundation into a broader industrial data management layer. On top of high-frequency production data stored in TDengine TSDB, TDengine Historian can organize industrial field data such as sensors and equipment through a tree hierarchy, build a unified Data Catalog with contextualization and standardization, and provide real-time analysis, visualization, event management, and alerting capabilities.

The next phase is to introduce TDengine Historian to build on the high-frequency time-series data already accumulated in TDengine TSDB and continue extracting value from shale gas production data. With capabilities such as intelligent scenario awareness, automated dashboard and report generation, anomaly detection, and alerting, TDengine Historian can help improve system-level operational insight and move shale gas development toward more intelligent, fine-grained, and efficient operations.