Building an Industrial Innovation Base with Historian Connector + TDengine

In the field of industrial automation, data collection, storage, and analysis are fundamental to efficient decision-making and operations. AVEVA Historian (formerly Wonderware Historian), a leading industrial real-time database, focuses on collecting and storing high-fidelity historical process data. Meanwhile, TDengine, a high-performance database specifically designed for time-series data, excels in processing and analyzing large volumes of real-time and historical time-series data.

In the latest TDengine version 3.2.3.0, we have further enhanced the data integration capabilities by introducing a new connector aimed at integrating AVEVA Historian with TDengine. This article will explain the innovative approach of integrating historical and real-time data from AVEVA Historian into TDengine, and how this combination can significantly improve data utilization, creating a solid foundation for industrial innovation and promoting efficient development in industrial automation.

Current Industrial Time-Series Data Processing Solutions:

1. Relational Databases (e.g., Oracle/MySQL)
   • Low read/write performance when handling large volumes of time-series data.
   • Poor distributed support; query speed slows down as data increases.
   • Typically used in low-frequency monitoring scenarios and power SCADA historical databases.

2. Traditional Industrial Real-Time Databases (e.g., PI System, AVEVA Historian)
   • Effective in industrial data storage but outdated architecture with no distributed solutions; cannot scale horizontally.
   • Dependent on environments like Windows; weak in data analysis and often closed systems.
   • Mainly used in SCADA systems and production monitoring systems.

3. NoSQL Databases (e.g., MongoDB, Cassandra)
   • Poor real-time computation, slow query speed, high memory and CPU overhead, and lack of optimization for time-series data.
   • Mainly used for unstructured data storage and web scraping data.

4. Hadoop Big Data Platforms (e.g., HBase, Zookeeper, Redis, Flink/Spark)
   • Support distributed processing but handle time-series data in an unstructured way with a complex architecture and high maintenance costs.
   • Primarily used for general big data platforms and public opinion e-commerce big data.

In today’s industrial enterprises, there is a strong demand for innovative applications in critical scenarios like intelligent decision support, equipment fault prediction, product quality analysis and forecasting, intelligent manufacturing and digital twins, and energy management. These scenarios highlight the core value of industrial time-series big data in innovation. Industrial enterprises must adopt advanced solutions to accelerate innovation and achieve differentiation in a competitive market.

In industrial settings, multiple factories or workshops typically deploy independent SCADA/Historian systems, such as AVEVA Wonderware and PI systems, to manage real-time and historical data. This is a common practice in industrial automation, with data distributed across various on-site databases.

Advantages of Centralizing Time-Series Data:

Centralizing time-series data enhances overall data control, enabling enterprises to better utilize their data assets. Centralized data processing allows for global data visualization, inspiring business innovation and providing a means for quick validation. It helps enterprises better manage and analyze data, swiftly respond to insights, and make timely business decisions.

SCADA/Historian systems also offer data centralization solutions, which can meet some needs. However, they face challenges in supporting massive amounts of data points (sensors, devices, etc.), struggling to meet the demands of innovative applications that require extensive time-series data acquisition. When the data volume is very large, SCADA/Historian data consumption interfaces may perform poorly, experiencing high latency and even failing to retrieve the necessary datasets.

Addressing These Challenges:

To effectively address these challenges, the following optimizations are necessary:

1. Utilize Existing Investments: Enterprises often already have multiple SCADA/Historian systems in place, representing significant investment. The solution must consider how to fully leverage existing systems to avoid redundant investments.
2. Enhance Data Consumption Interfaces: Improve the capability of data interfaces to ensure fast, efficient data consumption even when dealing with large volumes.
3. Reduce Latency: Enhance system performance to minimize data processing and retrieval latency, ensuring timely data acquisition.
4. Integrate Real-Time and Historical Data: Integration can improve storage space utilization and provide a more complete data view for analysis and decision-making. Combining real-time and historical data supports more complex innovative applications like predictive maintenance, energy management, and operational optimization.
5. Support Massive Data Points: Increase the system’s capacity to handle numerous data points to accommodate the growing demand for sensor data.
6. Promote Innovation: Build an infrastructure that supports innovative applications, addressing the needs of new industrial applications like predictive maintenance, asset performance management, and energy efficiency optimization. This requires deeper data consumption, analysis, and faster processing.

TDengine, as a symplified time-series big data platform, offers high data writing and query performance, suitable for handling vast amounts of industrial time-series data. In addition to its core functionality of unifying real-time and historical time-series databases, it also provides message queues, event-driven stream processing, read-write caching, and multi-data source integration capabilities. TDengine provides a comprehensive solution for implementing the above optimizations.

Integrating AVEVA Historian Data into TDengine

In this solution, we utilize TDengine Enterprise taosX’s multi-data source integration capability to aggregate data from multiple AVEVA Historian systems into a central TDengine cluster.

Features:

1. Data Migration: Transfer existing historical data from AVEVA Historian to TDengine.
2. Data Synchronization: Support real-time and historical data synchronization, integrating AVEVA Historian’s real-time (Runtime.dbo.Live) and historical data (Runtime.dbo.History) into TDengine.
3. Support for Massive Measurement Points: TDengine supports 1 billion timelines, easily handling current industrial big data scenarios.
4. Maximize Existing Investments: Existing AVEVA Historian systems will continue to function while supporting innovative business development on the TDengine platform.
5. Optimized Time-Series Data Storage: Utilize TDengine’s efficient data compression and storage mechanisms to optimize data storage on the new platform.
6. Significant Query Performance Improvement: Compared to Historian, TDengine significantly enhances query performance in both projection and aggregation queries.
7. Support for Data Subscription: TDengine provides a structured message queue, allowing real-time consumption of data to drive innovative applications.
8. Support for Multiple Deployment Environments: Linux & Windows.
9. Complete ETL Features: The taosX component supports complete parsing, extraction, filtering, and data mapping, allowing zero-code external data source integration into TDengine.

Basic Environment Requirements:

• AVEVA Historian integration requires TDengine Enterprise support.
• Remote collection through agent mode requires deploying taosX Agent at the collection site.
• Supports AVEVA Historian 2017 and later versions.

Below is an example of migrating historical view data from AVEVA Historian to TDengine.

Migration Steps:

1. Preparation: Create a database and a supertable in TDengine.
2. Login to taosExplorer: Create a data writing task of type AVEVA Historian.
3. Fill in Basic Task Information: Task name, target database, Historian server address, port, authentication information.
4. Configure Collection Settings: migrate mode, select tag points range (default all points), set source data start and end time, query time window span.
5. Data Parsing and Filtering: No additional parser and filter configuration needed as Historian sends structured data.
6. Data Mapping: After selecting the target supertable, the system will automatically match some fields. Manually specify the mapping relationship for unmatched fields.
7. Start Task.

Data Synchronization:

Similar to data migration but with the following differences:

• Supports two sources: Runtime.dbo.History & Runtime.dbo.Live.
• Task end time can be left unset, meaning it can keep synchronizing until manually stopped.

Combining with OPC Solutions:

• Historical data imported into TDengine via Historian.
• Real-time data collected into TDengine via OPC, leveraging OPC’s subscription feature for immediate data collection.

This solution, relying on the taosX component as well, is easy to maintain.

Conclusion

For industrial enterprises that have invested in AVEVA Wonderware, the Historian Connector combined with TDengine offers a robust solution for digital transformation challenges. This solution not only facilitates overcoming digital transformation hurdles but also leverages the power of business innovation to unlock digital potential.

Moreover, this solution boasts excellent integration capabilities, seamlessly combining with various data collection solutions. For example, integrating with OPC Connector allows historical data to be smoothly imported from Historian to TDengine, while real-time data is collected via OPC, enabling efficient centralized processing of field time-series data.

The solution is based on the powerful features of TDengine Enterprise (with plans to launch a TDengine Cloud version in the future). If your enterprise is seeking such a solution, feel free to contact TDengine’s professional business team to explore advanced data processing methodologies.

About AVEVA Wonderware

Founded in 1987 and headquartered in Los Angeles, Wonderware is a renowned brand in the global industrial automation field. Acquired by Schneider Electric and later integrated into AVEVA, Wonderware has extensive applications across various industries, including tobacco production, water treatment, power, oil and gas, chemicals, steel metallurgy, food and beverages, pharmaceuticals, automotive manufacturing, and logistics warehousing. Its products include, but are not limited to, In Touch HMI, System Platform, and Historian.