The full power of TDengine, now free forever for up to 5,000 tags.

Explore More

From Data Archive to TSDB: Why the Industrial Data Foundation Must Be Rebuilt

In the previous article, we discussed the evolution of industrial data infrastructure: from data historians, to industrial data platforms, and now to AI-native industrial data foundations. This evolution is not just about adding new capabilities—it is fundamentally about rethinking the underlying architecture. At the center of this shift lies a critical but often overlooked change: the data storage layer itself is being replaced.

For decades, the Data Archive has been the core component of industrial data historians. It is responsible for ingesting time-series data from the field, compressing it, and making it available for query and visualization. This architecture has been remarkably stable and has supported industrial operations successfully for many years.

However, when viewed through the lens of industrial internet, IoT, and AI, the assumptions behind Data Archive no longer hold.

The Strengths and Limitations of Data Archive

Data Archive was designed in an era where both compute and storage were scarce. Its primary objective was to store as much data as possible using as little resource as possible. To achieve this, it adopted compression techniques such as the swinging door algorithm, which reduces data volume by discarding redundant points while preserving the overall shape of the signal.

Diagram illustrating how swinging-door compression reduces the number of time-series data points retained in a historian archive.
A diagram to explain how the swinging door compression algorithm works

This approach was highly effective at the time. It allowed systems to retain long periods of historical data within limited storage constraints, while still enabling engineers to understand trends and system behavior. This efficiency was one of the key reasons why data historians became widely adopted across industries.

But from today’s perspective, this design comes with clear trade-offs. First, compression techniques that discard data points, such as the swinging door algorithm, are inherently lossy. Readings filtered out before archival cannot be recovered for later analysis. While this may be acceptable for trend visualization, it becomes a major limitation in the AI era. Machine learning models, anomaly detection, and fine-grained analysis all depend on high-fidelity data. Losing original data points can directly impact accuracy and lead to incorrect conclusions.

Second, Data Archive does not follow an open query model. It typically does not support standard SQL, and instead relies on proprietary APIs or tools for data access. While this may work within a closed system, it becomes a significant barrier when integrating with external systems such as BI tools, modern data platforms, or AI pipelines. Additional middleware or custom integration is often required, increasing both complexity and cost.

At the architectural level, Data Archive was not designed for horizontal scalability. It is typically based on single-node or limited-scale deployment models. In an era where industrial internet and IoT are driving exponential growth in the number of tags and data points, this limitation becomes increasingly difficult to manage.

Overall, Data Archive solved the problem of efficient data storage extremely well—but it did so under the assumptions of a resource-constrained, closed-system environment. In today’s world of open, scalable, and AI-driven data systems, those same design choices are becoming constraints.

TSDB: Built for Scale, Openness, and Modern Infrastructure

Modern time-series databases (TSDBs) are built in the era of cloud computing and big data. Their goal is not only to store data efficiently, but to support large-scale processing, open integration, and future-facing data applications.

At the storage level, TSDBs typically use columnar storage combined with multi-stage compression techniques. This allows them to achieve higher compression ratios while preserving raw data. TDengine TSDB uses unique “one table per data collection point” data model to achieve even higher compression ratio. This is especially important in the AI era, where model training, anomaly detection, and behavioral analysis all require complete and high-fidelity datasets.

Diagram to explain how column storage works

In terms of query capability, TSDBs commonly support standard SQL. This is not just a matter of convenience—it fundamentally changes how industrial data can be used. Engineers can use familiar query languages, and data can be directly integrated with BI tools, visualization platforms, and modern data systems without relying on proprietary interfaces. For the first time, industrial data becomes truly interoperable within a broader data ecosystem.

Architecturally, TSDBs are designed to scale horizontally. They can distribute data across multiple nodes and handle increasing data volumes by adding more capacity. This is essential in industrial environments where the number of devices and data points is growing exponentially.

TDengine distributed architecture which can support one billion tags

Even more importantly, TSDBs align with modern infrastructure. They run natively on Linux, support containerized deployment, and can operate seamlessly in cloud environments. This is in sharp contrast to traditional historian systems that often depend on Windows-based or proprietary environments. In today’s cloud-native world, the ability to deploy, scale, and manage systems through automation is no longer optional—it is a baseline requirement.

TSDBs also provide open interfaces and integrate easily with data pipelines, messaging systems, and analytics tools. They are no longer isolated storage engines, but active components within modern data infrastructure.

From this perspective, TSDB is not just a better Data Archive – it is a fundamentally different class of system.

Data Archive was born in an era of resource constraints. TSDB was born in an era of data explosion.

TSDB Alone Is Not Enough: Toward TSDB + IDMP

A modern TSDB is a major step forward from a traditional data archive. TDengine TSDB provides scalable time-series storage, high-performance queries, and an open foundation for modern industrial data applications. Yet even the most powerful database cannot solve the full problem of industrial data management.

The value of industrial data comes from its context. A temperature, pressure, or vibration reading only becomes meaningful when it is connected to a specific asset, process, or operating condition. Without that context, engineers still need to spend significant effort interpreting the data, and AI systems struggle to understand what the measurements actually represent.

This is where TDengine IDMP takes industrial data management to the next level. As an AI-native industrial data platform, IDMP organizes data around assets and events, connects measurements with operational context, and brings visualization, advanced analytics, and AI capabilities into a unified environment. Instead of simply storing and querying data, organizations can use IDMP to understand operational behavior, investigate problems, and generate actionable insights.

TDengine TSDB remains a powerful data engine and a natural foundation for IDMP. However, IDMP is not limited to data stored in TDengine TSDB. It can also connect to existing databases such as MySQL, PostgreSQL, and InfluxDB 3, bringing data from different systems into a common industrial context without requiring every dataset to be migrated into TDengine TSDB first.

This creates a more flexible and open industrial data foundation. Organizations can benefit from the performance and scalability of TDengine TSDB while using IDMP to build asset models, understand relationships, analyze operational events, and make industrial data accessible to AI.

From this perspective, TSDB provides the data foundation, but TDengine IDMP unlocks its full value. Together, they move industrial data management beyond storage and toward an AI-native platform built for operational understanding and intelligence.

Conclusion

Data Archive played a critical role in the development of industrial systems, but it was designed for a different era—one defined by limited resources and closed architectures.

Today, industrial internet, IoT, and AI are fundamentally changing how data is generated, processed, and used. Data volume is growing exponentially, and expectations for data are shifting from storage to intelligence.

In this context, TSDB is replacing Data Archive as the new data engine. But the real transformation comes with TDengine IDMP, which brings together high-performance time-series data, industrial context, advanced analytics, and AI, enabling a shift from simply storing data to understanding and acting on it.

This is not just a technical upgrade — it is a generational shift in industrial data infrastructure.

  • Jeff Tao
    Jeff Tao

    With over three decades of hands-on experience in software development, Jeff has had the privilege of spearheading numerous ventures and initiatives in the tech realm. His passion for open source, technology, and innovation has been the driving force behind his journey.

    As one of the core developers of TDengine, he is deeply committed to pushing the boundaries of time series data platforms. His mission is crystal clear: to architect a high performance, scalable solution in this space and make it accessible, valuable and affordable for everyone, from individual developers and startups to industry giants.