Industrial data infrastructure is undergoing a major transformation.
For decades, data historians have been the backbone of industrial operations. They solved one of the hardest problems in industrial computing: collecting, storing, and accessing massive volumes of time-series data from machines and control systems. Systems like PI System became essential infrastructure in factories, power plants, and refineries around the world.
But the world around industrial data has changed. Modern IT architecture, cloud computing, and now artificial intelligence are redefining how organizations expect to use operational data. The question is no longer just how to store data, but how to turn that data into insights, intelligence, and decisions.
At the same time, another challenge has become increasingly clear: many traditional historian systems were built as relatively closed ecosystems, making it difficult to integrate industrial data into modern IT infrastructures.
To understand where industrial data infrastructure is going, it helps to first understand where it came from.
1. The Birth of Data Historians
Industrial data historians emerged in the late 1980s and early 1990s as industrial automation systems began generating massive volumes of operational data.
Sensors, PLCs, and SCADA systems continuously produce time-series signals such as temperatures, pressures, flows, and machine states.
Traditional relational databases were not designed to handle this kind of workload:
- high-frequency time-series data
- continuous streaming updates
- millions of data points
- long-term historical storage
Data historians were created specifically to solve this challenge. They provided specialized storage engines optimized for time-series data ingestion, compression, and retrieval.
For the first time, industrial organizations could store years of operational history and use that data to troubleshoot problems, analyze performance, and improve operations.
2. What Data Historians Did Exceptionally Well
Data historians quickly became the central data infrastructure for industrial environments because they solved several critical problems extremely well.
Reliable Time-Series Storage: Historians could ingest large volumes of streaming data while efficiently compressing and storing it for long periods.
Integration with Industrial Systems: They connect directly to SCADA systems, PLCs, and industrial protocols, making it easy to capture operational data.
Long-Term Operational Visibility: Engineers could access months or years of historical trends to investigate incidents and understand system behavior.
Operational Monitoring: Operators could visualize trends, alarms, and system conditions through dashboards and trend charts.
For decades, this capability made data historians one of the most valuable pieces of infrastructure in industrial operations.
However, these systems were primarily designed as self-contained operational systems, rather than as open data platforms integrated with the broader IT ecosystem.
3. The Typical Architecture of a Data Historian
A typical industrial data historian follows a layered architecture. Using PI System as an example, several core components are commonly found in historian systems.
Data Archive
At the core of the system is the Data Archive, which stores time-series data collected from industrial equipment.
Its responsibilities include:
- high-throughput data ingestion
- time-series compression
- long-term historical storage
- efficient query and retrieval
This component solves the fundamental challenge of reliably storing massive volumes of operational data.
Data Collection Interfaces
Data historians rely on a set of interfaces to connect with industrial devices and systems.
These interfaces collect data from sources such as:
- OPC / OPC-UA servers
- PLCs and controllers
- SCADA systems
- industrial communication protocols
The interfaces continuously stream data from plant floor systems into the historian.
However, these interfaces are often vendor-specific or proprietary, making integration with other data systems more complex than it should be.
Asset Framework
One of the most important innovations introduced by modern historians is the Asset Framework (AF).
Instead of viewing data as a flat list of signals, AF organizes signals around industrial assets and equipment.
For example:
This asset-centric structure makes operational data much easier for engineers to understand.
Analytics and Event Detection
Historians often include analysis services that allow engineers to define calculations and detect conditions in the data.
These may include:
- derived calculations
- KPI metrics
- rule-based analysis
- event detection such as Event Frames
This layer helps convert raw signals into operational information.
Visualization Tools
Finally, visualization tools such as PI Vision allow operators and engineers to monitor system performance through dashboards, trend charts, and reports.
These tools provide the human interface to industrial data.
For many years, this architecture worked extremely well and became the standard foundation for industrial data management.
But the industrial data landscape has changed.
4. The World Has Changed
Modern IT infrastructure has evolved dramatically over the past decade.
Organizations now operate in environments that include:
- cloud computing
- distributed data platforms
- real-time data pipelines
- machine learning
- AI-driven analytics
Industrial companies increasingly want to:
- integrate operational data with enterprise systems
- perform advanced analytics
- build predictive models
- enable real-time decision making
However, traditional historians were not designed with open data ecosystems in mind.
Integrating historian data with modern infrastructure often requires additional connectors, custom integrations, or data replication pipelines.
As a result, industrial data frequently remains isolated from the rest of the enterprise data landscape.
5. The Attempt to Bridge OT and IT
Over the past decade, many organizations recognized that industrial data held far more value than operational monitoring alone. Efforts to bridge the gap between OT and modern IT infrastructure accelerated. Industrial IoT platforms emerged. Cloud providers launched managed services for industrial data ingestion. Modern data platforms such as Databricks and Snowflake gained traction, promising the scalability and analytical power that traditional historians could never offer. Many organizations began exporting historian data into these platforms, hoping to finally unlock its value at enterprise scale.
The results were broadly disappointing — and not because these platforms lacked capability.
If anything, they were too capable in the wrong direction. They could process datasets at massive scale, support complex machine learning pipelines, and integrate with virtually any modern data ecosystem. But they were built for data engineers, not for the people actually running industrial operations. Constructing data pipelines, managing schemas, and writing analytical queries are routine tasks for an IT team. For a process engineer or plant operator focused on keeping equipment running, they represent an entirely different kind of burden.
There was a deeper problem, though — one that no amount of platform capability could solve.
When industrial data is exported into a general-purpose data platform, it arrives stripped of the context that gave it meaning in the first place. On the plant floor, a temperature reading is unambiguous: it belongs to a specific compressor, at a specific stage of a specific process, recorded just before a planned maintenance window. Moved into a data lake, that same reading becomes a row in a table — a floating-point number attached to a timestamp. The asset it came from, the process it belonged to, the events happening around it: all of that context quietly disappears in transit.
To do any meaningful analysis, engineers had to reconstruct that context manually — cross-referencing equipment records, process logs, and event histories before the actual work of analysis could even begin. The data was technically accessible. It just wasn’t understandable.
This is why so many OT-IT integration projects stalled at the proof-of-concept stage and never reached production. Bridging the gap required more than a more powerful platform. It required an architecture capable of preserving industrial context natively — from the moment data is captured, through storage, all the way to analysis. That is precisely what both traditional historians and general-purpose data platforms failed to provide, and it is the foundational problem that any AI-era industrial data infrastructure must solve.
6. Why This Becomes Even Harder in the AI Era
In the AI era, these limitations become even more significant.
AI systems do not simply require large amounts of data.
They require contextualized and accessible data.
Signals such as temperature, pressure, and vibration only become meaningful when the system understands:
- which asset generated the signal
- which process it belongs to
- what events occurred
- how equipment behaves over time
Without this context and open access to the data, AI systems struggle to produce meaningful insights.
This is one reason why many industrial AI initiatives struggle to deliver real operational value.
7. The Next Evolution: AI-Native Industrial Data Foundations
Industrial data infrastructure is now entering a new phase.
Evolution can be viewed as three stages:
A modern industrial data foundation must combine several capabilities:
- high-performance time-series storage
- asset-centric contextualization
- real-time stream processing
- event modeling
- modern visualization
- advanced analytics
- AI integration
- open architecture
Instead of simply storing signals, the system must help convert operational data into insights, predictions, and decisions.
Just as importantly, the system must be open, allowing industrial data to integrate naturally with enterprise platforms, analytics tools, and AI systems.
8. Where TDengine Fits
TDengine was founded in 2017 with a single focus: building a high-performance, horizontally scalable time-series database for industrial and IoT environments. That product — TDengine TSDB — has since been deployed across more than one million installations in over sixty countries, used by organizations ranging from fast-growing manufacturers to some of the world’s largest energy and automotive companies.
But a high-performance time-series database, as this series will argue, is a necessary foundation — not a complete solution. Two years ago, we began building TDengine IDMP, an Industrial Data Management Platform designed to sit above the TSDB layer and address everything a raw time-series engine cannot: asset modeling, data standardization, contextualization, event analysis, advanced analytics, and AI-driven operational insights.
Together, TSDB and IDMP form what we believe industrial data foundation should look like in the AI era — not a closed historian system that locks data inside proprietary formats, but an open foundation that preserves the operational context engineers depend on while making that data fully accessible to modern IT systems, analytics tools, and AI agents.
This is not a modest ambition. The industrial data foundation market has been dominated for decades by systems built on assumptions that no longer hold — assumptions about storage costs, data volumes, the role of AI, and what it means for a system to be “open.” We think those assumptions need to be replaced, not patched.
In the articles that follow, we will examine each dimension of this shift in detail: how asset modeling changes the way industrial data is organized and understood, why event analysis is more important in the AI era than it has ever been, what advanced analytics capabilities should be native to the platform rather than bolted on, and why openness is not just a feature but a prerequisite for any system that wants to remain relevant as AI reshapes industrial operations.
The goal of this series is not to sell a product. It is to make the case that industrial data infrastructure is at an inflection point — and that the organizations and vendors who recognize this early will have an enormous advantage over those who do not.
9. The Future of Industrial Software: Agent Interface + Data Foundation
In the AI era, the shape of industrial software itself is beginning to change.
Traditionally, industrial systems have been built as a collection of separate applications: SCADA, MES, data historians, reporting tools. Each system has its own interface, its own data model, and its own way of working. Engineers are forced to move between multiple systems just to answer a single operational question.
That model is starting to break, and AI is becoming the new interface.
Instead of navigating complex dashboards or manually querying data, engineers can interact with systems through natural language. AI can retrieve data, perform analysis, connect context, and generate explanations automatically.
In this new model: The interface becomes an AI agent, the system becomes the data foundation. In other words: The future of industrial software is Agent Interface + Data Foundation
AI agents are responsible for understanding intent, orchestrating workflows, and generating insights. The data foundation is responsible for providing complete, real-time, and contextualized data.
And this distinction matters.
Without a strong data foundation—one that understands assets, events, and operational context—AI cannot produce meaningful or reliable results. The intelligence of an agent is limited by the quality of its data foundation.
In this shift:
- Applications become less important
- Interfaces become AI agent
- The underlying data foundation becomes the real system
What used to be “software” is now being split into two layers:
- An intelligent interface (AI agents)
- A persistent, contextual data foundation
And ultimately: The core of industrial systems is no longer the application. It is the data foundation that everything runs on.
Key Takeaway
Data historians solved one of the hardest problems in industrial computing: reliably storing massive volumes of operational data. But in the AI era, simply storing data is no longer enough.
Industrial organizations need an open, scalable, AI-native data foundation that understands assets, events, and operational context — and that can power the next generation of industrial intelligence.
Traditional historians made data available, modern platforms made data scalable and AI-native data foundations make data understandable.
