In the age of IoT, effective data management, including accessibility, is critical. The proliferation of connected sensors generates a massive volume of timestamped data that needs to be shared, centralized, and analyzed. This three-part blog series reviews the value of collaborating on time series data and the challenges associated with it. Specifically, it explores the capabilities and limitations of PI System, a popular data historian used throughout manufacturing, and illustrates how TDengine can be used to extend this system to enable effective data centralizing and sharing.
Blog: Supercharging PI System for Data Sharing & Advanced Analytics: Part One
For more on extending PI System’s capabilities for data sharing and advanced analytics, download the full whitepaper.
Modernizing the Time Series Database
TDengine is a cutting-edge time-series database purpose-built for IoT. Designed as a distributed database from day one, it can support billions of IoT devices effortlessly while consistently outperforming other time-series databases in data ingestion, querying, and data compression.
Despite its impressive capabilities, TDengine is user-friendly and intuitive to use. On the surface, it looks like a relational database that uses SQL, however, it’s designed as a real-time data engine optimized for data in-flight. TDengine’s unique architecture also makes it easy to share and replicate data across multiple instances, enabling rapid access to vital data.
User Management
TDengine’s user management system is defined by two main concepts: privileges and resources.
- Privilege: a level of permission, for example, “Read Database”. Role-based access control allows admins to assign view and edit responsibilities to different roles or user groups.
- Resource: this is what the privileges apply to, which can be the entire organization, a specific instance, database, or even topic.
The user management system can be used to share data internally and, unlike many other systems, externally, allowing users to set fine-tuned data-sharing privileges for the entire organization, a specific instance, a specific database, or a specific SQL query through data subscriptions (e.g. a table filtered by a specific tag for a defined amount of time).
Data Subscriptions
Subscribing to Updated Data
TDengine data subscriptions and consumers act similarly to other widely-used messaging queue solutions. Applications can subscribe to real-time data updates, making it easier to process data in the order that events occur. TDengine doesn’t require users to deploy Kafka or continuously query the database. Partners get notifications as soon as the data is received, simplifying time series data processing and eliminating the need for additional messaging queue solutions.
Defining Topics
TDengine’s data subscription feature provides greater flexibility than similar products such as PI System, thanks to its use of topics defined by query conditions based on an existing table, subtable, or supertable – or in other words, a SELECT statement.
By leveraging SQL, data can be filtered by tag, table name, column, or expression, and then processed using scalar or user-defined functions.
This approach allows applications to control the granularity of data on demand, while filtering and preprocessing tasks are handled by TDengine rather than the application layer. As a result, the amount of transmitted data is reduced, and the complexity of applications is streamlined.
Data Sharing
TDengine enables data sharing with third parties, such as customers or consultants, using subscriptions and topics. This approach provides fine-tuned access control to data while allowing third parties to receive real-time updates on specific topics. Consumers can be programmatically created to consume messages in the queue for each topic, and TDengine can also consume topics, allowing data to be replicated between instances. Additionally, both real-time and historical data can be shared via topics, with the option to set start and end dates or continuously stream data.
Implementation
TDengine is a real-time streaming database that uses a write-ahead-log (WAL) to not only prevent data loss, but also for stream processing and data subscriptions.
The system keeps track of where individual subscription consumers are in the WAL. Eacn consumer receives updated data, and multiple consumers can be grouped together as a consumer group to improve efficiency and availability. If the data in a supertable is sharded across multiple virtual nodes (vnodes), multiple consumers can consume it more efficiently than a single consumer through multi-threaded, distributed data consumption.
TDengine also includes an acknowledgment mechanism to ensure at-least-once delivery, even in complex environments where machines may crash or restart.
See TDengine 3.0 Data Subscription for more information.
Integrating Advanced Analytics
It’s hard to imagine an enterprise that isn’t focused on improving their operations with the help of advanced analytics, and there’s obviously been a large increase in the number of legacy companies and new entrants that have launched new AI-enabled tools.
As an open system, TDengine makes it easy for companies to integrate a range of third party solutions, including providers of advanced analytics. The data sharing capabilities noted above also ensure these third parties are only working with the data they’ve specifically been giving access to.
Hybrid Solution
TDengine’s PI System integration allows data in PI and TDengine to coexist. New points are stored in TDengine while legacy points remain in PI – and with the TDengine Data Reference, all data stored in TDengine can be viewed and manipulated like PI points within AF and PI Vision.
Users can replicate data to TDengine Cloud for analytics and then view the results in PI Vision dashboards alongside PI data, without creating any new PI points.
Visualization
Users can also view new data stored in TDengine together with existing data stored in PI in the TDengine Data Explorer or other modern visualization tools. A number of options exist for building dashboards based on these analytics:
- Use the Grafana or Google Data Studio integrations for ad-hoc dashboards.
- Build custom dashboards with TDengine’s client libraries.
See TDengine’s documentation for information on creating dashboards with Grafana.
The Open System Future
TDengine is aligned with the open-source future of IIoT – meaning that breaking down silos and centralizing data is just the start. Users can integrate TDengine seamlessly with modern analytics and visualization tools, such as Grafana or Google Data Studio, or build custom dashboards by leveraging TDengine’s client libraries. The open nature of TDengine means there are endless opportunities to generate useful insights that drive greater efficiencies and improve outcomes.
Visit TDengine for PI System for information on how to easily centralize and share your PI System data.
