Whether we know it or not, time-series data is all around us, powering many of the products and systems that we depend on in our daily lives. Although the name may make it sound complex, time-series data is actually just the combination of a measurement and the time at which that measurement was taken. Essentially all monitoring and tracking systems — from something as simple as an app recording how many steps you take each day, to complex IIoT systems keeping track of thousands of industrial devices in smart factories — rely on time-series data to produce results.
And while time-series data is widely used across industries and applications, not everyone is using a time-series database (TSDB) — a database management system specially designed and optimized to store and process time-series data. This article will discuss the drawbacks of general-purpose and legacy systems when it comes to time-series data and how you can benefit by moving to a system built to purpose.
Other Options and Their Trade-Offs
Relational Databases
When designing systems, many developers and architects tend to stick with what they know. In the database world, this often means traditional relational databases like MySQL, Oracle, and SQL Server. And when these products are used to store and query time-series data, they seem to work well — at first.
In fact, while initial testing and proof of concept are often done on small amounts of data to speed up the development process, real-world time-series datasets grow rapidly over time, especially when new sites or devices are added as the business grows. Despite the best efforts of DBAs, there comes a time when the performance of relational databases just can’t keep up with the scale of the data: queries slow to a crawl, and reports that were once generated in seconds aren’t ready till the end of the day. Businesses that built their time-series systems on relational databases are then presented with two unsavory choices: either spend huge amounts of money to upgrade hardware until performance is acceptable, or rearchitect the entire system around a purpose-built time-series database.
NoSQL and Big Data Platforms
Developers are influenced by trends just as much as the rest of us are, and this can influence the systems that they choose to deploy as well. For a time, it was popular to use Hadoop for processing any dataset expected to be large, but now it seems that NoSQL databases like MongoDB have become the favorite all-purpose multitool. However, like the screwdriver on a Leatherman, it may work in a pinch, but when you really get down to work, you’ll wish you had the real thing.
It’s easy to get data into NoSQL databases, which are schemaless or have dynamic schemas, and the initial setup phase of a project can be accelerated in this way — there’s no need to have your data teams spend time designing an optimal schema when you can just cram everything into MongoDB. The problem comes when you try to get that data back out. Data analysts are forced to learn new query languages instead of relying on the industry standard SQL, and even then are not always able to retrieve they data they need — NoSQL databases are limited when it comes to complex queries and joins. The ease of use that these databases provide in reality is often just offloading data processing work onto your own development team.
Finally, with the massive scale of time-series datasets, disk space often balloons due to the inability of NoSQL databases to store and compress data efficiently. And time-series data is structured by nature, so the benefits of NoSQL are hardly worth the drawbacks.
Data Historians
In traditional industries like process manufacturing, data historians have been used for decades to work with time-series data. Like modern time-series databases, they are built to handle time-series workloads, and high-end products like PI System can deliver high performance and stability. However, data historians come at a technical and financial price that not all companies can afford to pay.
Many historians were originally designed thirty years or more in the past, and their architectures have not aged well. They often only run on Windows, offer minimal if any support for cloud computing, and refuse to interoperate with other systems without significant manual effort. What’s more, the price of these legacy systems can be astronomical, quickly rising to six or seven figures even for small-scale operations, and increasing based on the number of data points. Considering the affordability and flexibility of modern open-source and SaaS solutions, many enterprises would gladly migrate away from their historians — if they were not locked in by vendors that make it difficult to do so.
How Time-Series Databases Benefit You
Your family doctor may be excellent for yearly checkups and minor ailments, but if you have a serious or complex condition, you’ll want to see a specialist. The same is true for databases, and for storing and processing time-series data, a purpose-built time-series database can offer a wealth of benefits.
-
Ingestion speed: Time-series databases can write data much faster than general-purpose databases, meaning that performance remains high even as the scale of data collection increases.
-
Query latency: Time-series databases are optimized such that your queries return quickly even when aggregating data from large numbers of devices.
-
Storage space: Time-series databases store and compress data more efficiently, reducing the amount of disk space needed to store your data.
All three of the above items can greatly contribute to reducing the cost of your data operations — with higher ingestion, query, and compression performance, you can achieve superior results with less expensive hardware and fewer servers. While general-purpose databases require custom coding to implement most time-series functions and capabilities, time-series databases build them into the system, reducing the workload of your team. There is also a large ecosystem of products that offer analytics and visualization for time-series data, like Grafana and Seeq, so you can build the best infrastructure for your organization. And time-series databases are designed to handle large-scale data, so you don’t have to worry about scalability — your data infrastructure is ready to grow along with your business.
If you’re considering a time-series database for your next project, contact us to learn more about how TDengine — the only time-series database designed for industry — can meet your needs. For brownfield deployments, we offer a variety of data connectors to help migrate your data from existing systems like PI System, Wonderware Historian, MySQL, and MongoDB, and for new projects, we support industry-standard protocols like MQTT and OPC. We’re confident that we can help make your data infrastructure more efficient and reduce costs while maintaining the performance and stability that your organization demands.
