As new technologies like the Industrial IoT are implemented across enterprises, more data is being collected than ever before. The size of modern industrial datasets can quickly become overwhelming, with single sites easily generating terabytes of data per year. Without proper management, these datasets can incur massive storage costs, especially when data is stored in the cloud.
To ensure that enterprises have affordable access to all their data, TDengine offers tiered storage with S3 support. Note that this is an enterprise feature available only in TDengine Enterprise.
Tiered Storage
In industrial scenarios, the most recent data is generally the most valuable. TDengine’s built-in caching functionality automatically stores this data in memory so that it can be accessed as quickly as possible. For data written from cache to disk, you can configure a three-tier storage model that stores data in specified time ranges on different storage media. A typical configuration is shown in the following figure.
In this configuration, data is written from cache to solid-state drives (SSD) where it is stored for 10 days. Once this period has elapsed, the data is transferred to cheaper magnetic storage. Finally, after six months, the data is moved to S3 buckets in the cloud where it is retained at low cost for three years. A configuration like this can provide the best balance between data access and storage costs. By transferring older data to cheaper but slower storage media, you benefit from the most cost-effective solution while ensuring that the performance of your applications is not affected.
Tiered storage in TDengine is completely transparent to applications, which access your data in the same manner no matter whether it is stored on an SSD or in S3. This means that processing historical and real-time data is unified, enriching the insights of streaming data with context from vast reserves of historical information without any special handling on the application side. Best of all, tiered storage is completely configurable in TDengine with simple SQL statements, so you can define the optimal policy for your organization.