As new technologies like the Industrial IoT are implemented across enterprises, more data is being collected than ever before. The size of modern industrial datasets can quickly become overwhelming, with single sites easily generating terabytes of data per year. Without proper management, these datasets can incur massive storage costs, especially when data is stored in the cloud.
To ensure that enterprises have affordable access to all their data, TDengine offers tiered storage with S3 support. Note that this is an enterprise feature available only in TDengine Enterprise.
Tiered Storage
In industrial scenarios, the most recent data is generally the most valuable. TDengine’s built-in caching functionality automatically stores this data in memory so that it can be accessed as quickly as possible. For data written from cache to disk, you can configure a three-tier storage model that stores data in specified time ranges on different storage media. A typical configuration is shown in the following figure.
In this configuration, data is written from cache to solid-state drives (SSD) where it is stored for 10 days. Once this period has elapsed, the data is transferred to cheaper magnetic storage. Finally, after six months, the data is moved to S3 buckets in the cloud where it is retained at low cost for three years.
A configuration like that described above can provide the best balance between data access and storage costs. By transferring older data to cheaper but slower storage media, you benefit from the most cost-effective solution while ensuring that the performance of your applications is not affected. This tiered storage policy is completely configurable in TDengine, so you can define the optimal policy for your organization.