The full power of TDengine, now free forever for up to 5,000 tags.

Explore More

How to Size an Industrial Edge Data Buffer

Juno Qiu

November 4, 2026 /

An edge buffer can hold collected records while a downstream connection is unavailable. Its useful outage duration depends on the actual data written, free disk space and queue behavior. Recovery also depends on whether the restored path can drain the backlog faster than new data arrives.

The buffer covers only the path after records reach it. It cannot recover a measurement that a failed sensor, PLC interface or collector never captured.

Estimate storage using measured traffic

Measure the serialized bytes written to the queue per second under representative conditions, including protocol or queue overhead where applicable. Use a peak or conservative rate for the outage scenarios you need to survive. Then estimate:

required usable bytes = peak queued bytes/second × outage seconds × safety factor

For illustration, 20,000 bytes per second for eight hours is 576,000,000 bytes before a safety factor. A factor of 1.5 gives 864,000,000 bytes, approximately 824 MiB. This is an example, not a recommended factor or device specification. Allow separately for filesystem reserve, logs, other applications and growth. Confirm how the queue behaves when its configured limit or disk space is exhausted.

Use observed bursts and changing tag counts rather than assuming that record count alone determines storage. Compression, metadata, batching and queue implementation can change the actual bytes per record. Measure again after configuration changes.

Check that the backlog can drain

If the link recovers while collection continues, use consistent units for rates. A first approximation is:

drain time = backlog bytes / (sustained transfer bytes per second − new queued bytes per second)

This is meaningful only if sustained transfer exceeds the continuing arrival rate and rates and units are comparable. For a 576 MB backlog, 100 kB/s sustained outbound transfer capacity before subtracting new arrivals, and 20 kB/s continuing arrivals, the net backlog drain rate is 80 kB/s. The simple estimate is 7,200 seconds, or two hours. In practice, destination throttling, acknowledgments, retries, competing traffic and batching may change the result. If available throughput is at or below arrivals, the backlog will not clear without a change.

Test the failure boundary

With site approval, interrupt the intended downstream path for a bounded period. Record queue growth, alerts, source load, oldest record age and behavior near capacity. Restore the path and compare observed drain time, duplicates and missing intervals with the estimate. Confirm how restart, disk failure and power loss affect queued records for the actual software and hardware.

Some products offer persistent queues; for example, TDengine’s taosX-Agent documentation describes a particular disk-based option and its latency trade-off. Check edition, version, connector support and configuration rather than generalizing that behavior to every pipeline. Assign an owner for disk monitoring, recovery and exception review.

Related Articles