An MQTT connection can recover while a historian still has a permanent gap. The missing question is not simply whether the broker or collector reconnects. It is where each measurement was durably held while the path was unavailable.
Store and forward reduces the chance that an outage becomes a history gap by retaining data near the point where a failure occurs and replaying it after recovery. It cannot recover a measurement that was never collected or never persisted at any boundary. Design it as a recovery path with a defined capacity, timestamp policy, and verification test—not as a blanket promise of complete history.
Map the failure boundaries before choosing a buffer
An industrial MQTT path commonly has four stages: publisher, broker, collector, and historian. Each stage owns a different loss boundary.
| Boundary | What can be lost or delayed | Useful protection |
|---|---|---|
| Publisher to broker | Measurements generated while the publisher cannot reach the broker | Local source or gateway buffer and replay |
| Broker to collector | Messages accepted by the broker while the collector is offline | Broker queue or eligible persistent session, subject to broker policy |
| Collector to historian | Messages already consumed but unable to be transformed or written downstream | Collector-side persistent buffer and replay |
| Historian write and query | Rejected, mis-mapped, or incorrectly timed records | Write-error handling, validation, and post-recovery queries |
The placement decision follows the failure you need to survive. A source-side buffer can protect measurements during a site-to-broker interruption. A broker-side queue can help an offline subscriber receive messages that the broker already accepted. A collector-side buffer can protect messages after collection but before the historian confirms storage. One of these mechanisms does not automatically cover the others.
Do not treat QoS or a session as historian storage
MQTT QoS describes delivery behavior on an MQTT hop. A persistent session can preserve eligible subscription state and queued deliveries under the broker’s configured limits. Both are useful, but neither proves that the source retained a measurement before it reached the broker, nor that a collector stored it after receiving it.
For example, a gateway may publish at QoS 1 and reconnect successfully after a WAN outage. If it discarded samples while offline, the broker has nothing to deliver later. Conversely, the broker may have queued messages for a collector, but the collector can still fail after consuming a message and before the historian write succeeds. The recovery design has to name the durable handoff at every important boundary.
Put store and forward as close as practical to the failed link
Start with the longest and least reliable link. A remote site with intermittent VPN access generally needs persistence on the site side, where measurements still exist. A central collector that temporarily loses access to the historian needs a buffer on or immediately behind that collector. Avoid assuming an upstream broker queue protects a downstream database failure unless the collector has not acknowledged or consumed the message and the broker is configured to retain it.
For a TDengine TSDB Enterprise deployment, the documented taosX-Agent Store and Forward capability is available for MQTT real-time collection tasks. When enabled, the Agent writes collected data to a local persistent queue during an Agent-to-taosX interruption and resumes sending from a confirmed breakpoint after recovery. The documentation also makes the trade-off explicit: disk-backed buffering adds end-to-end latency and consumes local disk capacity. Treat capacity planning, local-disk health, and the maximum supported outage duration as architecture requirements.
The MQTT ingestion documentation describes Cache Realtime Data as the setting used to enable disk-backed buffering for the applicable collection task. It is disabled by default and is not a separate recovery layer. This mechanism protects collected data along the Agent-to-taosX path; it does not recover measurements lost before they reach the collector. Confirm the deployed edition, connector version, cache directory, and failure behavior before relying on it. For the historian layer itself, see TDengine TSDB.
Preserve event time and make replay safe
Replay should preserve when the industrial event happened, not make an old sample look new. Include a source timestamp in the payload when the source can supply one, then validate its unit, time zone convention, and clock quality. Keep ingestion time separately when it is useful for diagnosing delay. During recovery, sorting by arrival order can otherwise create misleading trends and late-data calculations.
Duplicate handling matters as much as buffering. A resend can occur after an ambiguous failure, and a collector may restart before it records that a downstream write succeeded. Where the source provides a stable event identifier or sequence number, retain it with the record and define how the historian or ingestion layer handles repeats. Do not silently deduplicate solely by value: two identical readings can be two legitimate observations.
Ordering is also a design choice. A replay backlog may arrive alongside live traffic, and messages from different publishers can legitimately interleave. Decide whether the historian should prioritize event time, source sequence, or arrival time for each measurement class. Alarm-like events and high-frequency telemetry often need different rules.
Size the backlog for a real outage, not an optimistic one
Estimate required storage from the worst credible outage, sustained message rate, average serialized message size, and an operating margin. Include retries, serialization overhead, and free-space alarms. The useful question is: “How many hours of this source can we retain locally before the buffer begins rejecting or overwriting data?”
Also decide what happens at capacity. A policy can slow collection, discard by priority, generate an OT alarm, or require operational intervention. Each option has consequences. The unsafe default is having no visible policy until a full disk ends collection during an incident.
Test recovery as a history-completeness exercise
Test the exact deployment before calling it resilient. Generate a controlled stream with a known source timestamp and monotonically increasing sequence number. Then test one failure at a time:
- Disconnect the publisher or gateway from the broker.
- Keep the broker reachable but stop the collector.
- Break the collector-to-historian route while collection continues.
- Restore each route and query the historian for sequence gaps, duplicates, late arrivals, and timestamp correctness.
Record the buffer depth, disk consumption, reconnect time, and the exact number of records expected versus stored. Repeat with an outage longer than a routine reconnect, plus a collector restart. A green connection state is not the pass condition; the pass condition is an explainable, queryable historical record set after recovery.
Reliable recovery is therefore a chain of intentional guarantees: collection at the source, durable persistence at the right boundary, controlled replay, and a historian query that proves the result. MQTT can carry the telemetry, but the architecture determines how much history survives an outage.
