The full power of TDengine, now free forever for up to 5,000 tags.

Explore More

How Long Should You Keep Raw Historian Data?

Juno Qiu

November 10, 2026 /

Keep detailed historian records long enough for the investigations that require them to be discovered and the evidence secured. Before replacing those records with minute summaries, test whether the summaries can still answer the actual troubleshooting questions. A useful long-term trend can be inadequate evidence for a short process upset.

For a historian engineer deciding which records may expire, the starting point is a list of questions the retained data must answer. Storage limits matter, but they do not establish that an average preserves the information an investigation needs.

Here, raw history means the records retained by the historian before this additional reduction. Collection settings, deadbands, or archive compression may already have removed detail. Verify that earlier boundary using the guidance on OPC UA sampling intervals and deadbands.

Separate faster queries from deleting detail

An aggregate query, an additional summary table, and a policy that replaces older records have different consequences. Identify which operation the proposed configuration performs.

For example, Ignition 8.1 Datasource History Providers can maintain pre-processed partitions alongside normal partitions. This uses additional storage to help queries. Data pruning is a separate setting that deletes old partitions when all their data exceeds the prune age. Enabling pre-processing alone does not establish a long-term archive or a policy for replacing raw records.

Other mechanisms deliberately replace detail. The TDengine Enterprise RSMA documentation describes downsampled storage from version 3.3.8.0: corresponding raw files are deleted after downsampling, and raw and downsampled time ranges do not overlap. That behavior makes the retained functions and windows part of the evidence decision. It should not be generalized to every TDengine query or edition.

Write down whether the proposed change adds summaries, moves recoverable detail to another store, or removes detail. Test retrieval from each destination before treating any of them as an archive.

Test what min, max, and average cannot distinguish

Consider a synthetic pressure signal with one good sample each second over [12:00:00, 12:01:00). Each sample represents the following one-second interval; the value is held until the next sample. Both histories start and finish at 10 bar.

HistorySeconds with a recorded pressure of 20 barAll other seconds
A: sustained excursion10, 11, 12, 13, 14, 1510 bar
B: separate spikes10, 12, 14, 16, 18, 2010 bar

Both contain six high samples and 54 low samples. Their minute summaries are identical:

minimum = 10 bar; maximum = 20 bar; count = 60

average = (54 × 10 + 6 × 20) / 60 = 11 bar

The first and last values are also identical. Under the stated hold model, both spend six seconds above 15 bar, so even retaining total time above that threshold would not distinguish them.

But their longest continuous excursion differs: A stays above 15 bar for six seconds; B has six separate one-second excursions. A question such as “did pressure remain above 15 bar for at least three continuous seconds?” has different answers.

A summary containing only those statistics cannot answer it. This example makes no claim about the exact fields a vendor’s native aggregate stores; timestamps of extrema or other metadata can preserve additional information. Inspect the actual representation. Nor does one-second history prove what happened between physical measurements: the duration calculation here follows an explicit model.

If continuous excursion duration is required, preserve adequate detailed history or a validated event representation containing the necessary boundaries, quality, and calculation rules. A derived event for one threshold will not necessarily support a later investigation using another threshold.

Choose retained records by investigation task

Use the following as a design worksheet, then validate it with the process owner. These are task-dependent choices, not default retention settings.

Required questionCandidate retained evidenceCheck before discarding detail
Is the process average drifting between shifts?Defined averages with window boundaries and coverage informationDoes the weighting and quality policy match the comparison?
Did a recorded value cross a limit?Valid minimum/maximum and enough context to locate the relevant windowDoes the result distinguish valid measurements from invalid readings?
Was an excursion continuous for three seconds?Adequate detailed records or validated excursion intervalsCan it distinguish histories A and B, including crossings at window edges?
Did pressure rise before a valve command?Sufficiently precise records for both signals with trustworthy timestampsDoes the retained representation preserve order at the required time scale?

Averages can remain useful after detail expires. Label that period’s available resolution and functions so a consumer knows which questions are still answerable. A chart should not imply that a minute average is a measurement taken at the minute boundary.

Retain quality and coverage information appropriate to the collection model. A count of 45 good observations out of 60 scheduled one-second samples indicates incomplete coverage. A count of 45 records from change-based collection does not, by itself, prove 15 seconds of missing data.

Set the raw window from discovery to preservation

Investigations do not always begin when an event occurs. Include the age of the oldest required context when the problem is discovered, the delay before someone preserves the records, and a justified operating margin.

Suppose a defect is discovered five days after the process event. The engineer needs two hours of preceding context and may need one additional day to identify the tags and preserve the records. The oldest required record is then six days and two hours old at preservation time, before any margin. A five-day raw window fails this scenario. These durations are illustrative inputs, not a recommended policy.

Base the inputs on incident and investigation records where available. Include delayed quality results, weekends, and access delays when they affect this process. Mandatory record-retention requirements must be established separately with the responsible owner; a troubleshooting estimate does not replace them.

An event-triggered capture can protect selected windows, but an unknown or undetected failure may never trigger it. Decide how that residual risk is handled. When detail moves to an offline archive, test its retrieval time as well as its existence.

Rehearse expiration before enabling it

Use a nonproduction copy or isolated test dataset. Keep a reference copy while exercising the proposed policy; do not shorten production retention to perform the experiment.

  1. Replay representative investigations. Include a quiet period, a short excursion, a sustained excursion, and an event crossing a summary boundary. Record the questions and required timing accuracy before comparing outputs.
  2. Introduce incomplete and late data. Include a known bad-quality interval and a late record within an already summarized window. Verify which results change and how recomputation works. The InfluxDB OSS v1 example explicitly limits continuous queries to recent data; its behavior is specific to that version and configuration.
  3. Exercise the expiration boundary. After the test policy removes detail, run the same investigation through the actual client or API. Confirm that it reads the intended tier and identifies unavailable detail. Restore an archived window if recovery is part of the design.
  4. Record the decision per signal group. Keep the questions still supported, detail and summary windows, aggregation and quality rules, archive location if any, measured retrieval time, and responsible owner. If a required question fails, change the retained representation or extend detail retention before proceeding.

Correction behavior also needs checking in the selected mechanism. TDengine’s RSMA documentation, for example, states that updates and deletes do not automatically trigger recalculation and describes accuracy limitations for some recalculations. Do not assume that the presence of a summary means it incorporates every later change.

The practical acceptance record is specific: “After detail expires, this dataset can still answer these investigation questions within these limits.” Where that statement cannot be demonstrated, retain the detail needed for the unresolved question.

Related Articles