Keep detailed historian records long enough for the investigations that require them to be discovered and the evidence secured. Before replacing those records with minute summaries, test whether the summaries can still answer the actual troubleshooting questions. A useful long-term trend can be inadequate evidence for a short process upset.
For a historian engineer deciding which records may expire, the starting point is a list of questions the retained data must answer. Storage limits matter, but they do not establish that an average preserves the information an investigation needs.
Here, raw history means the records retained by the historian before this additional reduction. Collection settings, deadbands, or archive compression may already have removed detail. Verify that earlier boundary using the guidance on OPC UA sampling intervals and deadbands.
Separate faster queries from deleting detail
An aggregate query, an additional summary table, and a policy that replaces older records have different consequences. Identify which operation the proposed configuration performs.
For example, Ignition 8.1 Datasource History Providers can maintain pre-processed partitions alongside normal partitions. This uses additional storage to help queries. Data pruning is a separate setting that deletes old partitions when all their data exceeds the prune age. Enabling pre-processing alone does not establish a long-term archive or a policy for replacing raw records.
Other mechanisms deliberately replace detail. The TDengine Enterprise RSMA documentation describes downsampled storage from version 3.3.8.0: corresponding raw files are deleted after downsampling, and raw and downsampled time ranges do not overlap. That behavior makes the retained functions and windows part of the evidence decision. It should not be generalized to every TDengine query or edition.
Write down whether the proposed change adds summaries, moves recoverable detail to another store, or removes detail. Test retrieval from each destination before treating any of them as an archive.
Test what min, max, and average cannot distinguish
Consider a synthetic pressure signal with one good sample each second over [12:00:00, 12:01:00). Each sample represents the following one-second interval; the value is held until the next sample. Both histories start and finish at 10 bar.
| History | Seconds with a recorded pressure of 20 bar | All other seconds |
|---|---|---|
| A: sustained excursion | 10, 11, 12, 13, 14, 15 | 10 bar |
| B: separate spikes | 10, 12, 14, 16, 18, 20 | 10 bar |
Both contain six high samples and 54 low samples. Their minute summaries are identical:
minimum = 10 bar; maximum = 20 bar; count = 60
average = (54 × 10 + 6 × 20) / 60 = 11 bar
The first and last values are also identical. Under the stated hold model, both spend six seconds above 15 bar, so even retaining total time above that threshold would not distinguish them.
But their longest continuous excursion differs: A stays above 15 bar for six seconds; B has six separate one-second excursions. A question such as “did pressure remain above 15 bar for at least three continuous seconds?” has different answers.
A summary containing only those statistics cannot answer it. This example makes no claim about the exact fields a vendor’s native aggregate stores; timestamps of extrema or other metadata can preserve additional information. Inspect the actual representation. Nor does one-second history prove what happened between physical measurements: the duration calculation here follows an explicit model.
If continuous excursion duration is required, preserve adequate detailed history or a validated event representation containing the necessary boundaries, quality, and calculation rules. A derived event for one threshold will not necessarily support a later investigation using another threshold.
Choose retained records by investigation task
Use the following as a design worksheet, then validate it with the process owner. These are task-dependent choices, not default retention settings.
| Required question | Candidate retained evidence | Check before discarding detail |
|---|---|---|
| Is the process average drifting between shifts? | Defined averages with window boundaries and coverage information | Does the weighting and quality policy match the comparison? |
| Did a recorded value cross a limit? | Valid minimum/maximum and enough context to locate the relevant window | Does the result distinguish valid measurements from invalid readings? |
| Was an excursion continuous for three seconds? | Adequate detailed records or validated excursion intervals | Can it distinguish histories A and B, including crossings at window edges? |
| Did pressure rise before a valve command? | Sufficiently precise records for both signals with trustworthy timestamps | Does the retained representation preserve order at the required time scale? |
Averages can remain useful after detail expires. Label that period’s available resolution and functions so a consumer knows which questions are still answerable. A chart should not imply that a minute average is a measurement taken at the minute boundary.
Retain quality and coverage information appropriate to the collection model. A count of 45 good observations out of 60 scheduled one-second samples indicates incomplete coverage. A count of 45 records from change-based collection does not, by itself, prove 15 seconds of missing data.
Set the raw window from discovery to preservation
Investigations do not always begin when an event occurs. Include the age of the oldest required context when the problem is discovered, the delay before someone preserves the records, and a justified operating margin.
Suppose a defect is discovered five days after the process event. The engineer needs two hours of preceding context and may need one additional day to identify the tags and preserve the records. The oldest required record is then six days and two hours old at preservation time, before any margin. A five-day raw window fails this scenario. These durations are illustrative inputs, not a recommended policy.
Base the inputs on incident and investigation records where available. Include delayed quality results, weekends, and access delays when they affect this process. Mandatory record-retention requirements must be established separately with the responsible owner; a troubleshooting estimate does not replace them.
An event-triggered capture can protect selected windows, but an unknown or undetected failure may never trigger it. Decide how that residual risk is handled. When detail moves to an offline archive, test its retrieval time as well as its existence.
Rehearse expiration before enabling it
Use a nonproduction copy or isolated test dataset. Keep a reference copy while exercising the proposed policy; do not shorten production retention to perform the experiment.
- Replay representative investigations. Include a quiet period, a short excursion, a sustained excursion, and an event crossing a summary boundary. Record the questions and required timing accuracy before comparing outputs.
- Introduce incomplete and late data. Include a known bad-quality interval and a late record within an already summarized window. Verify which results change and how recomputation works. The InfluxDB OSS v1 example explicitly limits continuous queries to recent data; its behavior is specific to that version and configuration.
- Exercise the expiration boundary. After the test policy removes detail, run the same investigation through the actual client or API. Confirm that it reads the intended tier and identifies unavailable detail. Restore an archived window if recovery is part of the design.
- Record the decision per signal group. Keep the questions still supported, detail and summary windows, aggregation and quality rules, archive location if any, measured retrieval time, and responsible owner. If a required question fails, change the retained representation or extend detail retention before proceeding.
Correction behavior also needs checking in the selected mechanism. TDengine’s RSMA documentation, for example, states that updates and deletes do not automatically trigger recalculation and describes accuracy limitations for some recalculations. Do not assume that the presence of a summary means it incorporates every later change.
The practical acceptance record is specific: “After detail expires, this dataset can still answer these investigation questions within these limits.” Where that statement cannot be demonstrated, retain the detail needed for the unresolved question.


