The full power of TDengine, now free forever for up to 5,000 tags.

Explore More

Keep Downstream Data Current When Historian History Changes

Juno Qiu

November 12, 2026 /

To keep a downstream copy current, discover changes to previously copied records and apply them using a stable record identity and an explicit revision policy. Advancing a query only by measurement time will miss a correction made today to last month’s value. The transfer also needs a defined treatment for deletions and any calculations that used the earlier value.

For a historian engineer, this is a continuing synchronization task after the initial backfill. The source may be a historian, a lab system or another authorized operational record store. The available interface determines how much of the procedure can be automated.

Keep event time separate from change position

An observation can have several useful times: when the sample or event belongs in the process history, when it was changed, and when the downstream system received it. These fields answer different questions.

The integration needs a reliable way to find changes since its last completed synchronization. A source may offer an ordered change sequence, a versioned export, a modification timestamp with documented guarantees, or no suitable change feed at all. Do not substitute the greatest measurement timestamp for that position.

The public LIMS-to-PI correction discussion describes a concrete failure: an edited lab value did not update its PI copy. Suggested rereads and database approaches demonstrate possible designs, not a verified universal fix for OPC HDA interfaces.

Choose a mechanism the source actually supports

MechanismWhat it can provideBoundary to establish
Ordered change feed or database change trackingIdentifies changed keys and potentially deletion operationsRetention, checkpoint validity, ordering and current-value retrieval
Modification-time queryFinds rows changed after a recorded positionTimestamp ties, commit visibility, clock behavior and deleted rows
Re-read and compare historical windowsDetects differences in the reread scopeConsistent source view, permitted query load and maximum correction age
Manual revision notificationSupplies approved correction scope when no machine-readable mechanism existsNamed owner, response time and traceable re-extraction

For example, SQL Server Change Tracking uses synchronization versions and requires checking whether an older version is still valid. It can identify deleted keys; current rows and consistent enumeration require additional handling. This is an example for an authorized, configured SQL source, not a feature to assume in a historian connector or to enable in a vendor database without approval.

A modification timestamp is weaker than a documented ordered feed. Two edits may share a timestamp; a transaction can commit after an enumeration has passed its apparent time. Use the source’s consistency protocol. An overlap and deduplication strategy can reduce risk, but it does not prove completeness without a bound on delayed visibility. Hard deletions need a retained deletion marker, change log or authoritative comparison.

Work through a correction stream

This fictional source provides immutable observation IDs, monotonically ordered revision numbers and retained deletion operations. The destination keeps both received revisions and a current projection.

RevisionObservationEvent timeOperationPayload
101O17Monday 10:00Insert8.0 mg/L, Good
102O18Monday 10:05Insert7.0 mg/L, Good
103O17Monday 10:00Replace8.4 mg/L, Good
104O18Monday 10:05DeleteObservation withdrawn

After revision 102, the current projection contains O17 = 8.0 and O18 = 7.0. After 104, it contains O17 = 8.4 and an explicit withdrawn state for O18. A consumer must not interpret the withdrawn reading as a measured zero.

If revision 103 is replayed, the same observation and revision should not create another current reading. If 102 arrives after 104, the older revision must not resurrect O18. Those outcomes depend on the example’s source revision ordering; they cannot be inferred from arrival order.

Persist the applied revision and checkpoint so a restart can replay safely. Where source and destination cannot share one transaction, retain a durable intake log and use replay-safe application before advancing the intake checkpoint. Keep errors visible rather than silently skipping a revision and reporting the stream current. For a TDengine TSDB destination, check its primary-key and partial-write rules.

A modified-history read is not automatically a change feed

OPC UA Historical Access provides Read Modified functionality for supported historical nodes. Its historical-time domain and modification records need careful interpretation: a replacement record can contain the old value that was changed, not the current replacement. Multiple modifications and retained detail are server-dependent.

Do not treat that response as a globally ordered “changes since last synchronization” stream. Establish how to enumerate affected historical times, obtain the current value or deletion state, and distinguish the operations the server exposes. The specification’s update-type enumeration distinguishes insert, replace, update and remove; it does not establish that every server or connector supports them.

If the only permitted strategy is to reread the previous seven days, document the consequence: corrections older than seven days are outside automatic discovery. Add periodic wider reconciliation or a manual process if those changes matter. Do not describe a lookback as complete unless the business and source constrain correction age accordingly.

Carry the revision into derived results

List the reports, aggregates and extracts that depend on the corrected observation. In the example, an arithmetic average of the two initially valid readings would change from 7.5 mg/L to 8.4 mg/L after replacement and withdrawal, assuming the report excludes withdrawn values. Its accepted sample count changes from two to one. The revised report should expose that change rather than merely replacing the displayed average.

Recompute the affected scope under the same documented quality and calculation policy, retain lineage to the input revisions, and mark an earlier delivered result superseded when appropriate. Retract or flag a result if a required input has been withdrawn and no valid calculation remains. Record gaps in change discovery separately from normal transfer delay.

Test replacement at an old event time, replay, out-of-order revision delivery, deletion, a correction beyond the lookback, and an expired source checkpoint. Acceptance means the downstream current state and dependent results reflect the supported revision contract. The multi-site integration article provides broader ownership context; the correction contract should name who investigates every unresolved divergence.

Related Articles