Twin State Reconciliation Under Disordered Telemetry

#Abstract

Observations reaching an industrial digital twin arrive late, duplicated and out of order, at rates that make disorder the normal case rather than an exception. This report gives a reconciliation construction based on event-time folding with a per-channel reorder window derived from measured lateness, argues that staleness must be represented as a value rather than an absence, and states the conditions under which interpolation across a gap is a defect rather than a convenience. The lateness measurements the window is derived from are not reproduced here, which is the report's main evidential gap.

#Motivation

A digital twin is a virtual counterpart of a physical object maintained by a data connection between the two [1]. Classifying implementations by the direction and automation of that connection separates digital models, digital shadows and digital twins proper [2], and in every category the state of the virtual object must be derived from observations that the physical world delivers on its own schedule.

That last clause is the whole difficulty. A pipeline built in an office receives observations promptly, in order, once each. A pipeline connected to a remote compressor station receives them when the link comes back, sometimes twice, stamped by a clock that has been drifting since the last maintenance visit. The gap between those two situations is not a matter of degree. It changes which designs are correct.

On an operating site, late data is a double-digit percentage of the total. A design that treats it as an exception will spend its life in the exception.

#Four sources of disorder, which need separating

Disorder is usually discussed as a single phenomenon and handled with a single mechanism. The four sources below behave differently enough that a mechanism tuned for one is actively wrong for another. The characterisation is qualitative, drawn from operating sites rather than from a published measurement campaign, and the absence of that campaign is this report's main evidential gap.

#Buffered flush

An edge gateway loses its uplink and continues collecting to local storage. When connectivity returns it delivers everything at once. The batch is internally ordered and collectively very late, sometimes by hours. This is the most common source by volume and the least damaging by nature, because the ordering within the batch is intact and the event times are honest. What it breaks is any design that sizes buffers or windows for the live rate, since the arrival rate during a flush bears no relation to the sampling rate.

#Redundant collection

Two collectors are attached to one source, usually because somebody wanted redundancy and got it. They produce byte-identical observations at unpredictable relative delays. This is benign if identity is handled and quietly corrupting if it is not: a counter that sums observations will double, and a rate computed from consecutive samples will halve or spike depending on which pair it sees.

#Clock skew at the device

Field devices stamp observations from their own clocks, which drift, and which are sometimes set by hand during commissioning. Observations arrive stamped slightly in the future, or occasionally very far in the future when a clock has reset to an epoch default. Synchronisation protocols bound this where they are deployed [3], and the useful framing from the clock literature is that any timestamp carries an uncertainty interval rather than a point [4], an idea later systems made explicit in their APIs [5]. Most field devices expose no such interval, so the uncertainty exists and is simply unstated.

#Bulk backfill

Historical data is imported from a historian or a previous system. The volume and rate resemble nothing in the live stream, and the event times may span years. This is the case that turns a correct incremental design into an outage, because every mechanism that re-projects on late arrival will re-project across the entire history at once.

1source              order within batch   lateness magnitude   duplicates   frequency
2------------------  -------------------  -------------------  -----------  -----------
3buffered flush      preserved            minutes to hours     no           routine
4redundant collect   n/a                  near zero            yes          continuous
5clock skew          preserved            negative (future)    no           per device
6bulk backfill       preserved            days to years        sometimes    rare, planned
Copy

Reading down the duplicates column explains why deduplication and reordering are frequently conflated: only one of the four sources produces duplicates at all. Reading down the lateness column explains why a single global window cannot serve, since the range spans roughly nine orders of magnitude.

#State as an event-time fold

State is defined as a fold over observations keyed on the time each observation describes rather than the time it arrived. This is the event-time and processing-time separation that stream processing settled some time ago [6] [7].

Because the fold is keyed on the event time τ(o), replaying observations in any order converges on the same state. That property is what makes a derived store recoverable and what makes historical questions answerable.

#A per-channel reorder window

Folding on event time removes the dependence on arrival order but does not by itself say when a value may be read. If a state at time t can always be revised by an observation that has not arrived yet, no answer is ever final, and a system that emits nothing until it is certain emits nothing at all. Sorting requires a window, and the window must close.

The choice this report argues for is that the window is derived per channel from that channel's own observed lateness rather than fixed globally.

Observations arriving within ω sort into place before state is recomputed. Observations arriving outside it are accepted and stored as corrections, which re-project their window and propagate forward. Nothing is discarded for lateness, which is the point at which this construction departs from the usual watermark treatment, where late arrivals past the allowed lateness are dropped or diverted.

#Why the floor is the sampling interval

The maximum with λ, the channel's declared sampling interval, prevents a degenerate case that occurred in practice. A wired channel with very low lateness produces a 99th percentile of a few milliseconds. Setting the window to that value means every observation is treated as late relative to its predecessor's window, because the window closes before the next sample is even taken. The floor makes the window at least one sampling period, so that consecutive samples have a chance to be considered together.

#Why the 99th percentile rather than the maximum

Sizing the window at the observed maximum lateness makes the window as slow as the worst outage the channel has ever had, which for a site with a history of week-long link failures is useless. Sizing at a high quantile accepts that a small fraction of observations will arrive as corrections, and corrections are handled correctly by construction. The quantile is therefore a latency knob, not a correctness knob, which is the property that makes it safe to tune. Choosing 0.99 specifically is a default rather than a result; no sensitivity analysis is offered here.

#Why per channel rather than global

The justification is empirical: a satellite-linked wellhead and a wired unit in the same plant differ in lateness by orders of magnitude. A constant chosen for the satellite link makes every wired channel unnecessarily laggy. A constant chosen for the wired unit puts the satellite link permanently in correction, which is correct but wasteful, since it re-projects windows continuously.

This matches where the out-of-order stream processing literature arrived. Slack and punctuation-based approaches both make progress a per-source property rather than a global one [8] [9] [10], and the general treatment of unbounded, disordered streams assumes sources differ [11] [12]. The contribution here is not the idea of a per-source bound but the specific derivation from the source's own measured history, applied to industrial telemetry where the spread between sources is extreme.

#Duplicates, and why identity beats deduplication

Observation identity is derived from channel, event time and value. A redelivered observation therefore collapses onto the original with no lookup before insert and no constraint violation to catch, making ingestion idempotent under replay.

The alternative, which the first implementation used, is an explicit deduplication step holding recently seen observations and discarding matches. That approach has a window of its own, and the window is a second thing to size. When the two windows disagree, and they will, a duplicate that arrives after the deduplication window but within the reorder window is inserted twice, in order, with no error. Deriving identity from content removes the second window and therefore the possibility of disagreement.

The cost is that identity now depends on the value. Two genuinely distinct observations of the same channel at the same event time with the same value are indistinguishable, and are collapsed. For a sampled measurement this is what you want. For an event counter that legitimately records two identical events at one timestamp it is wrong, and such channels must carry a sequence number in the identity tuple.

#Staleness is a value, not an absence

Each channel declares an expected interval. When the elapsed time since the last observation exceeds a multiple of it, the value's state becomes stale.

Stale is returned as the last known value together with its observation time and an explicit flag. No read path returns the value alone. Industrial protocols already carry this information as a status code [13], and a pipeline that discards it is discarding a signal the field devices provided deliberately.

The reason this must be a value rather than an error or an absence is a matter of what the consumer can do. An absence forces every consumer to decide what absence means, and they will decide differently: a dashboard blanks the tile, an alarm rule treats it as unchanged, a rollup skips it, and the three now disagree about the same instant. A stale value with a timestamp lets each consumer apply its own tolerance against a shared fact. An operator deciding whether to dispatch an engineer needs to know the reading is forty minutes old. The same reading is fine for a monthly energy total and unacceptable for a trip decision, and only the consumer knows which it is.

#The interpolation question

The construction rules out three things that are commonly done and rarely labelled.

• Last value carried forward as a state semantic. It is a rendering choice and must be labelled as one at the point of rendering, not baked into the stored state.
• Interpolation across an absent interval without an explicit model the consumer opted into.
• Any read interface that cannot distinguish a measured zero from no observation.

The second deserves the argument, because it is the one people push back on. Interpolation is genuinely useful. A chart with gaps is harder to read than a chart without them, and a model that requires evenly spaced input needs evenly spaced input from somewhere.

The objection is not to interpolation but to interpolation that is indistinguishable from measurement downstream. A twin that closes the loop on physical equipment is part of a system carrying assurance requirements [14] [15], and in that setting the provenance of a value is part of the value. Consider a pressure reading interpolated across a six minute gap during which the actual pressure spiked and returned. The interpolated series is smooth, plausible, and wrong in exactly the interval that mattered. If the interval is labelled as inferred, an engineer reviewing the incident sees immediately that there is no evidence there. If it is not, the smooth line is evidence of something that was never observed.

Interpolation is a claim about what happened when nobody was looking. It can be a reasonable claim. It cannot be an unmarked one.

The practical form of this is that interpolation happens at the read boundary, on request, with the method named in the response, and never during ingestion. Storing interpolated values is what makes them indistinguishable later, because the next system to read the store has no way to tell which rows were measured.

#What deployment changed about the design

Two things came out of running this that were not in the original design and would not have been predicted from it.

The first is that operators asked for the observation time to be shown by default, not the staleness flag. The flag answers whether the value is trustworthy according to a threshold somebody else chose. The timestamp lets them apply their own judgement, which for people who know their own equipment is better information. The flag remains, because automated consumers need a threshold, but the human interface leads with the time.

The second is that the most valuable diagnostic turned out to be the per-channel lateness distribution itself, which the window derivation computes as a byproduct. A channel whose lateness distribution shifts is a channel whose link or gateway is degrading, and that shift is visible days before the link fails outright. The mechanism was built to size a window and ended up being the most useful piece of condition monitoring in the ingestion path, which is a reminder that the instrumentation a design needs internally is often worth exposing.

#The unresolved case: derived values across channels

Two channels feeding one computed value may sit at different points in their respective windows. A computed efficiency figure that divides a flow reading with a two second window by a pressure reading with a four hour window has no principled staleness state, and no honest way to report one.

Three answers were considered and none is satisfactory. Taking the maximum staleness across inputs is defensible but makes the derived value as stale as its worst input permanently, which for a mixed set of channels means always stale. Taking the minimum is indefensible. Computing only when all inputs are fresh means the derived value is frequently absent, which pushes the problem to consumers as an absence, precisely what the staleness design was meant to avoid.

This is the most consequential gap in the construction, because derived values are what operators actually read. Raw channel values are for engineers and for the pipeline. A control room looks at efficiencies, rates and margins, all of which are derived. A staleness model that is rigorous for raw channels and undefined for derived ones is rigorous in the wrong place.

#Limitations

Window adaptation lag. ω is estimated from history, so a channel whose lateness distribution changes spends a period with an incorrect window. Since the distribution shift is itself the useful diagnostic described above, an adaptation policy that reacts quickly would suppress the signal it should raise. No policy resolving that tension is given here.

Correction storms. A large backfill produces corrections across many windows simultaneously. The re-projection is correct and the downstream delta volume is not rate limited in the implementation described, so a backfill can saturate consumers that were sized for the live rate. Treating backfill as a distinct ingestion mode rather than as very late live data is probably the right answer and was not implemented.

Clock skew is detected, not corrected. Observations stamped in the future are accepted and flagged. No attempt is made to estimate and subtract a per-device offset, which is a well studied problem [4] and would materially improve the event-time fold for devices whose clocks drift predictably.

No published lateness distributions. The per-channel argument rests on measurements not reproduced here, and the qualitative table above is a characterisation rather than data. Publishing anonymised distributions across a range of site types would be the most useful contribution a follow-up could make, and would let others test both the quantile choice and the claim that per-channel derivation beats a well chosen global constant [16]. Until that exists, the central design decision in this report is supported by argument and operational experience rather than by evidence a reader can check.