The Label Problem in Industrial Predictive Maintenance

#Abstract

Supervised failure prediction on industrial assets assumes a labelled dataset that well-maintained equipment does not produce. This report characterises four distinct sources of label corruption, sets out three practical responses and the validation evidence each can support, ordered by the author's experience rather than by measurement on a common dataset, and argues that reporting a point accuracy for a failure model trained on a handful of events is misleading regardless of the value.

#The premise that fails

Given a dataset of asset states with failures marked, predicting failure is a conventional supervised problem. On operating industrial data that dataset does not exist, and the reasons are structural rather than a matter of data collection effort.

#Four sources of corruption

Rarity by design. A maintenance programme exists to prevent failures. A facility with an effective programme produces few positive examples, and those it produces are heterogeneous in mechanism.

Censored positives. A technician observes a developing fault and intervenes. The failure does not occur and the record shows routine maintenance. A positive example is recorded as a negative, and the rate of this is unknown.

Timestamp displacement. Work orders carry the date they were raised or closed rather than the onset of the condition. The displacement can be weeks, which matters when aligning a label to a sensor window.

Unstructured description. The actual failure mechanism is in free text written by many people over decades, with inconsistent vocabulary.

The second is the most damaging, because it is a systematic bias toward the negative class that is invisible in the data and correlated with exactly the conditions a model should detect.

#Why censored positives are the worst case

The four sources are not equally bad and the ordering is not obvious, so it is worth working through why the second dominates.

Rarity is a sample size problem. It is severe, it is well understood, and it is honest: a model trained on few positives has wide uncertainty and standard methods will say so. Timestamp displacement adds noise to the alignment between label and signal window, which degrades a model without systematically deceiving anyone about its quality. Unstructured description is a data extraction problem, tedious and tractable.

Censored positives are different in kind, because the mislabelling is not random. Consider the following illustration, with round numbers chosen for clarity rather than drawn from a site.

1                             recorded          actual
2failures that occurred          20               20
3faults caught and fixed          0               60   <- recorded as routine maintenance
4routine maintenance          1,000              940
5                             ------           ------
6positive rate                 2.0%             7.6%
Copy

Three quarters of the genuine positive class has been filed as the negative class. A model trained on the recorded column learns that the sensor signatures preceding a caught fault are normal, because that is what the labels say. Those signatures are precisely the ones the model exists to detect, since a fault a technician can catch by inspection is a fault with observable precursors.

The model is not merely undertrained on the positive class. It has been actively taught that the most detectable failures are normal operation.

The effect on evaluation is worse than the effect on training. Held-out data carries the same censoring, so a model that correctly flags a developing fault is scored as a false positive, and a model that ignores developing faults scores well. The measurement rewards the wrong behaviour, which means iterating on the metric moves the model away from what is wanted. Nothing internal to the dataset reveals this. It is visible only by talking to the people who did the intervening.

#What the label noise literature offers, and what it assumes

Learning with imperfect labels is a well developed field. The standard taxonomy separates noise completely at random, noise depending on the true class, and noise depending on the features [1], and methods exist to estimate a noise transition matrix from the data itself and identify likely mislabelled examples [2].

These methods are useful here and they do not solve the problem, for a reason worth being precise about. They generally assume the noise process is estimable from the observed data, which requires that some signal distinguishes mislabelled from correctly labelled examples within the dataset. Censoring by intervention does not satisfy that. The intervened cases were removed from the positive class by an outside process that also removed the outcome which would have identified them, and the resulting labels are internally consistent. A confident learning approach applied to this data will find the examples the labels disagree with the features about, which is not the same set.

What does help is treating the intervention record itself as a weak label source. A work order for unscheduled inspection following an operator report is evidence of a developing fault even where no failure followed, and combining several such imperfect sources with modelled agreement is the weak supervision approach [3]. This requires that intervention records exist and are distinguishable from scheduled work, which is a data availability question that differs by site and is frequently answered no.

#Why public benchmarks understate the problem

The widely used run-to-failure datasets in this area are simulated degradation trajectories, generated by running units to failure under controlled damage propagation [4]. They are valuable for comparing algorithms and they have properties operating data does not: every unit fails, onset is known exactly, no intervention occurs, and the failure mechanism is uniform.

A method validated on such a dataset has been shown to work on a problem where none of the four corruption sources exist. That is a real result about the method and it is not evidence that the method will work on plant data, and the gap between the two is where most industrial projects fail. The broader condition-based maintenance literature has long been clear that data quality and acquisition dominate this field in practice [5], which is easy to read past when the benchmark results look strong.

#Three responses

Change the target. Predict a measurable proxy with adequate positive examples, such as an operating parameter crossing a threshold or a derived health indicator changing. Less directly valuable and it produces a model that can be validated, which is the point.

Anomaly detection. Model normal operation and flag deviation. This requires far less labelled data and detects change rather than predicting failure. It generates false positives, which are tolerable only where review cost is low.

Reconstruct labels. Work with maintenance staff to rebuild failure histories from work orders, comments and recollection. Slow, unglamorous, and the highest value activity on such a project. That data work dominates practical machine learning effort is well documented [6] [7].

#Validation when positives are scarce

If the responses above are adopted, the question becomes what evidence a project can actually produce. The answer is less than people expect and more than nothing, and being clear about which is which prevents a project from promising evidence it cannot deliver.

A random train and test split is not available in any useful sense. With a handful of positives, a split either puts almost none in the test set or almost none in training, and repeated splits produce a variance so wide that the mean is uninformative. Leave-one-failure-out is the honest form, and its output is a small set of case outcomes rather than a metric.

Temporal splitting is mandatory regardless. Training on data that postdates the test period leaks information through maintenance actions taken in response to the very conditions being predicted, and given censoring, that leakage flows in the direction that flatters the model.

What can be produced credibly is a case series: for each historical failure, whether the model would have flagged it, how far in advance, and what else it flagged during the same period. That is not a metric and cannot be compared across projects, and it is the form of evidence a maintenance engineer can actually evaluate, because it is the form in which they already reason about equipment. A project that sets out to produce a case series rather than an accuracy figure tends to end up with something defensible.

Alongside it, the false positive burden should be reported in units of work rather than as a rate. Twenty alerts a month is meaningful to the people who must respond to them. A precision of 0.7 is not.

#Drift is structural here

Industrial equipment is maintained, modified and replaced, and operating conditions vary with season, product and demand. A model trained on last year describes a system that no longer exists in the same form, which is concept drift in its standard sense [8] [9].

Two features distinguish this setting from the usual drift discussion. The drift is often deliberate, since a maintenance action is an intentional change to the system being modelled, which means the times at which the model becomes stale are known in advance and recorded. A maintenance management system is therefore a drift signal, and few deployments use it as one. And the drift can be discontinuous: a replaced bearing does not degrade the model gradually, it invalidates the unit's history at a stroke.

Monitoring input distribution is therefore mandatory rather than advisable, and it is the leading indicator, because output quality metrics require ground truth that arrives late or never. In a setting where the ground truth for a thirty day prediction arrives thirty days later, and arrives censored, the input distribution is very nearly the only timely signal available.

#On reported accuracy

A model claiming to predict failure thirty days ahead, validated against a small number of historical events, carries very wide uncertainty on that claim. Reporting a single accuracy figure invites a decision the evidence does not support.

The computational science community has been direct that scaling from individually calibrated models to fleets requires uncertainty quantification that largely does not yet exist [10] [11]. The practical consequence adopted here is that a model output presented to an operator carries an interval, and a model that cannot produce one is not presented.

The original per-unit formulation in the aerospace structural life literature made its value proposition concrete in a way most industrial projects do not: replacing a fleet-average decision with a per-unit decision, measurable in inspection intervals [12]. Projects that cannot state their decision in those terms tend to struggle to demonstrate return.

#Limitations

The four corruption sources are drawn from work on a limited number of facilities and are not claimed to be exhaustive or to have representative frequencies. A quantitative study of censored positives, which would require access to maintenance records and staff recollection across several sites, would be the most useful follow-up and has not been performed.

No comparison of the three responses on a common dataset is offered, so the ordering reflects the author's experience rather than measurement [13].

The censoring illustration uses invented numbers. It shows the mechanism and it establishes nothing about magnitude. Whether the true censoring rate at a typical facility is five per cent or seventy five per cent is unknown to me, and the difference between those two changes whether this is a caveat or a reason not to attempt supervised failure prediction at all. That single number is the most valuable measurement anybody working in this area could publish.

The weak supervision suggestion is offered on structural grounds and has not been implemented on the sites this report draws on. Whether intervention records are separable from scheduled work in practice, and whether the resulting labels improve a model rather than adding a differently biased source, is untested here.