Predictive tools: why one error is not enough to disqualify them

Should a manager abandon a predictive tool after a conspicuous error when its comparative performance remains favourable? For SHR — Swiss Human Resources’ “Decision-making” programme, this question calls for a distinction between trust, acceptance and measured quality.

What the literature establishes

Berkeley J. Dietvorst, Joseph P. Simmons and Cade Massey (2015, Journal of Experimental Psychology: General) show in forecasting experiments that participants may avoid an algorithm after seeing it make errors, even when it outperforms the human forecaster. Observing its imperfections can therefore discourage the use of a comparatively better-performing tool. The important point is not that the tool is infallible, but that its errors and human errors do not necessarily receive the same treatment.

Control as a condition of acceptance

Berkeley J. Dietvorst, Joseph P. Simmons and Cade Massey (2018, Management Science) show that allowing participants to modify an algorithmic forecast, even slightly, can increase their willingness to use it. Limited scope for intervention can thus make a tool acceptable despite its acknowledged imperfections. This finding concerns the conditions of use; it does not demonstrate that every human adjustment improves the forecast.

The premature conclusion

One might conclude that simply allowing managers to adjust outputs will produce better decisions. The distinction is essential: greater acceptance of a tool and improved performance are separate outcomes. An adjustment may incorporate relevant information missing from the model, but it may also worsen an estimate; comparison is needed rather than assumption. (our executive and employee training programmes)

What the experiments do not guarantee

These experimental studies do not provide general validation of the predictive systems used in organisations. They do not establish that a tool performing well on average remains reliable for every population, in every situation or after a change in context. Particularly in federal administration and public health, legality, traceability and the severity of errors cannot be reduced to an average measure of accuracy.

A practical check in Berne

In Berne, an exercise within SHR’s “Decision-making” programme could address a recurring forecast with no direct effect on individual rights: caseloads in federal administration, logistical needs in public health, service interventions in telecoms or component demand in precision manufacturing. Before each outcome is observed, record an independent human estimate, the tool’s unadjusted forecast and any adjusted forecast, together with the reason for the adjustment. Over a period specified in advance, compare their mean absolute errors on exactly the same cases, also retaining a record of errors deemed critical under a predefined criterion. This check would help establish whether adjustments add measurable value, rather than letting the latest conspicuous error determine the decision. To go further: explore the Decision-making training in Bern in the canton of Bern, or browse our executive and employee training programmes in Switzerland.

In pictures: Decision-making in Bern

• decision-making
• Bern
• research
• training bern