A short run of success: what a few good results cannot establish
In Zurich’s financial services, insurance and tech sectors, and at the European headquarters of international groups, a successful pilot can quickly become an internal benchmark. The management question is specific: what should be checked before turning a short run of success into a rollout decision?
What the literature establishes
Amos Tversky and Daniel Kahneman (1971, Psychological Bulletin) describe the belief that a small sample should faithfully reflect the properties of the population from which it is drawn. Their survey of researchers trained in statistics shows that this expectation can lead people to overestimate the strength and replicability of findings from small samples. For managers, the important point is not that a few observations are worthless, but that they can appear far more representative than they actually are.
The mechanism that makes the run convincing
Amos Tversky and Daniel Kahneman (1974, Science) describe, among other phenomena, insensitivity to sample size and mistaken intuitions about regression towards the mean. An exceptionally favourable performance can be followed by a more ordinary result without any change in the method or the team. If a pilot is selected precisely because its outcome is exceptional, attributing its entire performance to a lasting capability becomes particularly questionable.
The premature conclusion
One might conclude that managers should always wait for more data before deciding. Yet these studies establish no universal number of observations that is sufficient to generalise a result. The decision also depends on the cost of waiting, the consequences of an error and the diversity of situations to which the pilot will be extended. (our executive and employee training programmes)
What the available evidence does not resolve
These classic articles do not directly assess rollout decisions in Zurich organisations, nor do they demonstrate that a particular procedure improves their profitability. In practice, counting cases is not enough: results involving the same client, team or period may share the same underlying causes. An apparently large volume therefore does not necessarily provide a set of independent observations.
A practical check in Zurich
As part of SHR’s decision-making programme, a Zurich team could spend four weeks reviewing rollout proposals based on a pilot, whether for a banking journey, claims processing, a software feature or a headquarters process. For each proposal, record the number of cases observed, the target population, the conditions shared by those cases and a benchmark result established in comparable situations. The measure would be the proportion of proposals containing all four elements before the decision, using the total number of proposals reviewed as the denominator. This check assesses the traceability of the reasoning, not the future quality of the decision: a complete file can still lead to a poor choice. To go further: explore the Decision-making training in Zurich in the canton of Zurich, or browse our executive and employee training programmes in Switzerland.
In pictures: Decision-making in Zurich
- decision-making
- Zurich
- research
- training zurich