AI on the job: understanding a process or recognising its explanation?

An assistant can explain an approval workflow, an insurance claims procedure or an escalation rule without its reader subsequently knowing how to apply it. The managerial question is precise: what evidence of understanding should be required before a team is authorised to use a process newly explained by AI?

What the literature establishes

Leonid Rozenblit and Frank Keil (2002, Cognitive Science) show that people overestimate their understanding of mechanisms and revise their assessments when asked to produce a detailed explanation. Recognising an intelligible description is therefore not equivalent to reconstructing the causal relationships it presents. These experiments do not concern AI, but they suggest distinguishing the feeling of understanding from understanding that has actually been demonstrated.

Access to explanations can distort self-assessment

Matthew Fisher, Mariel K. Goddu and Frank C. Keil (2015, Journal of Experimental Psychology: General) show that searching for explanations on the Internet can increase people's estimates of their own knowledge, including in domains they have not searched. The proposed mechanism is a confusion between access to external knowledge and possessing that knowledge internally. A conversational assistant might encourage a similar confusion, but this extension remains a hypothesis rather than a finding of the study.

The premature conclusion

One might conclude that employees should be asked to explain everything without assistance before being entrusted with a process. That would confuse operational understanding with exhaustive memorisation: consulting a procedure is normally part of the work. The relevant question is instead whether someone can identify a missing condition, explain a critical dependency and recognise when escalation becomes necessary. (our executive and employee training programmes)

What the evidence cannot guarantee

These studies assess neither mastery of financial procedures nor the effects of workplace training in generative AI. They cannot establish a universal threshold of understanding sufficient for independent work. In practice, a satisfaction questionnaire or the question “is that clear?” does not directly test the ability to apply a process.

A practical check in Zurich

As part of SHR's IA on the job programme, a relevant exercise for financial services, insurance, tech and the European headquarters of international groups in Zurich would be to select a non-sensitive internal process and prepare a routine case and an exception case with its business owner. After consulting the AI-generated explanation, each participant would state their confidence level, then work through the cases with the official procedure available, explaining the decision point and any need for escalation. The primary measure would be the proportion of answers given with confidence that contain an error at a predefined decision point, as validated by the business owner. This result would not establish a causal effect of AI; it would identify where a feeling of mastery is insufficient grounds for authorising independent work. To go further: explore the AI on the Job training in Zurich in the canton of Zurich, or browse our executive and employee training programmes in Switzerland.

In pictures: AI on the Job in Zurich

• ai on the job
• Zurich
• research
• training zurich