System noise (or simply "noise") is the unwanted and unexpected variability in judgments that should, ideally, be identical.
Noise is an especially crucial concern in lottery systems - such as court cases, insurance claim estimates, patient diagnoses, etc. - where individuals with the same qualifications and knowledge are randomly assigned to a specific case.
Noisy judgments can be time-consuming, costly, and have severe impacts on individual lives as well as organization trajectories.
Based on research by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein, this workflow will guide you through conducting a noise audit on your own team.
As Kahneman, et al advise, ask yourself the following question before you begin:
If the results of our simulation indicate a high level of noise, will people in the company accept that there is noise in the actual judgments of the unit?
If the answer to this question is anything but yes, the noise audit will be a waste of time and shouldn't be undertaken.
Source material: Kahneman, D., Olivier Sibony and Sunstein, C.R. (2021). Noise: A flaw in human judgment. New York: Little, Brown Spark.