Noise Audit Workflow Template

System noise  (or simply "noise") is the unwanted and unexpected variability in judgments that should, ideally, be identical. 

Noise is an especially crucial concern in lottery systems - such as court cases, insurance claim estimates, patient diagnoses, etc. - where individuals with the same qualifications and knowledge are randomly assigned to a specific case. 

Noisy judgments can be time-consuming, costly, and have severe impacts on individual lives as well as organization trajectories. 

Based on research by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein, this workflow will guide you through conducting a noise audit on your own team. 

As Kahneman, et al advise, ask yourself the following question before you begin:

If the results of our simulation indicate a high level of noise, will people in the company accept that there is noise in the actual judgments of the unit?

If the answer to this question is anything but yes, the noise audit will be a waste of time and shouldn't be undertaken. 

Source material: Kahneman, D., Olivier Sibony and Sunstein, C.R. (2021). Noise: A flaw in human judgment. New York: Little, Brown Spark.

This task contains conditional logic. If you answer yes to the first question, then you will be provided fields to enter the consultant's details. If you aren't using an external consultant, then those fields will remain hidden.

Learn about using conditional logic here. 

The following tasks use several features that will save you time by automating different functions. 

The file upload feature allows users to update documents to individual workflow runs in order to keep a record of work done. 

Dynamic due dates allow you to link deadlines to specific tasks. Deadlines will be automatically calculated based on when certain tasks are completed. Read more about dynamic due dates here. 

Variables allow you to display "variable" information that's unique to each workflow. They act as placeholders for specific details you've entered, such as a date or email address. Learn more about using variables here. 

The Send email widget creates pre-written emails within your workflows that can be sent with a click of a button. In this workflow, variables (date, location, email addresses, etc.) about the prelaunch meeting are automatically pre-populated in the email, so you don't have to worry about re-entering that information again. 

Check out the video below for more details:

Before you conduct the study, you need to outline the parameters that will be used to assess the level of noise in your department. 

There are no strict criteria for what information the cases should include, although they should provide enough detail for the participants to make a reasonable judgment. 

You may choose to provide a simplified list of details in hypothetical scenarios, or you may prefer to use detailed summaries of complex cases. You may also decide to use a combination of simplified and complex cases.

You will need to determine which approach will be most beneficial to your team. 

You should prepare a questionnaire for each case the participants will assess. The purpose of this questionnaire is to provide deeper insight into the decision-making process that led to each participant's final judgment.

The questionnaires should only be administered once all cases have been judged.

The questionnaire should include the following:

1. Open questions about what factors led to the participant making the judgment.
2. A section for the participant to rank key facts of the case in order of importance (to the participant).
3. Estimates of how the case in question relates to others in the same category. For example, in assessing an insurance claim, the participant should estimate how much below or above the average claim amount that case is. 

Before beginning the study, you need to hold a prelaunch meeting with your organization's leadership.

As the project manager, you will need to present your audit strategy and plan for conducting the study to your organization's leadership. 

It's important that you're prepared to discuss:

• Possible outcomes of the study, including finding unacceptable system noise.
• Any objections to the planned study.
• Any adjustments or improvements that need to be made to the plan.
• Leadership's commitment to act on the results, whatever they may be.

Send an invitation to all meeting attendees using the send email widget below. 

Remember to proof the email before sending it!

The tasks in this section utilize role assignments, approvals, and stop tasks. 

Role assignments allow you to dynamically assign individuals to a task based on their roles. 

In this workflow, the next task is assigned to the Project Manager. Whoever performs that role in each workflow run (regardless of whether there is a different Project Manager in each run) will be notified that they're responsible for that task.

Approvals let you streamline any processes that require authorization. 

Once the prelaunch meeting is conducted, leadership will be notified that they need to approve the meeting's results before the study can begin. 

Stop tasks are an incredibly powerful tool that can be used to control the order tasks are completed within a workflow. 

The stop task in this section prevents users from continuing the workflow until the approval task has been completed. 

Check out the videos below for more details on approvals and stop tasks:

Noise Audit Pre-Launch Meeting 

Date: {{form.When_will_the_meeting_take_place?}}

Location: {{form.Where_will_the_meeting_take_place?}}

Expected results: {{form.Describe_the_results_you_expect_to_find.}}

Acceptable variability: {{form.What_percentage_of_judgment_variability_is_acceptable?}}

Length of study: {{form.How_much_time_will_participants_be_given_to_assess_the_cases?}}

Factor assessed: {{form.What_attribute_will_be_used_for_assessment?}}

Questionnaire: {{form.Upload_the_questionnaire_draft}}

Test cases: {{form.Upload_the_sample_cases}}

Guidelines for administering the study

1. Do not use "noise" or "noisy" in relation to people and/or study.
   1. "The organization is interested in how [decision-makers] reach their conclusions."
2. Assure participants that their answers will be anonymized.
3. Anonymize the answers.
4. Stress that the participants will not face consequences for their answers; the purpose of the study is simply fact-finding. 
5. An accurate audit requires 100% participation by the team/department. Arrange for a set block of time for participants to focus on the study.
6. The assessment should be completed by all participants at the same time but not in such close physical proximity that they might influence each other's answers.
7. Participants should have access to the audit team throughout the study to answer any questions.

Assessing the study's findings and determining any conclusions should take approximately 3 months to complete. 

{{form.Was_the_presence_of_noise_detected_in_the_study?}}

{{form.Describe_the_level_and_pattern_of_noise_present_in_the_study.}}

{{form.Was_the_presence_of_bias_detected_in_the_study?}}

{{form.Is_the_noise_level_within_the_acceptable_variability_parameters?}}

{{form.What_is_the_identified_source_of_judgment_variability?}}

{{form.What_is_the_identified_cause_of_judgment_variability?}}

{{form.How_will_this_variability_be_resolved?}}

You can use Process Street Automations as well as Zapier integrations to export workflow data into your other apps. 

For example, you could export data into Google sheets, send a message through Slack, or even launch another workflow run. 

Check out the video below for more on Automations:

Use this task to assess the audit process itself, record any factors that impacted the audit, and what changes should be made for future audits. 

Congrats! You're almost done!

All that's left to do is fill in the final details and set the date for the next noise audit for your team. 

Remember: You can use Automations to trigger your next audit's workflow to run on the correct date!

Source material

Goodman, B. (2021). "Daniel Kahneman Says Noise Is Wrecking Your Judgment. Here’s Why, and What to Do About It". Barron's Magazine.

‌Kahneman, D., Sibony, O. and Sunstein, C.R. (2021). "Bias Is a Big Problem. But So Is 'Noise'." The New York Times.

Kahneman, D., Sibony, O. and Sunstein, C.R. (2021). Noise: A flaw in human judgment. New York: Little, Brown Spark.

Kahneman, D., Rosenfield, A.M., Gandhi, L. and Blaser, T. (2016). "Noise: How to Overcome the High, Hidden Cost of Inconsistent Decision Making." Harvard Business Review. 

Kotsonis, S., Change, J. and Chakrabarti, M. (2021). "How To Cut Through The “Noise” That Hinders Human Judgment." WBUR.

Matias, J.N. (2017). "Bias and Noise: Daniel Kahneman on Errors in Decision-Making." J. Nathan Matias.

McCulloch, A. (2021). "Daniel Kahneman: how “noise” damages decisions". Personnel Today.

Nesterak, E. (2021). "A Conversation with Daniel Kahneman About 'Noise'." Behavioral Scientist.

Newell, B. (2021). "Daniel Kahneman on 'noise' – the flaw in human judgement harder to detect than cognitive bias." The Conversation. 

Related workflows

• ADKAR Model Change Management Process Checklist
• Bridges Transition Model Process Checklist
• DACI Decision-Making Meeting Framework Checklist
• FMEA Template: Failure Mode and Effects Analysis
• Kotter’s Change Management Model Process Checklist
• Kubler-Ross Change Curve Process Checklist
• Lewin’s Change Management Model Process Checklist

• McKinsey 7-S Model Checklist Process

• Nudge Theory Change Management Model Process Checklist
• Process for Improving a Process
• Prioritization Matrix Checklist Template
• Satir Change Management Model Process Checklist
• Standard Operating Procedure (SOP) Template Structure
• SWOT Analysis Template
• Unconscious Bias Training Guide