Templates /
Noise Audit Workflow Template

Noise Audit Workflow Template

Use this workflow to determine the level of noise occurring within your decision-making framework.
1
Intro:
2
Enter the audit details
3
List the members of the audit team
4
Case materials:
5
Determine the criteria for selected cases
6
Prepare the case questionnaire
7
Schedule the prelaunch meeting
8
Send meeting invitation
9
Prelaunch:
10
Conduct the prelaunch meeting
11
Approval:
12
Study:
13
Administer questionnaire
14
Compile the questionnaire answers
15
Analysis:
16
Assess participants' responses
17
Determine corrective action
18
Generate a report on the study's findings
19
Approval:
20
Conclusion:
21
Review the audit process
22
Complete the audit
23
Sources:
24
Related Workflows:

Intro:

System noise  (or simply “noise”) is the unwanted and unexpected variability in judgments that should, ideally, be identical. 

Noise is an especially crucial concern in lottery systems – such as court cases, insurance claim estimates, patient diagnoses, etc. – where individuals with the same qualifications and knowledge are randomly assigned to a specific case. 

Noisy judgments can be time-consuming, costly, and have severe impacts on individual lives as well as organization trajectories. 

Based on research by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein, this workflow will guide you through conducting a noise audit on your own team. 

As Kahneman, et al advise, ask yourself the following question before you begin:

If the results of our simulation indicate a high level of noise, will people in the company accept that there is noise in the actual judgments of the unit?

If the answer to this question is anything but yes, the noise audit will be a waste of time and shouldn’t be undertaken. 

Source material: Kahneman, D., Olivier Sibony and Sunstein, C.R. (2021). Noise: A flaw in human judgment. New York: Little, Brown Spark.

Enter the audit details



List the members of the audit team

This task contains conditional logic. If you answer yes to the first question, then you will be provided fields to enter the consultant’s details. If you aren’t using an external consultant, then those fields will remain hidden.

Learn about using conditional logic here




Case materials:

The following tasks use several features that will save you time by automating different functions. 

The file upload feature allows users to update documents to individual workflow runs in order to keep a record of work done. 

Dynamic due dates allow you to link deadlines to specific tasks. Deadlines will be automatically calculated based on when certain tasks are completed. Read more about dynamic due dates here

Variables allow you to display “variable” information that’s unique to each workflow. They act as placeholders for specific details you’ve entered, such as a date or email address. Learn more about using variables here

The Send email widget creates pre-written emails within your workflows that can be sent with a click of a button. In this workflow, variables (date, location, email addresses, etc.) about the prelaunch meeting are automatically pre-populated in the email, so you don’t have to worry about re-entering that information again. 

Check out the video below for more details:

Learn more about using email automation in your workflows!

Determine the criteria for selected cases

Before you conduct the study, you need to outline the parameters that will be used to assess the level of noise in your department. 

There are no strict criteria for what information the cases should include, although they should provide enough detail for the participants to make a reasonable judgment

You may choose to provide a simplified list of details in hypothetical scenarios, or you may prefer to use detailed summaries of complex cases. You may also decide to use a combination of simplified and complex cases.

You will need to determine which approach will be most beneficial to your team. 




Prepare the case questionnaire

You should prepare a questionnaire for each case the participants will assess. The purpose of this questionnaire is to provide deeper insight into the decision-making process that led to each participant’s final judgment.

The questionnaires should only be administered once all cases have been judged.

The questionnaire should include the following:

  1. Open questions about what factors led to the participant making the judgment.
  2. A section for the participant to rank key facts of the case in order of importance (to the participant).
  3. Estimates of how the case in question relates to others in the same category. For example, in assessing an insurance claim, the participant should estimate how much below or above the average claim amount that case is. 

Schedule the prelaunch meeting

Before beginning the study, you need to hold a prelaunch meeting with your organization’s leadership.

As the project manager, you will need to present your audit strategy and plan for conducting the study to your organization’s leadership

It’s important that you’re prepared to discuss:

  • Possible outcomes of the study, including finding unacceptable system noise.
  • Any objections to the planned study.
  • Any adjustments or improvements that need to be made to the plan.
  • Leadership’s commitment to act on the results, whatever they may be.

Send meeting invitation

Send an invitation to all meeting attendees using the send email widget below. 

Remember to proof the email before sending it!

Prelaunch:

The tasks in this section utilize role assignments, approvals, and stop tasks

Role assignments allow you to dynamically assign individuals to a task based on their roles. 

In this workflow, the next task is assigned to the Project Manager. Whoever performs that role in each workflow run (regardless of whether there is a different Project Manager in each run) will be notified that they’re responsible for that task.

Approvals let you streamline any processes that require authorization. 

Once the prelaunch meeting is conducted, leadership will be notified that they need to approve the meeting’s results before the study can begin. 

Stop tasks are an incredibly powerful tool that can be used to control the order tasks are completed within a workflow. 

The stop task in this section prevents users from continuing the workflow until the approval task has been completed. 

Check out the videos below for more details on approvals and stop tasks:

Learn more about using approvals in your workflows!

Learn more about stop tasks and enforced order within your workflows!

Conduct the prelaunch meeting

Noise Audit Pre-Launch Meeting 

Date: {{form.When_will_the_meeting_take_place?}}

Location: {{form.Where_will_the_meeting_take_place?}}

Expected results: {{form.Describe_the_results_you_expect_to_find.}}

Acceptable variability: {{form.What_percentage_of_judgment_variability_is_acceptable?}}

Length of study: {{form.How_much_time_will_participants_be_given_to_assess_the_cases?}}

Factor assessed: {{form.What_attribute_will_be_used_for_assessment?}}

Questionnaire: {{form.Upload_the_questionnaire_draft}}

Test cases: {{form.Upload_the_sample_cases}}

Approval:

Will be submitted for approval:

  • Conduct the prelaunch meeting

    Will be submitted

Study:

Administer questionnaire

Guidelines for administering the study

  1. Do not use “noise” or “noisy” in relation to people and/or study.
    1. “The organization is interested in how [decision-makers] reach their conclusions.”
  2. Assure participants that their answers will be anonymized.
  3. Anonymize the answers.
  4. Stress that the participants will not face consequences for their answers; the purpose of the study is simply fact-finding. 
  5. An accurate audit requires 100% participation by the team/department. Arrange for a set block of time for participants to focus on the study.
  6. The assessment should be completed by all participants at the same time but not in such close physical proximity that they might influence each other’s answers.
  7. Participants should have access to the audit team throughout the study to answer any questions.



Compile the questionnaire answers

Analysis:

Assess participants’ responses

Assessing the study’s findings and determining any conclusions should take approximately 3 months to complete. 

Determine corrective action

Generate a report on the study’s findings

{{form.Was_the_presence_of_noise_detected_in_the_study?}}

{{form.Describe_the_level_and_pattern_of_noise_present_in_the_study.}}

{{form.Was_the_presence_of_bias_detected_in_the_study?}}

{{form.Is_the_noise_level_within_the_acceptable_variability_parameters?}}

{{form.What_is_the_identified_source_of_judgment_variability?}}

{{form.What_is_the_identified_cause_of_judgment_variability?}}

{{form.How_will_this_variability_be_resolved?}}

Approval:

Will be submitted for approval:

  • Generate a report on the study’s findings

    Will be submitted

Conclusion:

You can use Process Street Automations as well as Zapier integrations to export workflow data into your other apps. 

For example, you could export data into Google sheets, send a message through Slack, or even launch another workflow run. 

Check out the video below for more on Automations:

Learn more about using Automations in your workflows!

Review the audit process

Use this task to assess the audit process itself, record any factors that impacted the audit, and what changes should be made for future audits. 

Complete the audit

Congrats! You’re almost done!

All that’s left to do is fill in the final details and set the date for the next noise audit for your team. 

Remember: You can use Automations to trigger your next audit’s workflow to run on the correct date!

Sources:

Take control of your workflows today.