Incident Checklist

Create DataDog Incident

Add info to Slack Notification

Add Detection Method

Add Root Cause

Add Impact entry

Create tasks that need to be performed

Create DataDog Notebook with more data

Add important Notes to Incident

Update Incident Status

Create and add Post Mortem document

Create DataDog Incident

Go to DataDog and create incident

https://app.datadoghq.com/incidents/new

1

Choose severity
2

Input short title
3

Set "slack-packhelp-dev-incidents" as additional notification

Do you have or can create link to any Incident indicator (graph, user session, log link...)

If you have URL to a place where incident indicators are visible, or you can create any graph that shows data related to incident, then create it and add it to incident:

1

Add URL, or Graph to "Attached Signals"

Add info to Slack Notification

1

Copy URL to incident from DataDog (example: https://app.datadoghq.com/incidents/2)
2

Go to https://packhelp.slack.com/archives/CLH5E4QMT
3

Paste this link in Slack Notification Thread of this Incident
4

Mention @backend, @frontend, @devteam, @devops, or particular developer that could potentially know solution to issue

Add Detection Method

Add Root Cause

Add Impact entry

Create tasks that need to be performed

Depending on the issue, create and update list of tasks that need to be performed. Examples:

Update pricings
Migrate WordPress DB
Asses severity/impact of the issue on users
Find root cause
Revert changes
Prepare temporary fix
Prepare proper fix
...

Create DataDog Notebook with more data

If possible, create, and attach DataDog Notebook with graphs and data that shows the scope of Incident.

Add important Notes to Incident

Whenever new information is gathered, and is important, add it as a Note to Incident Timeline. Example of important information:

Root cause
Temporary fix
Proper fix
Severity/Impact changed
More information/data found
...

Update Incident Status

Update Incident status to proper one:

`ACTIVE`: Issue is still in full effect
`STABLE`: Issue was temporarily adrressed
`RESOLVED`: Proper fix was implemented and deployed
`COMPLETED`: Post Mortem document in Notion was created and added to Incident. Additional info and tasks were added and performed.