From: Jonas Weber, on-call lead, Tidepool Freight
"Since this morning's deploy, every tracking Pod restarts and shipment lookups fail. The release has the customs fix we promised a customer for today, so I'd rather not roll back unless we must. Keep the evidence in incident/, find out what broke and fix it, and give me a three-line summary for the postmortem."
Done when: the crash output, the warnings, the bad revision and a working response are saved, the release runs, and incident/summary.txt states revision, cause and fix.
An incident worked end to end in one cluster: keep the logs and events, tie the failure to a revision, decide between rolling back and fixing forward, prove the service answers, and write a summary another engineer can verify.
The first evidence of the incident.