- From
- Tomás PereiraHead of Engineering
- To
- Platform team
- Subject
- Friday's unlock failures
Release 2.14.0 of rides-api went out at 16:02 on Friday. For the next 45 minutes about a third of unlocks failed with a 502, until Sofia rolled back to 2.13.2. We heard about it from riders, not from an alarm.
Before we ship weekend pricing again: find the commit that broke payments and undo it without rewriting main, deploy the fixed release so a broken build can never take traffic, and set up the alarm that would have paged us. Then write the incident report for the whole team.
You have the load balancer's requests and 5xx counts per five minutes for Friday afternoon, and the timeline Sofia put together.
Done looks like: v2.14.1 tagged on main, running on 3 ready pods behind a readiness probe, an alarm on the 5xx count that pages on-call, and a short blameless incident report.