From: Mei Lin, QA lead, Fernhill Garden Supply
"Our checkout repo has a test that goes red a few times a week and green when someone reruns it. People have started rerunning everything by habit. I exported the last runs from CI: commit, test, result and the UTC time.
Which tests are flaky, which are actually failing, and what should we change? I want a recommendation I can put in a ticket."
Your answer: the list of flaky and failing tests, the pattern behind the flaky one, and the fix.
A flaky test passes and fails on the same code. Teams start ignoring it, and then a real failure hides behind it. This case study works through a CI history the way you would at work: decide what counts as flaky, look for a pattern in when it fails, separate it from tests that are simply broken, and recommend a fix.
Each step adds to the same function, so your own code carries forward. Runs are text lines of commit, test, result and UTC time. The second history in the tests has a flaky test with no time pattern, so a rule that only works on the first history won't pass.
The hour is characters 11 and 12 of the timestamp.