Nobody Needs an AI That Writes Tests. We Need One That Reads Failures.

Discover AnythingSignupWrite 184 readsNobody Needs an AI That Writes Tests. We Need One That Reads Failures.byAnil Reddy PutlacherubyAnil Reddy Putlacheru|@anilreddy2581Sr SDET at I am a SDET , Focused on AI And MLSubscribeJuly 24th, 2026TLDR Your browser does not support the audio element.Speed1xVoiceDr. One Ms. Hacker byAnil Reddy Putlacheru@anilreddy2581byAnil Reddy Putlacheru|@anilreddy2581Sr SDET at I am a SDET , Focused on AI And MLSubscribebyAnil Reddy Putlacheru|@anilreddy2581Sr SDET at I am a SDET , Focused on AI And MLSubscribeEvery AI-for-testing demo I have sat through follows the same script. Someone points a model at a requirements doc, it emits forty test cases, and the room nods. It's a good demo. It also solves a problem I've never actually had.Writing tests was never the bottleneck. I've worked on suites where the tests wrote themselves out of habit, one per bug, one per ticket, until there were thousands of them running every night. The bottleneck showed up the next morning, when sixty of them were red and someone had to decide what that meant.That decision is the job. And it's the part almost nobody is automating well.The morning triage taxRun the arithmetic on a mid-sized suite. Say two thousand tests, nightly, with a three percent failure rate. That's sixty red tests waiting when the team logs on. In my experience, four out of five of those aren't defects at all. They are a stale fixture, a timing window that closed, a device that did not finish rebooting, a data seed that drifted, a network hiccup in the middle of a retry.(Illustrative numbers. If the failure rate looks low to you, you have a bigger problem than this article can fix.)Someone still has to open all sixty. Ten minutes each, if the logs are decent. That's a full engineer-day, every day, spent sorting noise from signal. Teams call it triage and treat it like weather, something that happens to them. It's the most automatable workload in the entire pipeline, and it sits there untouched while everyone builds test generators.Hardware is where this gets uglyIf your tests run against pure software, you can at least assume the machine underneath you is honest. I've spent a good chunk of my career on integration testing where the system under test includes physical hardware, and that assumption dies on day one.A single red test on a device rig has, roughly, four candidate explanations:The code regressed.The firmware on the device changed, and the test is fine.The rig itself drifted. A cable, a calibration, a sensor, a thermal condition.The test is racing something physical and lost this time.Only the first one is a bug. The other three are why nobody trusts the suite by month six, and why teams start rerunning failures until they go green. That's the moment the suite stops being a safety net and becomes a ritual.A stack trace can't distinguish between those four. Neither can an LLM staring at a stack trace. Which brings me to the thing I actually want to argue.The agent is the last thing you buildEvery diagnostic agent I've seen fail, failed for the same reason. The pipeline underneath it had already thrown away the evidence.Typical CI keeps a stack trace, a truncated log, and a pass/fail bit, with seven-day retention. The device serial output, the screenshot at failure, the firmware version on the rig, the calibration timestamp, the last ten runs of that same test, the data seed hash: gone, or never captured, or sitting in four different systems with no shared key.You can't reason your way out of missing evidence. If a human engineer couldn't diagnose the failure from what your CI stores, a model can't either. It will just sound more sure of itself.So the first thing to build isn't the agent. It's the failure bundle: one object, one ID, everything a diagnosis could possibly need.failure_bundle:  run_id: nightly_1187  test_id: scan_pipeline.calibration_drift  status: failed  code_diff_since_last_pass: [commit_a1b, commit_c4d]  firmware_version: 3.