The Setup: Real App, Real Clicks, No Mocking
Zephyr Cloud's AI Platform is a desktop app where teams collaborate with AI specialists in channels. The bug-hunting agent drives this real, signed-in app using Playwright over Chrome DevTools Protocol (CDP). No stripped-down test build, no mocked APIs. It interacts with the same UI a human user would.
The agent runs hourly, exploring routes, opening dialogs, filling forms, and watching how the app responds. When it finds something off, it writes a proper bug report with reproduction steps and a screenshot, then files it as a GitHub issue.
A second agent picks up that issue, reproduces the bug for itself, patches it, captures before-and-after proof, and opens a pull request. A human still reviews and merges — the agents just handle the tedious middle bit.
The Honesty Filter: Why Noise Kills Trust
Getting an agent to find bugs is easy. Getting it to be honest about what it found is the hard part. An eager agent reports everything — including working-as-designed behavior, test data artifacts, and uncertain findings. That noise destroys trust.
The hunter classifies every finding into one of five categories:
- Bug: genuinely broken
- Expected but bad UX: works, but shouldn't
- Environment or data issue: setup, not the product
- Test gap: missing coverage, not a live bug
- Inconclusive: could not confirm it
It attaches a confidence level (high, medium, low). It only files an issue for high-confidence, reproducible product bugs. Everything else stays in a log. The rule: better to say "I could not confirm this" than to cry wolf.
Real Bugs Found: Silent and Subtle
The agent has filed real issues — the kind of quiet, low-key bugs that slip past when you're focused on shipping features. Examples:
- A chat channel that sits on "Fetching message history..." forever. No error, no timeout — just stuck.
- A GitHub token field that appeared to save but didn't, because the token format was off. No feedback given.
- A settings page that loads the home screen instead, leaving home buttons unclickable.
- A chat pane that shrinks to a one-pixel sliver when all side panels are opened.
None of these throw errors. None would fail a normal test. They are just wrong in quiet ways, and having something patiently check every hour catches them early.
The Full Circle: Agent Catches and Fixes Its Own Bug
The hunter found that the workflow editor had no unsaved-changes guard. Editing a workflow, navigating away, and changes vanished silently. It filed an issue. The team added a guard.
Weeks later, the hunter returned and found a new problem: that guard now fired a spurious "Unsaved changes" dialog after every successful save, even when nothing was unsaved. It filed that too.
Then the fixer picked it up, reproduced it, traced it to a save that navigated before React had re-rendered, patched it, and opened the PR. The agent flagged the missing feature, we built it, the same agent caught the bug that feature introduced, and another agent fixed it. A full circle, barely touched by a human.
The Hard Rule: Reproduce Before You Fix
The fixer is not allowed to touch code until it has reproduced the bug itself. No repro, no PR. This is the difference between a fix and a guess. Often the honest outcome is "I could not reproduce this" or "this needs a human." In those cases, it does not open a PR.
When it does open a PR, it includes a before screenshot showing the real broken state and an after screenshot showing it resolved. The before shot must be genuine — an agent will happily produce a convincing "before" from an already-fixed branch if you let it. That's locked down.
What Does Not Work: The Honest Breakdown
The webview wall. CDP only sees inside the app's webview. It cannot see the OS around it: native file pickers, system login windows, OS notifications, keychain prompts. A whole category of bugs lives just outside its reach.
Silent failures cause hallucinations. The hardest problems are not loud errors. They are quiet failures where something breaks without saying so, and the agent narrates a success that didn't happen. Most engineering effort went into making failure loud so the agent notices and admits it.
Not a CI gate. These runs drive a real, running, signed-in app. That's wonderful for realism but useless as a fast check on every commit. It's a separate, slower tier.
What to Steal
The mechanics are easy: Playwright over CDP, a schedule, a couple of prompts. The hard part is the honesty. Make the agent classify findings, attach confidence levels, reproduce before fixing, and allow "I could not" as a valid answer.
An agent that files ten real bugs and admits to three it was unsure about is worth far more than one that files thirteen and makes you check every one. The bugs were never the impressive bit — building something you can trust to be honest about them was.
