The QA loop that runs itself

How a discovery agent walks user journeys across web and mobile, finds bugs, proves them with generated tests, and hands confirmed defects to a development agent. No one prompts any of it.

Jun 29, 2026

TL;DR: We built a QA system where a discovery agent walks real user journeys — across web, mobile, and multiple countries — and files GitHub issues when something looks wrong. A two-tier pipeline then writes a test, runs it against a live build, and only confirms the bug if the test actually fails. Confirmed bugs go straight to a development agent. The fix for a noisy QA agent isn't a better model; it's a verification loop.

Nobody on our team kicks off the QA run anymore.

A discovery agent does. It walks the real user journeys — sign-up, deposit, withdrawal, transfer — on both the web app and the mobile app, routing through VPN exit nodes in several countries. When something looks off, it opens a GitHub issue. From there, a pipeline takes over: it writes a test that reproduces the problem, runs that test against a real build, and only keeps the bug if the test actually fails. The confirmed ones get handed to a development agent. No human prompts any of it.

The first version was a disaster.

We pointed an agent at the product and told it to find bugs. It found plenty. Most weren’t real: flaky selectors, flows it invented that no real user walks, edge cases that only happen if you hold the phone upside down. Within a week, the team stopped reading the issues. A QA signal nobody trusts is worse than no signal, because it still costs money to produce, and now it costs attention to ignore.

The gap between those two versions is the whole point of this post.

Stop prompting. Engineer the loop.

Addy Osmani has a clean line for what changed: loop engineering is “replacing yourself as the person who prompts the agent — you design the system that does it instead.” The LangChain team frames it as a stack of loops: an agent loop that calls tools until it’s done, wrapped in a verification loop that checks the output, wrapped in an event-driven loop that fires the whole thing on a signal.

Our QA loop is exactly that stack made concrete. The discovery agent is the agent loop. The reproduce-the-bug step is the verification loop. GitHub’s issue events are the event-driven loop.

The thing that took us from “noisy and ignored” to “trusted and unattended” wasn’t a better model. It was getting the verification loop right.

The journey is fixed. The search is not.

One distinction runs underneath all of it.

A user journey is deterministic. Sign-up is sign-up. A real user picks a country, enters an email, verifies a code, sets a password — the same way every time. That path doesn’t change run to run, and you don’t want it to.

Finding bugs inside that journey is the opposite. You want the agent to think, vary its inputs, submit the form twice, paste an emoji where a name goes, run the whole thing from a different country. That part is non-deterministic on purpose. It’s where exploration earns its keep.

So you pin the first and free the second. The journey is the guardrail you hand the agent. The exploration is the engine you let run inside it.

Most disappointing QA agents blur the two: they either script the exploration until it finds nothing surprising, or they let the agent wander with no idea what a real user actually does — which is how you get the upside-down-phone bugs.

Discovery: an agent, not a script

The discovery agent reads its journeys as skills — plain markdown files, one per journey, that spell out the steps and the rules. It drives the web app with Playwright and the mobile app with a mobile UI runner. It can switch its VPN exit node, so the same journey runs from different countries.

When a journey behaves wrong, it opens a GitHub issue describing what it saw.

That last capability earns its keep more than I expected. Some bugs only surface in specific regions: geo-gated content, a currency that renders wrong in one market, a third-party integration that only times out from one location. Running each journey from several countries surfaces those. A single-region run never would.

The issue it opens is a candidate, nothing more, which brings us to the part that actually matters.

Two tiers: cheap before expensive

The pipeline wakes on a new issue and runs in two tiers.

Triage runs first, on a cheap model. It does two boring, high-value things: checks whether the issue is in scope (out-of-scope requests get closed, not worked), and classifies it — bug, feature, chore, docs — then labels it. That’s it. Triage never writes code and never opens a PR.

Validation runs only on issues that are triaged as bugs, and only here do you spend a strong model. It reads the journey skills and the source to find the exact on-screen text (it never guesses a label), then writes a test that reproduces the report — a Playwright test for web, a mobile flow for mobile. Then it does the thing that makes the whole loop trustworthy: it runs that test against a real build.

Pre-condition check — worth copying separately. Before running the bug test, the pipeline checks whether the app can launch and log in at all. If that fails, the run is an environment problem, not a confirmed bug, and it’s reported as that. A flaky environment never gets to masquerade as a real defect.

The verdict is mechanical:

Test fails on the live build → bug reproduced → pipeline opens a PR containing the regression test and labels the issue bug-confirmed
Test passes → bug cannot be reproduced → issue is closed with that note
Bug can’t be checked through the UI (API-only or network fault) → labelled needs-manual-test, not guessed at

A bug isn’t real until a generated test reproduces it. That single rule is what stopped the flood.

Show the wire

A journey skill is plain markdown the agent loads before it acts:

---
name: sign-up
description: Create a new account end-to-end — choose country, sign up with a test
  inbox, verify the OTP, set a password, complete the profile.
---

# Sign-up journey
1. Ask only for the country; generate a fresh test-inbox address (never a real user's email).
2. Submit the form; read the OTP from the test inbox directly.
3. Set a password; fill the profile with a randomised name and date of birth.
4. Confirm the account screen appears.

The loop is wired from GitHub’s own events — no scheduler, no human trigger:

# triage — wakes when the discovery agent opens an issue
on:
  issues:
    types: [opened]

# validate — wakes only after triage has labelled it a bug
on:
  issues:
    types: [labeled]   # runs when the label is "bug"

And the verification logic, in plain terms:

run the regression test on the live build:
  passed  → bug NOT reproduced → label "cannot-reproduce", close the issue
  failed  → bug reproduced     → open a PR with the test, label "bug-confirmed"
not UI-testable (API/network)  → label "needs-manual-test"

Playwright is swappable for Cypress on the web; the mobile runner for Appium. The shape holds when you change the box.

The two tiers are the cost lever

Using a cheap model on triage and a strong model on validation is not an implementation detail. It’s what makes the agent affordable to run.

The cheapest false positive is the one you reject before spending a strong model on it. Triage kills the noise and the out-of-scope requests for almost nothing. The expensive tier — reading source, writing a test, building the app, running it on a real device — fires only on issues that earned it.

In a handful of journeys, that ratio is irrelevant. At hundreds, it’s the difference between a QA agent you can fund and one you quietly switch off.

Onboard the agent like a new hire

Notice that the journeys, the triage rules, and the test standards are all data — markdown the agent reads before acting, not logic baked into code.

The cleanest way to think about that is onboarding. You’d never drop a new QA hire in front of the product and say “find bugs.” You’d walk them through how sign-up works, what done looks like, which flows are sacred. Then you’d let them poke at the edges. An agent is no different.

Leave a journey undocumented, and it fills the gap by guessing — inconsistent one run, off-brand the next.

That’s why “too many bugs” was never a model problem for us. It was a missing briefing and a missing verification loop.

The handoff, and where humans stay

A confirmed bug doesn’t sit waiting. The bug-confirmed label and the regression-test PR signal a development agent to implement the fix on the same branch, so the test ships with the code that makes it pass. That’s the verification loop feeding the next loop — the return arrow that reaches back in.

But the loop is bounded on purpose. The fix only starts when a maintainer with write access comments to proceed. The PR still gets merged by a human. Out-of-scope issues auto-close.

The agent owns the mechanical span between those gates. We own the gates.

Addy’s caveat is worth keeping pinned: “your job is to ship code you confirmed works.” An autonomous loop nobody understands is just a faster way to pile up problems you can’t explain.