The previous version was a set of commands inside one project. The current one is a plugin: it installs into any repository and any agent harness, reads the project profile, and after every session edits its own instructions.
// the whole session, before the textOne session from the command to the report, in compressed time. The rest of the article is how it works inside.
~/acme-shop · claude
>
session files
docs/test-sessions/2026-09-17-PROJ-4821/
1-plan.md…
2-test-design.md…
4-manual-result.md…
99-report.md…
roles
👔QA Managercoordinateswaiting
📝qa-analysttest designwaiting
🔍qa-manualrun on the envwaiting
⚙️qa-automatorautotestwaiting
A demo, not a screen recording: timings are compressed and the task is made up. Everything else is the plugin's real order of steps.
claude plugin marketplace add codecube01/qa-cube
claude plugin install qa-cube@qa-cube
# Codex, opencode and any other agent — see the README
/qa-setup # interview: where tasks live, where cases live, which envs, which runner, which browser
/qa <task link> # the session starts
/qa <the same task> # retest: the next round lands in the same session folder
/qa-day # what I tested today; --days 7 — for the week
The process lives in one place and improves once. Everything project-specific — tracker, test management, environments, autotests, browser — the project declares in its own .claude/qa-profile.md.
In March I described a multi-agent QA system on Claude Code: five agents, a parallel run, findings exchanged through discoveries.md, all of it living in one project's .claude/. The system worked — and that is exactly why it had to be rewritten.What works in one project cannot be handed to a second one without copying a hundred and fifty kilobytes of prompts by hand. The previous version lived in .claude/commands/ and .claude/agents/ of a specific repository: every new project meant copying files, editing paths manually, and versions drifting apart within a week.This article is about what came out of the rewrite. Briefly: three commands, four roles, parallelism dropped on purpose, everything project-specific moved into a single file that the engine reads but never writes itself — and the engine itself is no longer tied to a single agent either.
The table of changes
Previous version
Current
files in every project's .claude/
a plugin from the marketplace, updated by command
Claude Code only
Claude Code, Codex, opencode — and any agent with files and a shell, through adapters
one executor per round; browser and API are the same job
discoveries.md to exchange findings between parallel agents
gone, and not planned: there is no one to exchange with
project knowledge dissolved into agent prompts
a project profile plus a versioned profile contract
one report per round
one report per session, updated by every round
a retro «when there is time»
mandatory step 7, with an address for every edit and numbers from transcripts
prompts grew with no ceiling
a size budget on every file that is read in full; enforced in CI
lessons are read by whoever does the work
the manager reads the catalog's table of contents; the catalog itself goes to a separate reviewer with no session context
language nailed to the project
the engine is English, output is any language from the profile
The main idea of the previous article — «architecture beats one clever prompt» — still holds. What changed is where the boundaries run: not between API and UI, but between the process and the project.
Since the first edition of this article
what changed from v0.14 (September) to v0.19 (November)The engine is no longer about one harness. The instructions name an action by role — «launch the qa-manual role», «ask the user» — and a harness adapter says how that is done in Claude Code, Codex, opencode or any other agent.The browser is an adapter, Playwriter by default. Your own Chrome with your own logins, driven from the shell — and therefore from any harness. Claude in Chrome and Playwright MCP are supported too.The lesson catalogs outgrew their reader. The largest is about 220 KB. The manager reads only the table of contents, and a separate reviewer with no session context checks the whole catalog against the plan, the brief and the report.The retro counts the numbers nobody remembers: time, peak context, tool errors and compactions for every agent — from the transcripts on disk.Engine lessons take effect from the next session even without an engine clone: the project now has its own layer of engine rules, .claude/qa-cube-feedback.md.Environment state moved out of CLAUDE.md into a separate snapshot, environment-state.md, and the session registry became a table and nothing more.Three times the code, four times the tests: 15 scripts and 237 tests against 6 and 66.
02// architecture
One manager, three executors
The QA Manager is your main agent session. The rest are subagents it launches and waits for. Each has its own context, its own toolset and its own area of responsibility.
The executors run in sequence, and that is a deliberate choice rather than a simplification. The manual run's result is the automator's input; the analyst's cases are what the manual run follows. There is nothing to parallelise here, and the file scheme, the running numbering and the acceptance step are all built on «one executor at a time».Only one thing runs in parallel — the catalog reviewers: they touch nothing on the environment, read files and return a list, so they do not get in each other's way.
Nine steps of the pipeline
/qa is a procedure, not a request to «test it well».
0
profile and harnessharness adapter, capabilities, degradations, language, contract version, the project's layer of engine rules
1
the task and its commentslinked tasks, session mode, environment, a verdict on requirement quality
2
session foldernew, or a continuation of an existing one; monthly archive cleanup by script
3
planscenarios by priority, each with its expected business effect; a reviewer checks the plan against the catalog
3.5
test designthe analyst expands scenarios into cases (only when test-cases: upfront)
4
manual runpreflight → brief → brief review → launch the executor → accept the result
5
automationthe round's status list goes to the chat first, then the decision: what to automate, what to fold it into, run it until green
6
the 99-report.mdone per session, updated by every round; a reviewer checks it against the report catalog
6.5
deliver the result to the chat — before the retrothe report sits in a file the user cannot see; the retro is a long series of edits
7
retronumbers from transcripts, edits to their addresses immediately, in the same pass
8
wrap-upsession registry, environment snapshot, tracker comment, a summary of retro edits, the action block
9
the session is still openwhatever the user says after delivery — a defect, or a gap in the rules
why 6.5 existsThe step appeared after a session wrote the report, announced «now updating the knowledge base and the retro», and went quiet for several minutes of edits. So the result is delivered before the cleanup — and the wrap-up then repeats it in full under a «Test results» heading: the whole retro lies between 6.5 and the wrap-up, and people read the last screen.
03// agents
What each one does
Four roles, split not by tool but by stage: who writes the expectation, who checks the fact, who commits the result to code.
👔 QA Manager — coordinates, does not execute
Reads the task with its comments, pulls linked tasks into scope, plans the round, writes briefs, accepts results, writes the report, runs the retro. The only role that asks the user anything at all: one or two questions right at the start — and only if the plan falls apart without them.Two exceptions where the manager does touch the system: a quick fact-check to sharpen the plan — and a re-check with its own eyes when the user disputes a fact in the report. An objection is not verified by re-reading your own report.The previous article admitted honestly: «the manager still wants to do it itself; only repeating the rules helps». Repetition did not cure it. What cured it was a table of «red flags» at the end of SKILL.md: twelve thoughts that arrive right before a known rule is broken — «let me just check this myself, it's faster», «the MR diff will show what to test», «the report is written — retro first, the user can wait». A rule read at step 0 does not survive a waiting user and a shortcut at hand; a thought caught at the moment of decision does.
📝 qa-analyst — the paper role
Expands the plan's scenarios into test cases from the requirements: before the run, without seeing the environment, without reading the code. The point in one sentence: if a case is written from a requirement, a mismatch between fact and expectation reads as a product defect rather than a broken case. A case written from the code legitimises the implementation instead of the requirement.Its most valuable output is not the cases but the list of requirements that did not turn into one: not observable from the outside, contradicting another requirement, with no task behind the implementation. That list goes into the report as «not covered», with reasons.
🔍 qa-manual — checks the feature on the environment
Checks the business effect, not the response code: «200 OK» is not a test result. Frontend tasks are checked exclusively through the real UI with the network under observation; imitating frontend requests with curl does not count as a check and is not accepted.Writes the result immediately after each scenario, with no buffering — a rule inherited from the previous version: the context runs out before the run does. The verdict on each requirement is one of four values: pass / fail / partial / not covered. «Partial» comes with an account of which half passed; «not covered» always comes with a reason.Every frontend finding candidate gets exactly one screenshot on disk — at the moment of confirmation, a close-up of the area with the symptom, lossless. The tool's usual screenshot is compressed to fit an image budget, and the very pixel it was taken for cannot be read on it. A backend finding needs no screenshot at all.
⚙️ qa-automator — turns evidence into an autotest
Not documentation, not assumptions — the run's actual result: real endpoints, request bodies, observed responses. The order of preference is strict: a test already covers it → write nothing; a test exists for the same area → add a step to it; and only then a separate test. A small feature on top of an existing flow is covered by a step in the shared test, because the main win is reusing the setup.The cost of automation is now counted, not guessed: it includes the fixed cost of the host class — provisioning, per-test hooks — not just the new lines. And the question «shall we automate?» is asked only after the round's status list has gone to the chat as a separate message: a person decides on the autotest already knowing what was found.If the feature is broken, the test must fail honestly rather than work around the problem with soft assertions.
the shared rule for executorsNo executor ever asks the user a question. Ever. It acts, makes mistakes, corrects itself.A blocker is no reason to stop: an API error → diagnose by trace id, no test data → create it through the API, a missing setting → set it.An irreversible step the user allowed in the chat is a user's click or a manager's call — never an executor's.
04// profile
Portability rests on a single file
The engine knows nothing about your project, and it should not. Everything project-specific is declared in .claude/qa-profile.md — committed to the project repository, so a colleague gets a working /qa along with the clone.
The profile header — a declaration of capabilities
Every none has a documented degradation — described in the engine; the profile neither invents nor overrides it.
Capability = none
What changes
tracker
input is only the user's description; the session mode is asked for; no tracker comment is prepared
tms
cases become md files in docs/test-cases/; «mark as automated» becomes a line in the case file
test-cases
cases are not kept at all: the analyst is not launched, the test links to the task
environments
checks run against a local build
logs
diagnosis from response bodies only, with no trace id — and that is not the executor's fault
autotests
step 5 is skipped entirely, and the report says so
browser
UI scenarios do not enter the plan; a purely frontend task blocks the session with an explicit message
secrets
the session needs no accounts; a scenario that does need one yields a blocker — accounts are never invented
knowledge
product findings from the retro are stored next to the sessions
engine-clone
the retro does not edit the engine, engine lessons go into the project's rule layer — the normal state of a marketplace plugin
A missing key reads as none — profiles on older contract versions keep working with safe degradations. The single exception is test-cases: its absence reads as inline, otherwise a contract update would silently switch cases off in every project written earlier.The harness is never written into the profile. The profile is a team file in git, and colleagues on one project sit in different agents. The harness is detected from its own tool list, not from a name in the system prompt: a wrapper around another harness exposes the inner one's tools, and those are what the engine will call. It is recorded only in the session passport — so the retro and /qa-day know what the round ran in.
The profile contract and its versions
The profile carries a version (contract-version); the contract carries a version history. The session compares the two and, if the profile is behind, does not block: it says so in the first line, before any other call, names what is new, and carries on with degradations.The last two contract versions are about where things live, not about new keys:
v6 — two rule layers, no duplicates. A lesson true for any project but written without an engine clone lives in .claude/qa-cube-feedback.md next to the profile, rather than being rephrased into the profile «so it takes effect sooner». The profile holds only what is true for this project and no other.
v7 — a thin CLAUDE.md, state included. What the environment holds right now lives in the <sessions>/environment-state.md snapshot, where every fact is dated and replaced rather than appended. How runs ended lives in the session registry. CLAUDE.md keeps only pointers.
The question «does this edit require anything from project profiles» is asked on every engine edit. A missed version bump is the most expensive retro mistake: the engine has several consumers, and their profiles drift apart silently. The reverse is also true: an edit that does not touch the profile does not bump the contract version — otherwise every project gets a false «your profile is behind» alarm.The profile deliberately has no schema and no validator. Its readers are agents; what matters is that it answers questions, not that it parses.
/qa-setup — reconnaissance, not a questionnaire
Onboarding first studies the repository: CLAUDE.md and skills, test directories and runner configs, docs/, traces of past test sessions, mentions of a tracker and TMS in configs, connected MCP servers and CLIs on PATH, logging configs. For each capability there is a verdict: answer found (naming the file), hypothesis (one clarifying question), unknown.What is found with confidence does not become a question — it is written straight into the profile, and the user sees all of it in the final review. Only hypotheses and unknowns are asked about, and the hypothesis is offered first, marked «(Recommended)».Then the access channels: if a channel is missing, the user picks in order of quality (MCP → CLI → generate a project REST skill), followed by a live check: one read call per declared tool. The browser is chosen the same way: Playwriter is recommended, Claude in Chrome and Playwright MCP are offered if connected — and the chosen one is checked live. If it fails, TODO: not verified — <error> goes into the profile and onboarding is not blocked.A repeat /qa-setup is also an update: it pulls the new engine version, reconciles the project's rule layer with what has already landed upstream, and offers to give the rest to the community. The session does none of this itself: its job is to say in one line that the engine is behind.
secretsOnly the source and the variable names go into the profile, never the values.The user fills in the values; the engine does not ask for secrets in the chat and does not write them on the user's behalf.The skeleton file is checked with git check-ignore, and if it is not ignored, a line is appended to .gitignore.
05// session
A folder per session, a report per session, rounds inside
The number in a file name is the order of its appearance, which is the order of the work. Numbering runs through the folder and is never reset: the second round continues from eight.
docs/test-sessions/
├── README.md # registry: a table, one line per session, verdict in a phrase
├── environment-state.md # environment snapshot: what it holds now, every fact dated
├── side-findings.md # bugs outside the scope of any task
├── 2026-09-14_acme-412-coupon-double/
│ ├── 0-session.md # passport: tasks, mode, environment, harness, round log
│ │ # + the raw task text and comments, verbatim
│ ├── 1-plan.md # scenarios by priority, each with a business effect
│ ├── 2-manual-task.md # brief for the manual tester
│ ├── 3-manual-result.md # what actually happened: requests, responses, trace ids
│ ├── 4-automator-task.md # brief for automation
│ ├── 5-automator-result.md # the test, the run log, a link to the case
│ ├── 6-retro.md # what went wrong, where each rule landed, round numbers
│ ├── 7-plan-2.md # round 2 continues the same numbering
│ ├── 8-manual-task-2.md
│ ├── 9-manual-result-2.md
│ ├── 10-retro-2.md
│ ├── screenshots/ # one per frontend finding
│ └── 99-report.md # one report per session, updated by every round
└── 2026-08/ # closed months move into archives
Two files stand outside the numbering and are updated in place rather than duplicated: 0-session.md, the entry point, and 99-report.md, the report. Ninety-nine keeps the report last in the listing whatever the number of rounds, and removes the question «which report is current»: there is always exactly one.The next file number comes from a script, not the agent: session.py next-file — one call per file. Two files on the same number are rejected by the checker. The next free finding ID is a script too, session.py max-id, and it reads not only the reports but also the side-findings registry: an ID issued only there was once issued a second time as «the next one».
A retest is the same session
/qa on the same task does not open a second folder. The folder is renamed to the current date (returning from the month archive to the root), and every link to the old name across the repository is rewritten. New files continue the numbering with a round suffix, and task comments that appeared since last time are appended to the «raw source».The first thing round 2+ checks is whether the previous one finished: is there a retro, was automation decided, were its report's promises kept. Whatever was left undone is done in this round. And it reads the triage of past findings in every thread they were sent to, not only in its own task's thread. A precedent: the frontend part of the findings was pasted into a neighbouring frontend task, the customer's verdicts on five of them — two withdrawn, three «fixed» — sat there, and the round did not plan to check the claimed fixes, learning about them only from its own report reviewer.The report is not recreated: its sections are brought to the current state. A finding whose status changed moves between sections rather than being relabelled in place: a fixed one goes to the «Fixed» block under the table, one that broke again returns to the table under its own ID marked «regression R2» — a new ID would cut the history of one defect in two. The round's delta goes on top as a «round history» block. Nothing from previous rounds is erased silently: a fact that has stopped being true changes status, it does not disappear.
The raw source is the session's only raw layer
Everything read from the tracker at step 1 goes into 0-session.md verbatim: the full text of the task, of linked tasks, and all comments, with no paraphrase and no cuts. After that the manager, the executors and the next round all read the requirements from there instead of going back to the tracker.
Cleanup is a script too
Closed months move into archives with session.py sweep: it moves the folders and rewrites every link to them — in the project map, the profile, the knowledge base, the cases, the registries. mv breaks them all silently. A precedent: 8 folders moved in a second, and 20+ links across six files were fixed by hand in two passes. A link that leads nowhere is fixed by where its target went, not by its prefix: a prefix replacement keeps pointing at folders that were renamed along the way.
06// report
The report is written for someone who never opened the session
Findings are phrased in business language on purpose: the table is read by a developer who has no session, no files and no run history.
**Source:** ACME-412 · **Current verdict (round 2, 2026-09-14):** bugs — one finding is worth it
**Task goal:** not met — resubmission still gives a double discount
**Method:** API + browser, logs by trace id
## Confirmed (matches FR/AC)
- **FR-1.** ✓ The coupon reduces the cart total by its face value (round 2; checked on three carts)
- **FR-3.** ✓ An expired coupon is rejected, and the message names the expiry date
## Findings
| # · Priority | Requirement | Description and risk |
|---|---|---|
| **B-17 · Critical** | FR-2 (spec §4.1) | **A buyer who submits the order twice gets the discount
twice — the shop loses the difference.** The spec requires rejecting a repeat application; the total
fell from 4,200 to 3,400 on resubmission. Anyone who double-clicks pays less than the order is worth,
and the audit log records one application instead of two.
*Tech: POST /cart/coupon is not idempotent; trace 8f31c2, 2026-09-14* |
Fixed: B-15 · Major — the expiry message showed the date in UTC; confirmed in round 2.
### Findings with no violated clause
| # · Priority | Requirement | Description and risk |
|---|---|---|
| **B-18 · Minor** | «FR-4 is silent on who sees the status change» | **The operator does not see that
the order changed status.** … **Needs an analyst's decision: should the operator see it.** |
## Not covered (with reasons)
| What | Reason | Round |
|---|---|---|
| Stacking a coupon with a loyalty discount | no loyalty accounts on the environment | 2 |
A fragment of 99-report.md
Under the verdict line sits a separate task goal line: the user story in one phrase — met, not met, not confirmed, and why. It never collapses into a requirement count: nine greens out of ten and an unmet goal is a normal and important outcome.The order inside the cell is mandatory, 3–5 lines and no more than ~850 characters — the limit is the same for Critical and for Minor:
the first sentence — bold, in business language, with no endpoint or field names (the test: would someone who has never opened the code understand it);
evidence with numbers — what the spec requires against what came out;
the risk in user terms, not «the data does not match»;
«needs an analyst's decision» — if it does, then explicitly and in bold;
the technical cause on the last line, in italics: *Tech: …*.
The full mechanics — codes, bodies, chronology, every measurement point — live in *-result*.md; the report links to them rather than repeating them.
Not everything wrong is a finding against a requirement
In the first edition the report had a single findings table. Now there are three places with distinct meanings, and a finding goes where what exactly was violated puts it:
Section
What goes there
Findings
a specific clause of the requirements is violated
Findings with no violated clause
the spec is silent or contradicts itself; a known defect blocked the check; a «pass» that holds only on today's data
Closed by a development decision
development decided in the comments: «done differently», «skipped on purpose» — not a finding and not a line above
Awaiting an analyst's decision
the verdict rests on someone else's decision
All of this used to sit in one table, and a table copied whole into the tracker mixed «you have a bug» with «you have a hole in the spec». A developer handed that mix would start arguing with the requirement in the defect's thread.A finding's title has a mandatory shape, Area \ what happens, under what condition — because the title is exactly what the tracker searches for duplicates by. A precedent: a Critical went out with a 144-character title that welded three defects together, while the tracker had had its own title convention for six months — and in the end neither the finding nor its duplicates could be found.
Before it is recorded — cross-check the tracker
A finding is cross-checked against the tracker before it reaches the report: search by the defect's keyword, not only inside its own task and not only in its comments — the same defect is sometimes filed in a neighbouring project. An already-filed one is presented as «confirmation of a known problem: reproduces on the environment as of <date>, no new action needed».
and separately — search among closed tasksBefore writing «this does not exist and there is no task for it», check whether a Done task covers exactly that scope: the wording changes from «this needs to be filed» to «the work was closed as done and it is not in the product» — and that is a very different conversation with the team.A precedent: a backend task for list filters and sorting had been closed six months earlier, and not a single filter had appeared in the contract.
ready-made bug reports — only on requestIf the tracker is read-only, the session no longer writes bug report files by default and does not offer them. A person reads the report first; if they ask, they get one file per defect, with the finding ID inside the body and no QA-repository paths.
07// delivery
The status list and the action block
The first thing a person sees is the round on a single screen. The last thing the session prints is what is required of them.
Round 2 on ACME-412, stage3. Verdict: bugs — one finding is worth it.
🟢 FR-1 — the coupon reduces the cart total by its face value
🟢 FR-3 — an expired coupon is rejected, the message carries the expiry date
🟡 FR-4 — the order status changes, but the operator cannot see it
⚪ Coupon stacked with loyalty — no loyalty accounts on the environment
⚪ Behaviour on payment rollback — needs a day of worker wait time
…findings and a ready-to-paste tracker comment…
⚠️ Ask the analyst whether the operator should see the status change (FR-4): the spec is silent
💡 Create a loyalty account on stage3 and the next round will close coupon stacking
💡 Two carts named eval-* were left on the environment
Greens are listed individually up to five, then collapse into a counter: a screenful of green buries the two lines people actually read for. In round 2 and later, a changed verdict carries the transition — «🟢 FR-5 — was 🔴, fixed» — which is the reason to read a retest at all.
icons never reach the files99-report.md is copied into the tracker whole, where an emoji fares about as well as HTML inside a cell.A line must read without its icon — a terminal may not render it, and a reader may not tell red from green.
In the action block ⚠️ means «I cannot close this without you» and 💡 means «worth doing, safe to ignore». Every line states an action, not a problem: «Ask the analyst whether the rule applies to the finding: the spec is silent», not «the spec is silent».⚠️ is rare by rule — zero or one line is a normal round, two is already a lot — and is allowed for exactly two things: a verdict blocked on someone else's decision, and access that must be fixed before the round closes. If everything is urgent, nothing is; and a red line that turns out to be optional costs the credibility of every red line after it.Routine gets no line at all. The first edition had an example here: «⚠️ Paste the comment into the task — the tracker token is read-only». That example became the precedent: a person pastes the comment after every round and knows it anyway, and a line repeated every round teaches people to skip the whole block — along with the one line it exists for.Subagents do not print an action block at all: they never address the user — so ⚠️ stays rare by construction rather than by discipline.
08// retro
The engine edits itself — and that is a pipeline step
The retro is mandatory. Its output is a table of «observation → rule → where it landed», and the edits are made immediately, in the same pass.The inputs are the «executor retro» sections from the results (executors are required to write them), the analyst's retro from its final summary, plus the manager's own view: where it got stuck, what it violated, what information was missing, what it planned badly itself.
Numbers nobody remembers
From inside a session you cannot see what it cost. So the retro starts with a script: transcripts.py reads the transcripts on disk and prints, for every agent in the round, its time, peak context, tool errors, refusals, retries after a failure and context compactions. The table goes into the retro file as is, and every flag the script raises gets an answer: either a retro line with an address, or one line of «fine, because…» («navigate ×6 — waited for a deploy, per the brief»). A flag skipped silently leaves the numbers on the page and not a single conclusion drawn from them.The precedent the script was written for: over a week of sessions it showed managers at 400–750 thousand tokens of context and one compaction in the middle of a session — invisible from inside.
the enginethe process: roles, acceptance, formats, degradations, lesson catalogs, agent instructions — with a clone, straight into the files; without one, into the project's rule layer
the profile and local skillstool pitfalls, access, code maps — what is true only for this project
the project knowledge basewhat was learned about the product: API contracts, feature behaviour, limits
the contract versiona mandatory question on every engine edit
If you are unsure whether something is engine or profile: the rule would hold in any project → engine; it names a specific tool, environment or class → profile. Every rule lives in exactly one layer: a «just in case» copy in the neighbouring layer is a duplicate that will drift.
What to do when the engine is not yours
This is where the previous and the current version diverge most. The retro used to edit files in the same repository — because the engine was part of that repository. Now, for most users, the engine is an installed plugin: a marketplace copy with no .git and a version cache that the next plugin update will overwrite. An edit to such an address disappears without an error, while the version bump is reported as though the lesson had landed.So step 7 checks the address first: the directory exists, it contains .git, and the path is not inside the harness's install cache. If the check fails the lesson is redirected, not discarded: it is written as a line in .claude/qa-cube-feedback.md next to the profile — in the form it would take in the engine, tagged with the plugin version it was written under.In the first edition such a lesson simply piled up for better times. Now every session reads the open lines of that file at step 0 as additions to the engine — the lesson works from the very next session, with no clone and no waiting for a release. Lines are never deleted there: a lesson that reaches the engine moves into a «Closed» block with a version — the only place where the user sees their observation land.Nothing is sent anywhere on its own. The session reports in one line that the engine is behind upstream; the update, the reconciliation of the rule layer and the offer to send open lines upstream are done by /qa-setup, not by the session. The user publishes, not the engine.
With a clone — by checklist
If there is an engine clone, edits go straight into its files — but following references/engine-edits.md, read in full before the first edit: first a search for a twin (maybe the rule already exists and needs extending rather than a second one), at the end a CHANGELOG line, the validator and a fresh-eyed diff review. The mechanical half is held by check_engine_edits.py — ratchets against HEAD: a bump without a CHANGELOG entry, an edit without a bump, an entry with no line in the table of contents, a link to a section that does not exist, a file that outgrew its budget.
Anonymisation — a rule enforced by code
The engine is published publicly, so its files cannot contain a task id from someone else's tracker, an environment host, a service name, a company or an account. The precedent is written as mechanics; the address is dropped:
✗ not allowedPrecedent ACME-303: round 1 checked the path once and got a pass
✓ allowedprecedent: round 1 checked the path once and got a pass, landing in the only working state
The value of a lesson is in its mechanics; a task number is customer data, and nobody gave permission to publish it. The rule is defended at three levels: an instruction in the engine itself (the only level that reaches sessions in other people's projects), git hooks over the commit content, file names, message and branch name — and CI as the last line, because a hook can be bypassed with --no-verify and CI cannot. The definition of a «leak» lives in one script shared by all three levels, and it now catches more than task numbers: hosts outside the allow-list, emails, IPs, home paths, random UUIDs, secret shapes.
09// corpus
The economics of context: a budget, a table of contents and a reviewer
The previous article recorded honestly that «prompt size ≠ quality». The current version turned that observation into mechanics — and then ran into the lesson catalogs outgrowing their reader.SKILL.md is the procedure: what a step produces, which subagent it launches, what acceptance requires. Its size is capped at 71,500 characters (~19 thousand tokens), and CI enforces it. It currently sits at 66,300. The ceiling has moved both ways: once the procedure was squeezed from 70 to 60 thousand characters by moving step lessons into catalogs, and the last raise was 10% — where the file hit its budget. Every file that is read in full now has its own budget — catalogs and agent instructions too: the agent pays for such a file in every session, including the ones where it does not apply.
The whole instruction corpus is about 774 KB across 5,250 lines in 34 files; the first edition had 332 KB across 2,880 lines. But far from all of it is read per session: that was the point of the split.
The catalog as a table of contents and a reviewer
In the first edition the lesson catalogs were read in full, once per session. That worked while the planning catalog was small. Now it is about 220 KB, the brief catalog about 150, the report catalog about 95: reading one of those in full means giving away a third of the context before the first scenario.The solution comes in two halves:
The catalog's table of contents is the rules. One line per situation shape: the trigger and what the plan (brief, report) does. The manager reads only the table of contents and opens one entry for each line that fires.
The whole catalog is read by a reviewer. As soon as 1-plan.md is written, the manager launches a general subagent with no session context: the catalog path, the session file paths and the task «read the catalog in full, entry bodies included, then the files; return the entries whose situation this round is in and the file does not account for — no more than eight, and only those that would change the verdict, the budget or an irreversible step». Steps 4 and 6 reuse the same trick for the brief and the report; the brief reviewer also returns every unfilled template slot.
Why a reviewer rather than reading it yourself: the catalog's value is in the scenario shapes you did not think of. An author who read the whole catalog for them pays for every entry at every following step. A reader with no stake reads once and leaves. The executor's brief is written once, and an unforeseen situation cannot be added to it afterwards — so the executor launches only after the review.
Each of the three big catalogs is larger than the whole procedure. The manager pays only for the table of contents; the body is read by the reviewer — once, with no session context.
Catalog
When it is read
By whom
harness/<adapter>.md
step 0, before the first action the adapter answers for
on a trigger: cleanup, round 2+, legacy scheme; an objection after delivery
manager
The last two rows are a direct saving: most sessions never reach an objection after delivery and never touch the archives, so they do not pay for those lessons. Every lesson is closed by the precedent that paid for it — without one, the next reader will delete the rule as inconvenient. The catalogs now hold 424 precedents; in mid-September there were 183.The same logic applies to the profile: subfiles are read selectively, each with a TL;DR header (the fast path on the first screen, the pitfalls below), and in the executor's brief the manager lists exactly what to read — it built the plan and knows that better than the executor. A backend task does not need browser techniques.
and a separate rule about the project's CLAUDE.mdIt is loaded into every session and every subagent, so it keeps the map of the project, while QA specifics and environment state move into profile subfiles and environment-state.md.A precedent: 79% of the file turned out to be a log of 18 runs.
10// guards
What code checks now, instead of an agent's eye
Rules written in prose hold as long as they do not get in the agent's way. Everything that can be checked mechanically moved into scripts the pipeline runs itself.
Script
What it takes off the agent
check_session.py
not a single secret value in session files; as many separators in a table row as in the header; a finding cell no longer than ~850 characters; every brief scenario present as a heading in the result; two files on the same number
session.py
the next file number and the next finding ID; archiving with link rewriting; a check that every link to a session folder leads somewhere; screenshot names by finding ID
transcripts.py
round numbers for the retro: time, context, errors, compactions per agent
qa_day.py
a digest of sessions for a day or a week across all projects
check_engine_edits.py
ratchets on engine edits: bump, CHANGELOG, tables of contents, section links, budgets
check_identifiers.py
leaks of real project identifiers — in files, commits, PRs, history
The pipeline runs check_session.py at steps 4 and 5 over the executor's result — before the manager reads it — and at step 6 over the finished report before delivery.The secrets half matters most: the rule was repeated in four instructions precisely because the instruction was the only thing enforcing it — while session folders are committed to the customer's repository and travel into backups. It catches issued accounts, bearer values, JWTs, private key blocks, cloud and forge tokens, and one-time links: a magic link is an account too, even if the executor requested it itself. The masked forms the engine requires — <TOKEN>, ***, $VAR — stay silent.
237
tests for the scripts
57
paired tests for the session checker
8
behavioural evals
$4.06
a full eval pass
The checker's tests are paired by design. A checker of someone else's text breaks in two ways and only one of them is visible: it can miss a token — or fire on correct work and be switched off a week later. So every «must catch» case has a twin «must stay silent», and one test runs the entire engine corpus (the largest available body of correct prose, full of masked credentials and wide tables) and demands silence. It has already caught a real false positive — a phrase from a lesson catalog — and that is how the «letters and digits» requirement entered the rule.The tests for qa_day.py and session.py are about parsing, and parsing breaks quietly. A parser that stops recognising a file prints a shorter digest — and a short digest looks exactly like a quiet day.
Evals: the rule is not only written, it fires
CI has checked structure for a while — file names, frontmatter, the budget, the absence of identifiers. But structure says nothing about whether a rule actually fires. Hence a set of evals: six gate cases, one per invariant, and two experiments.
Case
The rule it defends
profile-missing
the session does not start without a profile and sends you to /qa-setup instead of improvising
degradations-declared
a profile of all none is respected: no autotest, no case, no promised comment
language-ru
language: ru drives the output even though every instruction before the agent is English
session-continuation
the second round lands in the existing folder and updates the same 99-report.md
engine-clone-guard
an engine-clone pointing at a plugin installation is rejected, and the lesson goes into the project's rule layer
feedback-layer
two layers, no duplicates: a universal lesson goes into the rule layer, a project one into the profile; an open line works from the next session
plan-shapes, brief-shapes
experiments, not gates: a task with planted scenario shapes, one grader per shape — to compare two ways of reading a catalog
The runs themselves are deliberately not wired into CI: each one is a real agent session on a personal token, and the judge is non-deterministic. So the evals are a pre-release gate that a human runs. The full pass is measured: $4.06 and about ten minutes for six cases, two runs each, with a «no plugin» base branch.What does run for free on every push is what needs no model: a test checks that each case's rule is still written — and in the file read at its step — and that every «file.md, section «…»» link between engine files leads to an existing section. That is exactly the regression audits and text moves produce, and it needs no model.
an honest caveat worth holding while reading a green boardSome of the cases are «knowledge» cases: they ask the engine to name a rule rather than to execute it.Those rules live at the beginning or the end of a long procedure, and a run that gets that far costs three times as much and dies on timeout about as often as it arrives.A knowledge case proves the rule is readable and reachable; it does not prove it is obeyed under load.
11// principles
The principles it all rests on
Ten rules, each one paid for by a precedent.
01 · The coordinator does not executeThe manager thinks and delegates. There are exactly two exceptions, both named explicitly — and one of them exists because re-checking a user's objection by re-reading your own report is pointless.
02 · Executors never ask questionsEver. They act, make mistakes, correct themselves. Only the coordinator asks, and only at the very start.
03 · Check the business effect, not the response code«200 OK» is not a test result. After action X, did Y change: the balance, the status, the record.
04 · A case is written from a requirementNot from an observation. Then a mismatch reads as a product defect. A case written from the code legitimises the implementation.
05 · An autotest comes from run evidenceNot from documentation. The automator knows the endpoint works and what response to expect.
06 · A test mirrors realityIf the feature is broken, the test must fail honestly rather than work around the problem with soft assertions.
07 · Nothing disappears silentlyA fact that has stopped being true changes status. A withdrawn finding burns its ID and gets a paragraph, «withdrawn on re-check <date>», with reasoning.
08 · A lesson without an address is not a lessonAnd if a patch to an instruction repeats for the third time, fold it into the main text instead of adding a fourth phrasing.
09 · Every rule is closed by a precedentWithout one, the next reader will delete the rule as inconvenient.
10 · Authors don't check their own work against the catalogThe shapes you did not think of will be found by a reader with no stake and no session context — who reads the catalog once instead of you.
12// when-not
When you do not need this
Worth saying plainly, because the setup costs an interview and every session costs tokens.
A project with no tracker and no environment. The engine will work from a free-form description against a local build, but most of what it is good at — reading a task with its comments, finding the linked backend task, dating a retest from status history — is left without input.
A one-off check. «Does this button work» does not need a session folder, a report and a retro. Just ask the agent.
A team that wants a different process. This engine has opinions: one report per session, findings in business language, a mandatory retro, a file scheme that is not up for discussion. Much is configurable in the profile; the shape of the flow is not.
It pays off where the same feature comes back for a second round, where a finding has to survive a copy-paste into the tracker, and where what one session learned has to reach the next one.
13// how-to
How to build something similar
Eight decisions that transfer to any agent pipeline, not just a QA one.
01
Separate the process from the projectThe main lesson of the move from the previous version. Everything true for any project goes into the engine. Everything that names a specific tool, environment or class goes into the profile. Unsure? Ask whether the rule would hold in someone else's repository.
02
Declare capabilities instead of assuming themA header of keys, each none with a documented degradation. No TMS — cases become files. No autotests — the step is skipped and the report says so. That is how one engine serves both a Jira + TestRail + Playwright project and one that has only git and an environment.
03
Name the action by role, not by tool«Launch the role», «ask the user» — in the body of the instructions; how that is done in a particular agent — in the adapter. Then switching harnesses is a new two-page file, not a fork of the engine.
04
Give session files running numbers and one reportThe number is the order of the work. The report numbered 99 is always last in the listing and always singular. The next file number comes from a script.
05
Make the retro a pipeline step, not a wishWith an address for every edit, a mandatory question about the contract version and numbers from transcripts. A lesson without an address is not a lesson.
06
Put a budget on the size of every instructionAnd move situational lessons into catalogs read at their own step or on a trigger. Otherwise the coordinator pays for every lesson in every session, including the ones where none applies.
07
When a catalog outgrows its reader — hand it to a reviewerThe author reads a table of contents with one line per situation; the body goes to a subagent with no context, which returns only what would change the verdict.
08
What the agent checks by eye will have to be checked by codeSooner or later. Especially when the cost of a mistake is a leaked token in someone else's repository.
14// surprises
What surprised me this time
Eight observations, each one worth a rewrite.
Dropping parallelism cost nothingThe previous version was proud of running Web Tester and API Tester in parallel and of discoveries.md. In the current one the executors run in sequence — and not one session has yet found itself short of parallelism. It turned out we were parallelising not the work but the roles, invented along a tool boundary.
A separate web-tester role is unnecessaryA UI scenario and an API scenario belong to the same round. The split bought a state handover in the middle of a round and nothing else.
The most expensive retro mistake is a missed version bumpA wrong edit shows up in the next session. A missed bump never shows up: consumer profiles drift apart silently, and no /qa will warn you.
Edits to an installed plugin vanish while reporting successA marketplace copy with no .git and a version cache the next update overwrites. The worst class of error: not «it broke» but «it reported that it worked». Hence the address check before any engine edit — and a whole eval case for it.
A rule people know is broken at the moment of decision«The manager still wants to do it itself» was not cured by the fifth or the sixth repetition. What helped was writing down the thought that comes right before the violation, next to the rule it violates: a red-flag table catches the moment, repetition does not.
A catalog read by its author works worse than the same catalog in a stranger's handsThe author searches the catalog for what is already in their head, and finds it. A reviewer with no context finds the rest — and costs three minutes.
The instruction language and the output language are different thingsBut the agent has to be told so in the prompt. While the language lived only in the profile, a subagent would start working and narrate it in the console in English inside a Russian session — it opens the profile only after its first actions. Cured by one line: Output language: <lang> at the top of every executor prompt.
Onboarding that ends with a profile listing leaves people without an answerA precedent: after the finished profile was shown, the question came back — «so what do I type to start a session». A listing answers «what I wrote down», not «what is available to me now». The first onboarding now closes with twenty lines of cheat sheet — and only the first.
15// 4 months in the field
4 months in the field: the numbers and what changed since the first version
Above is how the system works now. Below are four months of real data from the docs/test-sessions/ of five projects: 229 sessions, 394 rounds, July to early November. Next to them — the first article's numbers for January to April, counted the same way and without embellishment. Where it grew, where it got harder, where a number went somewhere I did not expect.
229
sessions in 4 months
707+
findings with a priority
67%
sessions with a retest in October
5,250
lines of instructions
Month by month
All numbers are real /qa sessions across five projects, by session start date. November is not in the charts: it has three days and six sessions so far. On the left of every chart is the first version of the article, on the same scale.
Volume: growing again, but from a different base
The first version had 2 → 2 → 14 → 20 sessions a month, and the growth was explained by the launch threshold dropping. Here the threshold is already on the floor: a start at 33 sessions a month, a peak of 76 in September. October is slightly lower in sessions but higher in rounds — sessions got longer, not rarer. Since mid-September that is 16 sessions and 35 rounds a week across five projects.
2
Jan
2
Feb
14
Mar
20
Apr
33
Jul
49
Aug
76
Sep
64
Oct
first versionnow · sessions
Sessions got more complex — more than in the first version
The average number of files in a session folder — the same complexity proxy as in the first version. There the growth was 6.5 → 19.6 over four months; here it is 2.4 → 22.5. An honest caveat about the left edge: in July a session still lived in two files — the passport and the report — and rounds, briefs and retros were not saved as separate files. So half the growth is the session scheme getting more detailed; the other half is a screenshot for every frontend finding and run artifacts (534 of them in October against 5 in August).
6.5
Jan
9.0
Feb
10.6
Mar
19.6
Apr
2.4
Jul
4.6
Aug
9.1
Sep
22.5
Oct
first versionnow · files
Retesting became the norm, not the exception
The share of sessions with two rounds or more. In the first version retesting was just appearing — 7% in March, 20% in April. Now two sessions out of three reach at least a second round, one in three reaches a third, and the record is eight rounds. This is not first-attempt quality dropping: the task comes back after a fix, and /qa on the same task puts the round into the same folder and updates the same report. The median multi-round session lives two calendar weeks.
0
Jan
0
Feb
7
Mar
20
Apr
0
Jul
18
Aug
43
Sep
67
Oct
first versionnow · %
More findings — and more per session
new — not counted in the first version
Findings with an explicit priority in reports, deduplicated: a finding re-checked in later sessions counts once — where it first appeared. Over four months — 707: 7 Blocker, 63 Critical, 330 Major, 307 Minor. Per session — 2.0 in July–August and 4.2 since mid-September; sessions with at least one finding — from half to two thirds. The Critical + Blocker share is stable: about 10% throughout. This is a lower bound — some early reports recorded priority in a form the count does not recognise.
43
Jul
129
Aug
237
Sep
287
Oct
now · findings
The same findings by priority
BlockerCriticalMajorMinor
Jul
430 · 7 · 17 · 19
Aug
1292 · 14 · 77 · 36
Sep
2372 · 28 · 105 · 102
Oct
2873 · 14 · 129 · 141
what is not hereThe «cross-agent communication» trend — lines in discoveries.md — from the first version does not continue: the file is gone, there is no one to exchange with.
// peak sessions
8 rounds, 168 files, 118 MB — three different records
8
rounds · 35 days
168
files · 157 artifacts
118 MB
4 rounds
In the first version the peak session held every record at once: 162 files, 5 rounds, 5 MB of artifacts. Now those are three different sessions, and that is an observation in itself. The most rounds — eight — belong to two sessions at once; the longest-lived of them took 35 calendar days and found 15 findings. The most files — 168 — belong to a two-round session where 157 of them are run artifacts and only 11 are markdown. The heaviest — 118 MB — is four rounds, with 57 non-markdown files. Length, volume and weight stopped being one axis: a long session is many rounds, a heavy one is a lot of evidence.
Instructions: 2,900 lines → 5,250
The first version had four prompt files totalling 2,900 lines and 159 KB, and the last month's growth was ~15% — «the backbone has stabilised». Now the corpus of 34 files is twice as long, and growth since early October is +37%. But the main procedure, SKILL.md, grew only 22% since the first edition of this article (54,400 → 66,300 characters), while the corpus grew 2.3×: the growth went into the lesson catalogs, which are read in full by the reviewer, not the manager. Every edit is still the story of a mistake that should not happen again: the catalogs now hold 424 precedents.
release
date
instruction lines
KB
precedents
tests
0.16
17 Sep
2,739
358
209
66
0.17.39
29 Sep
3,725
588
359
116
0.17.56
5 Oct
3,831
619
387
121
0.18.44
19 Oct
4,418
692
392
199
0.19.15
26 Oct
5,014
743
418
234
0.19.42
3 Nov
5,251
774
456
237
In total, 5,251 lines and 774 KB in 34 files; the first version had 2,900 lines and 159 KB in four. 74 commits and 227 CHANGELOG entries in seven weeks. The closest counterpart of the old qa-manager.md (667 lines, 56 KB) is SKILL.md: 590 lines but ~65 KB — the lines are longer, and CI enforces the 71,500-character ceiling.
More open questions — and that is good news
In the first version I was proud of 0.55 «open questions» per report: the agents follow through, and instead of «couldn't check» they go and create the data. Now it is 1.4 in September and 2.5 in October, and only 16% of reports have an empty section. By this number it looks like a regression. In fact what changed is what goes into that section. Two thirds of today's open questions are addressed to the analyst and to development: the spec is silent about a case, a requirement contradicts its neighbour, a fix is claimed in a comment but is not in the build. «Couldn't check» still barely occurs — the report has a separate «Not covered» section with reasons for that. The engine learned not to hide a hole in the requirements behind a green verdict, and the number grew by exactly that much. The section's goal is still empty; but empty because there are no questions, not because nobody asked them.
Open questions per report
On the left — the first version's only data point: April, 0.55. On the right — August, September, October.
0.55
Apr
0.4
Aug
1.4
Sep
2.5
Oct
v1now
Detail grew in parallel: the median report is 54 lines in August and 120 in October. Only it is no longer «170 rows of scenario tables» as in the first version: the details moved into result files, and the report holds what gets copied into the tracker — findings, what was confirmed, what was not covered, the round history.
How much time and context a round really takes
In the first version time was counted from file timestamps: the median active phase was 13 minutes in March and 50 in April. Now there is no need to count it — the retro takes the numbers from the transcripts itself. Across 56 rounds whose retro already contains them: from the manager's start to the last agent's end — a median of 96 minutes, with a median of five agents per round (nine at most). The manager — 26 minutes and 68 tool calls, the manual tester — 34 minutes and 130 calls, the automator — 19 minutes. The catalog reviewer is the cheapest participant: 3 minutes and 9 calls, and there are about three of them per round.
round economics, medians
time
calls
peak context
max
manager
26 min
68
368k
732k
manual tester
34 min
130
357k
967k
automator
19 min
74
207k
445k
catalog reviewer
3 min
9
131k
228k
A separate number that made all this worth measuring: context compaction — 4 cases in 299 agent launches. The median manager holds close to 370 thousand tokens, and the session cannot see that from inside — which is exactly why a script takes the numbers, not the agent's memory. Instruction size budgets and a reviewer instead of reading the catalog in full are what keep this number small.
2,065 retro lines — and more and more of them say «the rule already exists»
Over four months — 251 retro files and 2,065 lines of «observation → rule → where it landed». Every line has an address: the project profile — 30%, the engine — 25%, the knowledge base — 12%, project skills — 2%, other (environment snapshot, registry) — 8%. The remaining 23% are lines with no edit: the observation is recorded, but the rule already exists and worked, or it was a one-off quirk of the round.The dynamics are more interesting. The share of «no edit» lines — 7% in August, 15% in September, 31% in October; the share of engine edits — 29% → 28% → 22%. The retro more and more often finds that the lesson has already been learned. It is the same backbone stabilisation the first version wrote about — only now it shows not as a feeling that «there are fewer edits» but as a number.
month
retros
lines
engine
profile
knowledge base
no edit
August
17
147
29%
40%
16%
7%
September
98
867
28%
35%
13%
15%
October
129
1,009
22%
24%
12%
31%
And one number about discipline: 36% of sessions have a retro before mid-September and 96% after — ever since the retro became a pipeline step with a file rather than «when there is time».
What surprised me
The «open questions» number went up — and rightly soI expected it to keep falling towards zero. It quadrupled, because the engine stopped folding «the spec is silent» into a green verdict. The metric that looks good and the metric that tells the truth parted ways — and I had to pick the second.
Retesting became the main form of workIn the first version a multi-round session was an event. Now two out of three are multi-round, and the most valuable part of the engine turned out to be not the first round but how the second round reads the first: what was claimed fixed, where to check it, what changed since the last build.
The records split across different sessionsThe longest, the biggest and the heaviest session are three different sessions. Rounds make a session long, evidence makes it heavy. It used to be one and the same, because almost no evidence was kept.
Context runs out less often than I feared4 compactions in 299 launches with a median of 370 thousand tokens for the manager. The instruction budget and the catalog reviewer keep it there not because someone is watching but by construction.
The retro itself shows the engine maturingThe share of «the rule already exists» lines grew from 7% to 31% in three months. It is the best maturity indicator I have: not «fewer edits» but «the observation is already covered».
If you want to try this yourself
The first version's advice still stands: the first two months will look bad on paper, and that is normal — every mistake becomes a line in the instructions that was not there before. I will add one thing: start measuring from the first session. Not time by feel, but transcripts — time, context, errors — and not the number of edits, but the share of observations already covered by a rule. The first shows where you pay in context for what you do not need. The second shows when the system stopped learning on every round and started working.
And if a number goes somewhere you did not expect, first check what it measures. That is what happened to me with open questions: the metric did not get worse, it finally started counting the truth.
how it was countedThe docs/test-sessions/ folders of five projects, 29 to 89 sessions per project; the month is the session start date in the folder name. Rounds — from file suffixes and the round log in the passport; sub-rounds are not counted separately. Findings — «ID · priority» pairs in reports; a project-wide ID counts once. Time and context — transcripts.py tables in retros, available since mid-October: 56 rounds, 299 agent launches. The engine — qa-cube's git history by release tags.
16// outcome
Boundaries matter more than roles
The previous article ended on «architecture beats one clever prompt». Eight months later the refinement sounds like this: what matters is less the architecture of roles than where the boundaries run.Roles by tool (who goes to the API, who goes to the browser) are a false boundary: they split a single job. A role by stage (who writes the expectation, who checks the fact, who commits it to code) is a real one, because each stage has its own input and output, and they can be handed over in a file.The same goes for the project: the boundary between «the process» and «the project» turned out to matter more than the boundary between agents. Once it is drawn, the engine travels to any repository, and the project answers exactly the questions only it knows. And a third boundary, drawn after the first edition — between the engine and the harness: the process should not know which agent is running it.And yes — the agents still mess up. So do people. But unlike people, every one of their mess-ups is a line in a lesson catalog that was not there before, with the precedent that paid for it. The catalogs now hold 424 such lines.
// the previous version
A multi-agent QA system on Claude Code
March 2026: five agents, Web Tester and API Tester running in parallel, findings exchanged through discoveries.md, all of it inside one repository. The article is kept as it was — everything above is built on its conclusions, and its numbers are compared with today's in the field section.