feat: Claude Code multi-agent dev/test harness for TinyUSB

Add worker agents (builder, port-dev, driver-reviewer, hil-operator,
pr-monitor), deterministic workflows (validate, fanout-dev, driver-review,
hil-validate, full-check, pr-babysit) and a /pre-pr gate skill, so sessions
can fan build/test/review/PR-triage work out to tiered subagents. pr-babysit
drives a PR to green: triage CI + bot reviews, fix validated findings, verify,
push, and reply-to + resolve each inline review thread (fixed or refuted).

Replace the stop-the-runner HIL discipline with per-board flock locks:
test/hil/board_lock.py plus a fail-open guard in hil_test.py let CI and dev
sessions share the rig per board (locked boards fail fast and re-run;
HIL_NO_BOARD_LOCK=1 is a user-authorized bypass). The actions-runner is
never stopped.

Design spec, implementation plan, and real-rig smoke evidence under
docs/superpowers/.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
This commit is contained in:
hathach
2026-07-09 23:34:29 +07:00
parent fa750d6bf0
commit e3dd9245ef
19 changed files with 3047 additions and 51 deletions

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,59 @@
# Smoke Test Results — multi-agent dev/test harness
Date: 2026-07-09 (evening session)
## Task 13 — validate workflow (run wf_4f10863f-6ff, 2 boards)
| Stage | Result | Evidence |
|---|---|---|
| unit | PASS | ceedling 61/61, 2.72 s |
| size | PASS | stm32f407disco device/cdc_msc vs master: TOTAL 14167 B, +0.0% every file (`cmake-metrics/stm32f407disco/metrics_compare_device_cdc_msc.md`) |
| pvs | pass=false — **gate working as specified** | GA:1=20 / GA:2=36; flags 5 GA:1 in files changed vs local `master` (midi2_device.c:245 V547 branch-new upstream; usbd.c V763 x3 + usbh.c:2109 V1008 pre-existing lines in touched files). Root cause of "changed" set: local `master` lags `origin/master` (upstream MIDI2 merge), so upstream churn counts as changed. Logic per spec. |
| build:stm32f407disco | BLOCKED (env) | `agent type 'builder' not found` — see registry note below |
| build:raspberry_pi_pico | BLOCKED (env) | same |
Resume: `Workflow({scriptPath: '<worktree>/.claude/workflows/validate.js', resumeFromRunId: 'wf_4f10863f-6ff', args: {boards: [...]}})` — unit/size/pvs replay from cache.
## Task 15 — driver-review workflow (run wf_aa86dddc-11f) — PASS
- 2 dirs x 1 question (unbounded busy-waits), 22 agents (2 scanners + 20 verifiers), 0 errors, ~494k worker tokens.
- rusb2: 6 confirmed findings — the FRDY spin `dcd_rusb2.c:126` (matches the known hardware wedge) + CURPIPE spins (125, 249, 292, 477, ...) + CFIFO ISEL spin (347), each adversarially verified against code; one verifier corrected a scanner's claim that the Renesas manuals were missing from calibre (RA6M5/RX65N are present).
- Verification layer demonstrably filters and grades (severity/confidence preserved, refutation reasoning recorded).
## Harness facts discovered (affect all future sessions)
1. **Custom agent types register at session start, from the LAUNCH directory's `.claude/agents/`** — not the worktree's, and not on file changes mid-session. Runtime copies were mirrored to `/home/hathach/code/tinyusb/.claude/agents/` (untracked); a NEW session is required for builder/port-dev/hil-operator to resolve.
2. **Workflow `args` arrives as a JSON string** — all six scripts normalize with `if (typeof args === 'string') args = JSON.parse(args)`.
3. **Workflow-by-name resolution can serve a stale cached script** — invoke via `scriptPath` when iterating.
## Task 13 completion — PASS
Resumed run wf_4f10863f-6ff once agents registered: build:stm32f407disco 43 examples, build:raspberry_pi_pico 43 examples, both PASS; verdict mechanics correct (pass=false only from the pvs stage's stale-local-master base; re-run vs origin/master in Task 17 is fully green).
## Task 14 — fanout-dev — PASS (wf_d2fcc566-ebc)
2 port-dev workers (rp2040, stm32_fsdev): exactly 1-line diffs, `git clang-format` clean, both independently build-verified (`verifyBuild: true`), deps self-healed (get_deps stm32f0; PICO_SDK_PATH). Smoke edits reverted; tree clean.
## Task 16 — hil-validate + board locks — PASS mechanics (real rig)
- Lock-conflict run (wf_cab1ac1d-c4c): failed in 48 s, `detail` = `board locked: {holder JSON}` verbatim, `locked: ['raspberry_pi_pico']`, retry attempted, no flash.
- Force run (wf_d079a33e-dd8): flashed+tested with HIL_NO_BOARD_LOCK=1, holder pid survived (bypass, not theft), `locked: []`.
- Normal run (wf_87ec3c8f-e87): self-locking path live, board flashed/booted 18x, no lock messages.
- actions-runner `active` the entire time; svc.sh never touched.
- RIG FINDING (not this branch): pico PIO-USB host-port fixture devices (1a86_7523 CDC, 048d_04d2 MSC) not enumerating — 5 host-mode tests fail identically across runs; 13 device-mode tests all pass; firmware exonerated.
## Task 17 — /pre-pr end-to-end — PASS (wf_8a421ef1-fcf)
BASE=origin/master (local master stale). No C changes → minimal path: software-only, boards=[stm32f407disco]. Verdict `pass: true`: unit 61/61, 43 examples, size +0.0%, pvs green (no C diffs). HIL correctly skipped.
## Task 18 — pr-babysit dry + pr-monitor triage — launched
pr-babysit {pr: 3761, maxCycles: 1, autoPush: false} (wf_361c9e0a-d0e) + direct pr-monitor triage of PR 3750 (3 Copilot rounds) from a /tmp checkout of the PR head. Results recorded when complete.
## Post-smoke revisions
- Model tiering (owner): builder→haiku, hil-operator/pr-monitor→sonnet, unit/size→haiku, pvs/push/replies→sonnet; port-dev/driver-reviewer stay opus xhigh.
- driver-reviewer now checks MCU errata sheets; missing erratum workarounds are findings.
- Remaining: Task 18 verdicts, final whole-branch review, memory note update for the lock protocol.
Stop-gate extra (done this session): board_lock `cmd_hold` holder-signaled success via pipe (c326eaacc), storm-tested 10/10 exactly-one-winner.

View File

@ -0,0 +1,163 @@
# Multi-Agent Dev/Test Setup for TinyUSB — Design
Date: 2026-07-09
Branch: worktree-claude-agents-workflows
## Goal
Give Claude Code sessions in this repo a reusable, efficient multi-agent harness
for developing and testing TinyUSB: custom worker agents that already know the
repo's build/test/rig discipline, and small deterministic workflows that fan
them out. The orchestrator (main session) authors arguments and reads verdicts;
workers do the volume.
## Context
- Existing process skills: `hil`, `code-size`, `pvs`, `build-doc`, `usbmon`,
`usb-debug`, `usb-recover`, `make-release` (`.claude/skills/`).
- One prototype workflow exists in the master working tree (untracked):
`.claude/workflows/port-audit.js`. This design supersedes it.
- No custom agent definitions exist yet (`.claude/agents/` absent).
- Test infra: `test/unit-test` (ceedling), `test/hil` (`hil_test.py`,
`tinyusb.json`), `test/fuzz`; size metrics via
`tools/metrics_compare_base.py`.
## Architecture
Layered: **agents** (who does the work, with baked-in domain knowledge) ×
**workflows** (deterministic fan-out/join) × **one skill** (human entry point).
### Worker agents — `.claude/agents/*.md`
Tiered models (owner revision 2026-07-09; originally all-opus): `port-dev`
and `driver-reviewer` on **opus** at **xhigh**; `hil-operator` and
`pr-monitor` on **sonnet**; `builder` on **haiku** (mechanical, log-heavy).
| Agent | Effort | Role |
|---|---|---|
| `builder` | low | Build one board's example set with the canonical commands: `cmake -B cmake-build-<board> -DBOARD=<board> -G Ninja -DCMAKE_BUILD_TYPE=MinSizeRel` from `examples/` (exact dir name — HIL expects it), `python3 tools/get_deps.py` on missing deps, `. $HOME/code/esp-idf/export.sh` for Espressif boards, tolerate non-critical objcopy failures. Returns structured `{board, pass, failures: [{example, firstError}]}`. |
| `port-dev` | xhigh | Implement one well-scoped change in one port / file set. Follows repo rules: C99, 2-space indent, snake_case, `TU_ASSERT`, no dynamic allocation, ISR work deferred to task context. Runs `clang-format` (repo `.clang-format`) on touched files before finishing. Cross-checks the MCU datasheet in `$HOME/Documents/calibre-library` when changing dcd/hcd register logic. Verifies with a targeted build of one board using the port. Returns `{item, diffstat, buildOk, notes}`. |
| `driver-reviewer` | xhigh | Review one dcd/hcd directory against dimensions: correctness, ISR safety, register use vs. datasheet AND MCU errata (calibre library; missing erratum workarounds are findings), style. Returns structured findings `{file, line, snippet, why, severity, confidence}` — coverage-first (report everything; filtering happens downstream). |
| `hil-operator` | default | All rig interaction — the actions-runner service is NEVER stopped; per-board flock locks arbitrate with concurrent CI. `hil_test.py` runs rely on its per-board self-locking; manual hardware work (JLink/GDB, usbtest, serial) is wrapped in `test/hil/board_lock.py hold/release`; rig-wide ops (uhubctl, pci-rebind) require `hold --all`; on wedge `usb_recover.sh` + dmesg. Used strictly serially — never two instances concurrently. |
| `pr-monitor` | default | Triage one GitHub PR via `gh`: check CI status (`gh pr checks`), read failing run logs and classify each failure infra/flake vs real; re-run infra failures (`gh run rerun --failed`); harvest automated review comments (Codex/Copilot/Claude bots — knows their signals: Codex posts a "Didn't find any major issues" issue comment when clean; Copilot drops out of `requested_reviewers` when done; bot logins differ across APIs); adversarially validate each finding against the actual code. Returns structured triage `{ci: {status, infraRerun[], realFailures[]}, findings: [{source, file, line, claim, verdict, fixHint}]}`. Read/triage/re-run/reply only — never edits code. |
### Workflows — `.claude/workflows/*.js`
| Workflow | Args | Shape |
|---|---|---|
| `validate.js` | `{boards[], examples?, base?, skip?: ('unit'\|'size'\|'pvs')[]}` | One parallel stage: unit tests (ceedling) + one `builder` per board + code-size compare (`tools/metrics_compare_base.py` vs `base`, default master) + PVS analyze. Join → plain-JS verdict `{pass, failures[]}`. Barrier is correct here: the verdict needs all results. |
| `fanout-dev.js` | `{task, items[], board?, review?, worktree?}` | `pipeline(items)`: `port-dev` per item → `builder` verify → optional `driver-reviewer` pass. Workers share the tree by default (ports are disjoint directories); `worktree: true` switches on per-agent worktree isolation for collision-prone tasks. Returns per-item results. |
| `driver-review.js` | `{dirs[], dimensions?, question?}` | Supersedes `port-audit.js`. Scan stage per (dir × dimension) → adversarial verify per finding (verifier prompted to refute) → confirmed findings only. |
| `hil-validate.js` | `{boards[], force?}` | Strictly serial `for` loop of `hil-operator` calls; each board is protected by `hil_test.py`'s own per-board flock, so the actions-runner keeps running throughout. Boards found locked (a concurrent CI job mid-test) are retried once at the end of the loop; boards still locked are returned in `locked[]` for a user force/wait/accept decision. `force: true` (user-authorized only) bypasses locks via `HIL_NO_BOARD_LOCK=1`. Returns per-board `{board, pass, detail}` plus `wedged[]` and `locked[]`. |
| `full-check.js` | `{boards[], ...}` | Thin composer: `workflow('validate', ...)` → only if green → `workflow('hil-validate', ...)`. Single nesting level (children do not nest further). |
| `pr-babysit.js` | `{pr, maxCycles?, autoPush?}` | Cycle until CI green + review threads resolved, or `maxCycles` (default 3): `pr-monitor` triage (blocks on `gh pr checks --watch` while CI runs) → valid findings + real CI failures grouped by file/port → `port-dev` fix per group (pipeline) → `driver-reviewer` verifies each fix addresses its finding → one commit + push per cycle. Every actioned inline comment is both **replied to and marked resolved** (GraphQL `resolveReviewThread`): refuted findings get the refutation, fixed findings get a "fixed in <sha>" note. `autoPush` defaults true; **invoking this workflow is the explicit push authorization** for follow-up commits on that PR branch (scoped exception to the hold-pushes-until-told rule). |
### Board lock protocol — `test/hil/` (repo code)
CI and dev sessions share the rig concurrently; the actions-runner service is
never stopped. Arbitration is per-board kernel flocks in
`/tmp/tinyusb-hil-locks/<board>.lock` — auto-released when the holder process
dies (stale locks impossible; `/tmp` clears on reboot):
- **`test/hil/board_lock.py`** (new tool): `hold <boards|--all> --reason TEXT`
spawns a background holder process flocking each board file (JSON
`{pid, reason, since}` written inside for debuggability); `release
<boards|--all>` kills holders; `status` lists them. `--all` is required
before rig-wide operations (uhubctl power cycling, pci-rebind — bus
renumbering affects every board).
- **`hil_test.py` guard** (small patch to the per-board worker): take the
board's flock non-blocking before flashing and hold it for that board's
flash+test; on acquire it writes its own holder info
(`{pid, reason: "hil_test.py", since}`) so conflicts report truthfully in
both directions. If already held, FAIL the board immediately —
`FAILED (board locked: <holder info>)` — no flash, no waiting.
The CI job fails visibly for exactly those boards and `re-run failed`
passes once the lock is released (`build.yml` already retries the HIL step
once, absorbing short dev sessions). Guard defaults to proceeding if the
lock dir is absent/odd.
- **Re-entrancy rule:** dev sessions do NOT pre-hold boards they are about to
run `hil_test.py` on (it self-locks; pre-holding deadlocks it).
`board_lock.py hold` is for hardware work outside `hil_test.py` only.
- **Symmetric conflicts (CI running while an agent tests):** CI mid-test on a
board holds that board's flock, so the dev side hits it — `board_lock.py
hold` refuses showing the holder, and a dev `hil_test.py` run fails that
board fast. Two sessions can never double-flash a board.
`hil-validate.js` retries locked boards once at the end of its loop (CI
finishes a board in minutes); manual sessions wait and retry when the
holder reason is `hil_test.py`. Concurrent activity on *different* boards
is normal — CI's own `hil_test.py` already runs boards in parallel via
`multiprocessing.Pool`.
- **User decision on persistent locks:** workers cannot prompt the user, so
boards still locked after the retry are returned in `locked[]`; the main
session then asks the user — **force** (re-invoke `hil-validate` with
`force: true`, which runs `hil_test.py` with `HIL_NO_BOARD_LOCK=1`: a
bypass, never killing the holder, at the user-accepted risk of colliding
with a mid-test CI job), **continue waiting** (re-invoke later), or
**accept** the partial result. Workers never force on their own; forcing
requires the user's explicit authorization relayed in the prompt.
- **Propagation caveat:** CI enforces the guard only once the patch lands on
master (CI runs `hil_test.py` from each PR's merge-with-master ref). This
also widens blast radius beyond `.claude/` config into shared CI test
infra — the change is a small isolated guard, but it needs its own CI pass
and careful review on the eventual PR.
### Entry point — `.claude/skills/pre-pr/SKILL.md`
`/pre-pr` instructs the session to: scout the diff inline (cheap), map changed
`src/portable/<vendor>/<ip>` and `src/class/*` to affected families and pick
test boards from `hw/bsp` (fallback: `stm32f407disco`, `raspberry_pi_pico`),
launch `full-check` with that board list, and summarize the verdict. Markdown
carries the judgment; JS carries the orchestration.
## Model & effort policy
- Tiered worker models: `port-dev`/`driver-reviewer` **opus** `xhigh`;
`hil-operator`/`pr-monitor` **sonnet**; `builder` **haiku**.
- Inline workflow stages: unit/size **haiku**; pvs **sonnet** (low effort);
pr-babysit push/replies **sonnet**.
- Agent frontmatter `model:` is canonical for `agentType` calls; it is read
at session-start registration, so tier changes apply from the next session.
- Deterministic control flow (loops, joins, filtering, verdict assembly) is
plain JS in the workflow scripts — zero model tokens.
- The orchestrating session derives work lists inline (glob/grep) and passes
them as `args`; worker prompts carry paths, not file contents.
## Error handling
- Workers always return schema-validated structured output; validation retries
happen at the tool-call layer.
- Workflows `filter(Boolean)` for killed agents and `log()` every skipped or
dropped item — no silent truncation.
- `validate.js` reports partial results (a failed stage becomes a failure entry,
not a thrown error).
- `hil-validate.js`: board access arbitrated by `hil_test.py` self-locking; a
locked board fails fast with the holder info (never forced); wedged boards
reported, not retried blindly.
## Success criteria
Each piece smoke-tested on a minimal real scope before the branch is done:
1. `validate.js` on `stm32f407disco` + `raspberry_pi_pico` (real build, unit
tests, size compare, PVS) returns a correct verdict object.
2. `fanout-dev.js` on a trivial 2-port task on this branch; diffs build clean
and are `.clang-format`-clean.
3. `driver-review.js` on 2 driver dirs returns only verified findings.
4. `hil-validate.js` on 1 board against the real rig with the actions-runner
ACTIVE throughout: while a `board_lock.py` hold is in place the run fails
fast citing the holder and returns the board in `locked[]`; with the lock
still held, `force: true` proceeds (user-authorized bypass); after
release a normal run passes; a direct `hil_test.py` run under a held lock
fails that board without flashing.
5. `/pre-pr` end-to-end on this branch's own diff.
6. `pr-babysit.js` with `autoPush: false` (dry cycle) on a real open PR:
CI failures classified correctly, at least one bot finding correctly
validated or rejected, proposed fixes produced but not pushed.
## Out of scope
- CI (GitHub Actions) integration — this harness is for interactive sessions.
- Fuzzing orchestration.
- usbtest/usbtest-host stress batteries (existing separate branch).
- Removing the untracked `port-audit.js` prototype from the master working
tree happens when this branch merges (it is not tracked by git).