mirror of
https://github.com/hathach/tinyusb.git
synced 2026-08-18 11:02:16 +00:00
feat: Claude Code multi-agent dev/test harness for TinyUSB
Add worker agents (builder, port-dev, driver-reviewer, hil-operator, pr-monitor), deterministic workflows (validate, fanout-dev, driver-review, hil-validate, full-check, pr-babysit) and a /pre-pr gate skill, so sessions can fan build/test/review/PR-triage work out to tiered subagents. pr-babysit drives a PR to green: triage CI + bot reviews, fix validated findings, verify, push, and reply-to + resolve each inline review thread (fixed or refuted). Replace the stop-the-runner HIL discipline with per-board flock locks: test/hil/board_lock.py plus a fail-open guard in hil_test.py let CI and dev sessions share the rig per board (locked boards fail fast and re-run; HIL_NO_BOARD_LOCK=1 is a user-authorized bypass). The actions-runner is never stopped. Design spec, implementation plan, and real-rig smoke evidence under docs/superpowers/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
This commit is contained in:
1685
docs/superpowers/plans/2026-07-09-claude-agents-workflows.md
Normal file
1685
docs/superpowers/plans/2026-07-09-claude-agents-workflows.md
Normal file
File diff suppressed because it is too large
Load Diff
59
docs/superpowers/plans/2026-07-09-smoke-results.md
Normal file
59
docs/superpowers/plans/2026-07-09-smoke-results.md
Normal file
@ -0,0 +1,59 @@
|
||||
# Smoke Test Results — multi-agent dev/test harness
|
||||
|
||||
Date: 2026-07-09 (evening session)
|
||||
|
||||
## Task 13 — validate workflow (run wf_4f10863f-6ff, 2 boards)
|
||||
|
||||
| Stage | Result | Evidence |
|
||||
|---|---|---|
|
||||
| unit | PASS | ceedling 61/61, 2.72 s |
|
||||
| size | PASS | stm32f407disco device/cdc_msc vs master: TOTAL 14167 B, +0.0% every file (`cmake-metrics/stm32f407disco/metrics_compare_device_cdc_msc.md`) |
|
||||
| pvs | pass=false — **gate working as specified** | GA:1=20 / GA:2=36; flags 5 GA:1 in files changed vs local `master` (midi2_device.c:245 V547 branch-new upstream; usbd.c V763 x3 + usbh.c:2109 V1008 pre-existing lines in touched files). Root cause of "changed" set: local `master` lags `origin/master` (upstream MIDI2 merge), so upstream churn counts as changed. Logic per spec. |
|
||||
| build:stm32f407disco | BLOCKED (env) | `agent type 'builder' not found` — see registry note below |
|
||||
| build:raspberry_pi_pico | BLOCKED (env) | same |
|
||||
|
||||
Resume: `Workflow({scriptPath: '<worktree>/.claude/workflows/validate.js', resumeFromRunId: 'wf_4f10863f-6ff', args: {boards: [...]}})` — unit/size/pvs replay from cache.
|
||||
|
||||
## Task 15 — driver-review workflow (run wf_aa86dddc-11f) — PASS
|
||||
|
||||
- 2 dirs x 1 question (unbounded busy-waits), 22 agents (2 scanners + 20 verifiers), 0 errors, ~494k worker tokens.
|
||||
- rusb2: 6 confirmed findings — the FRDY spin `dcd_rusb2.c:126` (matches the known hardware wedge) + CURPIPE spins (125, 249, 292, 477, ...) + CFIFO ISEL spin (347), each adversarially verified against code; one verifier corrected a scanner's claim that the Renesas manuals were missing from calibre (RA6M5/RX65N are present).
|
||||
- Verification layer demonstrably filters and grades (severity/confidence preserved, refutation reasoning recorded).
|
||||
|
||||
## Harness facts discovered (affect all future sessions)
|
||||
|
||||
1. **Custom agent types register at session start, from the LAUNCH directory's `.claude/agents/`** — not the worktree's, and not on file changes mid-session. Runtime copies were mirrored to `/home/hathach/code/tinyusb/.claude/agents/` (untracked); a NEW session is required for builder/port-dev/hil-operator to resolve.
|
||||
2. **Workflow `args` arrives as a JSON string** — all six scripts normalize with `if (typeof args === 'string') args = JSON.parse(args)`.
|
||||
3. **Workflow-by-name resolution can serve a stale cached script** — invoke via `scriptPath` when iterating.
|
||||
|
||||
## Task 13 completion — PASS
|
||||
|
||||
Resumed run wf_4f10863f-6ff once agents registered: build:stm32f407disco 43 examples, build:raspberry_pi_pico 43 examples, both PASS; verdict mechanics correct (pass=false only from the pvs stage's stale-local-master base; re-run vs origin/master in Task 17 is fully green).
|
||||
|
||||
## Task 14 — fanout-dev — PASS (wf_d2fcc566-ebc)
|
||||
|
||||
2 port-dev workers (rp2040, stm32_fsdev): exactly 1-line diffs, `git clang-format` clean, both independently build-verified (`verifyBuild: true`), deps self-healed (get_deps stm32f0; PICO_SDK_PATH). Smoke edits reverted; tree clean.
|
||||
|
||||
## Task 16 — hil-validate + board locks — PASS mechanics (real rig)
|
||||
|
||||
- Lock-conflict run (wf_cab1ac1d-c4c): failed in 48 s, `detail` = `board locked: {holder JSON}` verbatim, `locked: ['raspberry_pi_pico']`, retry attempted, no flash.
|
||||
- Force run (wf_d079a33e-dd8): flashed+tested with HIL_NO_BOARD_LOCK=1, holder pid survived (bypass, not theft), `locked: []`.
|
||||
- Normal run (wf_87ec3c8f-e87): self-locking path live, board flashed/booted 18x, no lock messages.
|
||||
- actions-runner `active` the entire time; svc.sh never touched.
|
||||
- RIG FINDING (not this branch): pico PIO-USB host-port fixture devices (1a86_7523 CDC, 048d_04d2 MSC) not enumerating — 5 host-mode tests fail identically across runs; 13 device-mode tests all pass; firmware exonerated.
|
||||
|
||||
## Task 17 — /pre-pr end-to-end — PASS (wf_8a421ef1-fcf)
|
||||
|
||||
BASE=origin/master (local master stale). No C changes → minimal path: software-only, boards=[stm32f407disco]. Verdict `pass: true`: unit 61/61, 43 examples, size +0.0%, pvs green (no C diffs). HIL correctly skipped.
|
||||
|
||||
## Task 18 — pr-babysit dry + pr-monitor triage — launched
|
||||
|
||||
pr-babysit {pr: 3761, maxCycles: 1, autoPush: false} (wf_361c9e0a-d0e) + direct pr-monitor triage of PR 3750 (3 Copilot rounds) from a /tmp checkout of the PR head. Results recorded when complete.
|
||||
|
||||
## Post-smoke revisions
|
||||
|
||||
- Model tiering (owner): builder→haiku, hil-operator/pr-monitor→sonnet, unit/size→haiku, pvs/push/replies→sonnet; port-dev/driver-reviewer stay opus xhigh.
|
||||
- driver-reviewer now checks MCU errata sheets; missing erratum workarounds are findings.
|
||||
- Remaining: Task 18 verdicts, final whole-branch review, memory note update for the lock protocol.
|
||||
|
||||
Stop-gate extra (done this session): board_lock `cmd_hold` holder-signaled success via pipe (c326eaacc), storm-tested 10/10 exactly-one-winner.
|
||||
@ -0,0 +1,163 @@
|
||||
# Multi-Agent Dev/Test Setup for TinyUSB — Design
|
||||
|
||||
Date: 2026-07-09
|
||||
Branch: worktree-claude-agents-workflows
|
||||
|
||||
## Goal
|
||||
|
||||
Give Claude Code sessions in this repo a reusable, efficient multi-agent harness
|
||||
for developing and testing TinyUSB: custom worker agents that already know the
|
||||
repo's build/test/rig discipline, and small deterministic workflows that fan
|
||||
them out. The orchestrator (main session) authors arguments and reads verdicts;
|
||||
workers do the volume.
|
||||
|
||||
## Context
|
||||
|
||||
- Existing process skills: `hil`, `code-size`, `pvs`, `build-doc`, `usbmon`,
|
||||
`usb-debug`, `usb-recover`, `make-release` (`.claude/skills/`).
|
||||
- One prototype workflow exists in the master working tree (untracked):
|
||||
`.claude/workflows/port-audit.js`. This design supersedes it.
|
||||
- No custom agent definitions exist yet (`.claude/agents/` absent).
|
||||
- Test infra: `test/unit-test` (ceedling), `test/hil` (`hil_test.py`,
|
||||
`tinyusb.json`), `test/fuzz`; size metrics via
|
||||
`tools/metrics_compare_base.py`.
|
||||
|
||||
## Architecture
|
||||
|
||||
Layered: **agents** (who does the work, with baked-in domain knowledge) ×
|
||||
**workflows** (deterministic fan-out/join) × **one skill** (human entry point).
|
||||
|
||||
### Worker agents — `.claude/agents/*.md`
|
||||
|
||||
Tiered models (owner revision 2026-07-09; originally all-opus): `port-dev`
|
||||
and `driver-reviewer` on **opus** at **xhigh**; `hil-operator` and
|
||||
`pr-monitor` on **sonnet**; `builder` on **haiku** (mechanical, log-heavy).
|
||||
|
||||
| Agent | Effort | Role |
|
||||
|---|---|---|
|
||||
| `builder` | low | Build one board's example set with the canonical commands: `cmake -B cmake-build-<board> -DBOARD=<board> -G Ninja -DCMAKE_BUILD_TYPE=MinSizeRel` from `examples/` (exact dir name — HIL expects it), `python3 tools/get_deps.py` on missing deps, `. $HOME/code/esp-idf/export.sh` for Espressif boards, tolerate non-critical objcopy failures. Returns structured `{board, pass, failures: [{example, firstError}]}`. |
|
||||
| `port-dev` | xhigh | Implement one well-scoped change in one port / file set. Follows repo rules: C99, 2-space indent, snake_case, `TU_ASSERT`, no dynamic allocation, ISR work deferred to task context. Runs `clang-format` (repo `.clang-format`) on touched files before finishing. Cross-checks the MCU datasheet in `$HOME/Documents/calibre-library` when changing dcd/hcd register logic. Verifies with a targeted build of one board using the port. Returns `{item, diffstat, buildOk, notes}`. |
|
||||
| `driver-reviewer` | xhigh | Review one dcd/hcd directory against dimensions: correctness, ISR safety, register use vs. datasheet AND MCU errata (calibre library; missing erratum workarounds are findings), style. Returns structured findings `{file, line, snippet, why, severity, confidence}` — coverage-first (report everything; filtering happens downstream). |
|
||||
| `hil-operator` | default | All rig interaction — the actions-runner service is NEVER stopped; per-board flock locks arbitrate with concurrent CI. `hil_test.py` runs rely on its per-board self-locking; manual hardware work (JLink/GDB, usbtest, serial) is wrapped in `test/hil/board_lock.py hold/release`; rig-wide ops (uhubctl, pci-rebind) require `hold --all`; on wedge `usb_recover.sh` + dmesg. Used strictly serially — never two instances concurrently. |
|
||||
| `pr-monitor` | default | Triage one GitHub PR via `gh`: check CI status (`gh pr checks`), read failing run logs and classify each failure infra/flake vs real; re-run infra failures (`gh run rerun --failed`); harvest automated review comments (Codex/Copilot/Claude bots — knows their signals: Codex posts a "Didn't find any major issues" issue comment when clean; Copilot drops out of `requested_reviewers` when done; bot logins differ across APIs); adversarially validate each finding against the actual code. Returns structured triage `{ci: {status, infraRerun[], realFailures[]}, findings: [{source, file, line, claim, verdict, fixHint}]}`. Read/triage/re-run/reply only — never edits code. |
|
||||
|
||||
### Workflows — `.claude/workflows/*.js`
|
||||
|
||||
| Workflow | Args | Shape |
|
||||
|---|---|---|
|
||||
| `validate.js` | `{boards[], examples?, base?, skip?: ('unit'\|'size'\|'pvs')[]}` | One parallel stage: unit tests (ceedling) + one `builder` per board + code-size compare (`tools/metrics_compare_base.py` vs `base`, default master) + PVS analyze. Join → plain-JS verdict `{pass, failures[]}`. Barrier is correct here: the verdict needs all results. |
|
||||
| `fanout-dev.js` | `{task, items[], board?, review?, worktree?}` | `pipeline(items)`: `port-dev` per item → `builder` verify → optional `driver-reviewer` pass. Workers share the tree by default (ports are disjoint directories); `worktree: true` switches on per-agent worktree isolation for collision-prone tasks. Returns per-item results. |
|
||||
| `driver-review.js` | `{dirs[], dimensions?, question?}` | Supersedes `port-audit.js`. Scan stage per (dir × dimension) → adversarial verify per finding (verifier prompted to refute) → confirmed findings only. |
|
||||
| `hil-validate.js` | `{boards[], force?}` | Strictly serial `for` loop of `hil-operator` calls; each board is protected by `hil_test.py`'s own per-board flock, so the actions-runner keeps running throughout. Boards found locked (a concurrent CI job mid-test) are retried once at the end of the loop; boards still locked are returned in `locked[]` for a user force/wait/accept decision. `force: true` (user-authorized only) bypasses locks via `HIL_NO_BOARD_LOCK=1`. Returns per-board `{board, pass, detail}` plus `wedged[]` and `locked[]`. |
|
||||
| `full-check.js` | `{boards[], ...}` | Thin composer: `workflow('validate', ...)` → only if green → `workflow('hil-validate', ...)`. Single nesting level (children do not nest further). |
|
||||
| `pr-babysit.js` | `{pr, maxCycles?, autoPush?}` | Cycle until CI green + review threads resolved, or `maxCycles` (default 3): `pr-monitor` triage (blocks on `gh pr checks --watch` while CI runs) → valid findings + real CI failures grouped by file/port → `port-dev` fix per group (pipeline) → `driver-reviewer` verifies each fix addresses its finding → one commit + push per cycle. Every actioned inline comment is both **replied to and marked resolved** (GraphQL `resolveReviewThread`): refuted findings get the refutation, fixed findings get a "fixed in <sha>" note. `autoPush` defaults true; **invoking this workflow is the explicit push authorization** for follow-up commits on that PR branch (scoped exception to the hold-pushes-until-told rule). |
|
||||
|
||||
### Board lock protocol — `test/hil/` (repo code)
|
||||
|
||||
CI and dev sessions share the rig concurrently; the actions-runner service is
|
||||
never stopped. Arbitration is per-board kernel flocks in
|
||||
`/tmp/tinyusb-hil-locks/<board>.lock` — auto-released when the holder process
|
||||
dies (stale locks impossible; `/tmp` clears on reboot):
|
||||
|
||||
- **`test/hil/board_lock.py`** (new tool): `hold <boards|--all> --reason TEXT`
|
||||
spawns a background holder process flocking each board file (JSON
|
||||
`{pid, reason, since}` written inside for debuggability); `release
|
||||
<boards|--all>` kills holders; `status` lists them. `--all` is required
|
||||
before rig-wide operations (uhubctl power cycling, pci-rebind — bus
|
||||
renumbering affects every board).
|
||||
- **`hil_test.py` guard** (small patch to the per-board worker): take the
|
||||
board's flock non-blocking before flashing and hold it for that board's
|
||||
flash+test; on acquire it writes its own holder info
|
||||
(`{pid, reason: "hil_test.py", since}`) so conflicts report truthfully in
|
||||
both directions. If already held, FAIL the board immediately —
|
||||
`FAILED (board locked: <holder info>)` — no flash, no waiting.
|
||||
The CI job fails visibly for exactly those boards and `re-run failed`
|
||||
passes once the lock is released (`build.yml` already retries the HIL step
|
||||
once, absorbing short dev sessions). Guard defaults to proceeding if the
|
||||
lock dir is absent/odd.
|
||||
- **Re-entrancy rule:** dev sessions do NOT pre-hold boards they are about to
|
||||
run `hil_test.py` on (it self-locks; pre-holding deadlocks it).
|
||||
`board_lock.py hold` is for hardware work outside `hil_test.py` only.
|
||||
- **Symmetric conflicts (CI running while an agent tests):** CI mid-test on a
|
||||
board holds that board's flock, so the dev side hits it — `board_lock.py
|
||||
hold` refuses showing the holder, and a dev `hil_test.py` run fails that
|
||||
board fast. Two sessions can never double-flash a board.
|
||||
`hil-validate.js` retries locked boards once at the end of its loop (CI
|
||||
finishes a board in minutes); manual sessions wait and retry when the
|
||||
holder reason is `hil_test.py`. Concurrent activity on *different* boards
|
||||
is normal — CI's own `hil_test.py` already runs boards in parallel via
|
||||
`multiprocessing.Pool`.
|
||||
- **User decision on persistent locks:** workers cannot prompt the user, so
|
||||
boards still locked after the retry are returned in `locked[]`; the main
|
||||
session then asks the user — **force** (re-invoke `hil-validate` with
|
||||
`force: true`, which runs `hil_test.py` with `HIL_NO_BOARD_LOCK=1`: a
|
||||
bypass, never killing the holder, at the user-accepted risk of colliding
|
||||
with a mid-test CI job), **continue waiting** (re-invoke later), or
|
||||
**accept** the partial result. Workers never force on their own; forcing
|
||||
requires the user's explicit authorization relayed in the prompt.
|
||||
- **Propagation caveat:** CI enforces the guard only once the patch lands on
|
||||
master (CI runs `hil_test.py` from each PR's merge-with-master ref). This
|
||||
also widens blast radius beyond `.claude/` config into shared CI test
|
||||
infra — the change is a small isolated guard, but it needs its own CI pass
|
||||
and careful review on the eventual PR.
|
||||
|
||||
### Entry point — `.claude/skills/pre-pr/SKILL.md`
|
||||
|
||||
`/pre-pr` instructs the session to: scout the diff inline (cheap), map changed
|
||||
`src/portable/<vendor>/<ip>` and `src/class/*` to affected families and pick
|
||||
test boards from `hw/bsp` (fallback: `stm32f407disco`, `raspberry_pi_pico`),
|
||||
launch `full-check` with that board list, and summarize the verdict. Markdown
|
||||
carries the judgment; JS carries the orchestration.
|
||||
|
||||
## Model & effort policy
|
||||
|
||||
- Tiered worker models: `port-dev`/`driver-reviewer` **opus** `xhigh`;
|
||||
`hil-operator`/`pr-monitor` **sonnet**; `builder` **haiku**.
|
||||
- Inline workflow stages: unit/size **haiku**; pvs **sonnet** (low effort);
|
||||
pr-babysit push/replies **sonnet**.
|
||||
- Agent frontmatter `model:` is canonical for `agentType` calls; it is read
|
||||
at session-start registration, so tier changes apply from the next session.
|
||||
- Deterministic control flow (loops, joins, filtering, verdict assembly) is
|
||||
plain JS in the workflow scripts — zero model tokens.
|
||||
- The orchestrating session derives work lists inline (glob/grep) and passes
|
||||
them as `args`; worker prompts carry paths, not file contents.
|
||||
|
||||
## Error handling
|
||||
|
||||
- Workers always return schema-validated structured output; validation retries
|
||||
happen at the tool-call layer.
|
||||
- Workflows `filter(Boolean)` for killed agents and `log()` every skipped or
|
||||
dropped item — no silent truncation.
|
||||
- `validate.js` reports partial results (a failed stage becomes a failure entry,
|
||||
not a thrown error).
|
||||
- `hil-validate.js`: board access arbitrated by `hil_test.py` self-locking; a
|
||||
locked board fails fast with the holder info (never forced); wedged boards
|
||||
reported, not retried blindly.
|
||||
|
||||
## Success criteria
|
||||
|
||||
Each piece smoke-tested on a minimal real scope before the branch is done:
|
||||
|
||||
1. `validate.js` on `stm32f407disco` + `raspberry_pi_pico` (real build, unit
|
||||
tests, size compare, PVS) returns a correct verdict object.
|
||||
2. `fanout-dev.js` on a trivial 2-port task on this branch; diffs build clean
|
||||
and are `.clang-format`-clean.
|
||||
3. `driver-review.js` on 2 driver dirs returns only verified findings.
|
||||
4. `hil-validate.js` on 1 board against the real rig with the actions-runner
|
||||
ACTIVE throughout: while a `board_lock.py` hold is in place the run fails
|
||||
fast citing the holder and returns the board in `locked[]`; with the lock
|
||||
still held, `force: true` proceeds (user-authorized bypass); after
|
||||
release a normal run passes; a direct `hil_test.py` run under a held lock
|
||||
fails that board without flashing.
|
||||
5. `/pre-pr` end-to-end on this branch's own diff.
|
||||
6. `pr-babysit.js` with `autoPush: false` (dry cycle) on a real open PR:
|
||||
CI failures classified correctly, at least one bot finding correctly
|
||||
validated or rejected, proposed fixes produced but not pushed.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- CI (GitHub Actions) integration — this harness is for interactive sessions.
|
||||
- Fuzzing orchestration.
|
||||
- usbtest/usbtest-host stress batteries (existing separate branch).
|
||||
- Removing the untracked `port-audit.js` prototype from the master working
|
||||
tree happens when this branch merges (it is not tracked by git).
|
||||
Reference in New Issue
Block a user