test/hil: add board-pool health check, split hil_test into focused modules (#3794)
Add test/hil/hil_pool_check.py: per-board rig health scan — probe presence,
light-example flash (dfu_runtime; device_info + serial check for host-only
boards), uid re-enumeration, safe recovery (probe authorized-toggle, board
reset), verified board_test re-park, USB topology report, and a markdown
summary table. Missing firmware is built on the spot (tools/build.py, idf.py
for espressif, one get_deps retry); row statuses: ok, flash-failed, failed,
locked. Board locks are always respected, never bypassed.
Refactor hil_test.py into hil_lock.py (flock protocol, controller permits,
hold/release/status CLI; replaces board_lock.py) and hil_flash.py (flashers,
find_firmware, run_cmd). Update WCH probe uids and the board roster in
tinyusb.json; add the hil-pool-check skill.
The plan doc records why the fork exists and how each vendor source was
ported; the interim handoff it superseded is dropped.
CLAUDE.md: a new worktree should symlink the dependency dirs to the
primary checkout rather than re-fetching them, replacing a single
symlink only when the branch needs a different dep revision. Also allow
'linke' in codespell - WCH-LinkE is a product name.
- target-debug: fault frame lives on PSP when EXC_RETURN bit2 set (FreeRTOS
tasks) — decode LR before choosing $msp/$psp (Codex; valid, our verify
happened to fault on MSP)
- esp-target-debug: show the Xtensa gdb for S3 in the attach recipe; clarify
adapter serial = USB-SJ iSerial (colon MAC, hardware-verified) vs the
CP2102N flasher uids in tinyusb.json (Codex; second half of the finding
had the identifiers inverted)
- esp plan: align serial form with the verified command; record the real
console-gate outcome (UART0, USB-SJ console untested) (Copilot)
- target-debug plan: Task 4 now consistently $JB/ARMv7-M matching the
executed JLinkExe path (Copilot)
- drop IDE-local .idea files swept in by the rename commit (Copilot)
Spec (brainstormed): own-skill backend decision, PHY-conflict map, six
verification gates, external-JTAG TODO. Plan executed same-day: all gates
run on the rig; apptrace resolved per its own gate rule as (untested).
- Vector catch + Cortex-M fault autopsy, verified with a deliberate bad-load
on stm32f407disco: CFSR=0x8200 (BFARVALID|PRECISERR), BFAR = exact bad
address, stacked pc addr2lined to the faulting line; gotchas recorded
(stale FPB comparators fire phantom SIGTRAPs — scrub first; arm DEMCR
after reset; loads precise / stores imprecise; ARMv6-M has no CFSR/BFAR)
- SWO exception trace + hw PC sampling gate PASSED on F407: 680 KB of
packets in 3 s (0x17 PC samples in flash range, 0x0E SysTick enter/exit);
JLinkSWOViewerCL decodes stimulus only — raw SWORead is the recipe;
SWOStart needs an explicit speed headless
- verifybin 'Verify successful.'; FreeRTOS -rtos plugin lists all 6
cdc_msc_freertos tasks after a run->stop cycle (plain attach = 0xDEAD
placeholder); semihosting anti-note; monitor-mode pointer (untested)
- Intrusiveness table gains the new rows; agent playbook bullet updated;
retrieval gate 5/5 with a fresh reader; executed plan committed
Completes the debugging toolset (usbmon = what the host exchanged,
usb-debug = why the host acted, usb-sniffer = what crossed the wire):
TU_LOG/RTT capture, per-probe GDB autopsy without reset, RAM ring-buffer
event trace, J-Link DWT_PCSR PC-sampling, dual-side capture posture, and
board-lock rig discipline. Includes the implementation plan it executes.
Hard-won warnings baked in from real bring-up sessions: volatile ring
buffers vs -Os dead-store elimination, RTT NO_BLOCK_SKIP post-mortem
limits (no overwrite mode exists), DHCSR validity anchors for register
snapshots, release-lock-before-hil_test, and that a marginal just-recabled
link can fake a deterministic firmware bug.
Also ignore .claude/worktrees/.
Confirmed by a 10-finder / 28-verifier adversarial review pass:
board_lock.py — the flock is now the sole authority: drop cmd_hold's
pid-liveness pre-gate (a live hil_test.py pool worker's stale record no
longer blocks a genuinely free board); cmd_release probes the flock and
only signals a verified holder, refuses to kill hil_test.py holders
(CI mid-test), handles PermissionError; the holder daemon truncates its
lock records on SIGTERM and keeps the success pipe clear of fds 0-2
(closed-stdio hold used to leave an orphan holder while reporting
failure); --config default resolves beside the script.
hil_test.py — truncate the lock record on per-board release (pool
workers outlive their flocks); warn instead of silently failing open
when the lock dir is unusable; error out on -b names absent from the
config (was a silent zero-test exit 0, readable as a green HIL run);
drop an emptied board row in accumulate_report (variant boards left a
blank ghost row).
workflows — remove the stray positional arg that made the validate size
stage exit 2 on every run; wrap JSON.parse(args) in all six scripts;
factor pr-babysit's drifted reply recipe into postReplyRecipe and dedup
refutation replies across cycles; validate args.pr and maxCycles;
driver-review rejects an empty dimensions list; hil-validate drops a
dead guard clause and retries diagnostics with -v -r 1.
agents/docs — port-dev scopes git clang-format to its own files
(concurrent workers reformatted each other in shared checkouts);
hil-operator/hil skill wording matches actual fail-fast output; the
implementation plan is now a DO-NOT-EXECUTE historical record (banner +
checked boxes) so plan-executing agents cannot revert shipped files.
Verified: lock storm 1-winner-in-10, stale-record hold, closed-stdio
hold, dead-pid cleanup, CI-holder refusal, ghost-row 4-scenario merge,
unknown-board exit 1, py_compile + check.sh on all six workflows,
pre-commit clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Upj4hta5TNoAbidqeC1zZ6
Review-fix batch (owner-confirmed) on the multi-agent harness:
- board_lock: detach holder stdio so a captured `hold` cannot hang on the
daemon's inherited pipe; probe locks by holder-pid liveness instead of a
momentary flock, which could spuriously fail a concurrent acquirer
(storm-tested: 1 winner in 10, 0/15 acquire failures under probe storm)
- hil_test: locked board renders a visible board-locked fail row so the
report matches the exit code; stale marker cleared on a real re-run
- pr-babysit: autoPush now opt-in (default dry run); resolve recipe
paginates reviewThreads; post-push resolve gets issue-comment fallback
- validate: size stage honors non-default base via --base-branch; pvs
stage delegated to the new agent
- new static-analyzer agent (sonnet): PVS-Studio SAST+MISRA for one
board, structured findings gated on files changed vs base
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
Add worker agents (builder, port-dev, driver-reviewer, hil-operator,
pr-monitor), deterministic workflows (validate, fanout-dev, driver-review,
hil-validate, full-check, pr-babysit) and a /pre-pr gate skill, so sessions
can fan build/test/review/PR-triage work out to tiered subagents. pr-babysit
drives a PR to green: triage CI + bot reviews, fix validated findings, verify,
push, and reply-to + resolve each inline review thread (fixed or refuted).
Replace the stop-the-runner HIL discipline with per-board flock locks:
test/hil/board_lock.py plus a fail-open guard in hil_test.py let CI and dev
sessions share the rig per board (locked boards fail fast and re-run;
HIL_NO_BOARD_LOCK=1 is a user-authorized bypass). The actions-runner is
never stopped.
Design spec, implementation plan, and real-rig smoke evidence under
docs/superpowers/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
