A re-run attempt merged into an empty base: another PR's HIL job ran
between attempt 1 and the retry and rewrote the shared hil_report.json,
so the run-stamp guard (correctly) refused the foreign base but the
full-fleet results were lost - the retry report contained only the
re-run cells.
Give each (run id, job) its own report dir instead:
- attempts of the same run share a dir, so the retry always finds its
own sidecar and .failed spec intact
- interleaved runs of other PRs/jobs write elsewhere and cannot clobber
- the run-stamp mechanism (.failed.run file) becomes redundant and is
removed
- stale per-run dirs are pruned after 2 weeks
ceedling 1.1.0 (released 2026-07-17) fails this project's mock
preprocessing ('Failed to read _build/test/preprocess/.../raw/*.h for
comment stripping'); reproduced locally with an isolated 1.1.0 install
while 1.0.1 passes all 61 tests on the same tree. Unpin once fixed
upstream.
vendord_set_alt() aborted bulk/interrupt endpoints of the outgoing
altsetting (stall/clear-stall) but only dropped the iso endpoints'
tracking: an armed iso transfer stayed live in the dcd with its usbd
claim held and no tracked handle to stop it, and its completion fired
into an endpoint the class no longer recognizes. Reachable through the
usbtest example's alt0 (bulk) <-> alt1 (iso) SET_INTERFACE switching.
Track each selected iso endpoint's descriptor (points into the app's
static descriptor set) and deactivate on de-selection: with the
iso-alloc API re-activation is the abort/scrub primitive (resets
ep_status, aborts the stale transfer); without it usbd_edpt_close does,
and the next selection re-opens. Iso cannot be stalled like
bulk/interrupt, hence the separate path.
Build-verified: usbtest for stm32f072disco, ra4m1_ek, raspberry_pi_pico
(iso-alloc) and ch32v307v_r1_1v0 (close API).
FS-forced builds on the HS module are not a supported configuration, so
the 32-bit access path does not need to exist in full-speed builds:
CFG_TUSB_FIFO_HWFIFO_DATA_STRIDE returns to (2 | (TUD_OPT_HIGH_SPEED ? 4 : 0)).
Saves 28 bytes of text on FS-only parts (measured on ra4m1_ek cdc_msc);
high-speed builds keep both widths for the dual-module (FS+HS ports) case.
Build-verified: ra4m1_ek and ra6m5_ek cdc_msc.
Full-fleet profiling (HIL_PROFILE=1 instrumentation, included) showed each
uPD720201 controller's serialized usbtest battery chain dominates wall time,
and a board whose marginal device port bounces during concurrent batteries
can wedge or kill the controller ("xHCI host not responding to stop endpoint
command"). Every such death traced to mimxrt1015's port (its old "kills the
uPD720201" reputation) - it is removed from the config until recabled;
mimxrt1064's enum-retry stalls were a loose device cable (re-seated).
nrf54lm20dk moves to boards-skip until its failing J-Link probe is replugged.
With the hardware fixed both cards run width-4 batteries plus full flash
churn clean, so scheduling stays simple: two symmetric knobs, flashes and
batteries budgeted per controller.
- schedule_boards(): dispatch boards round-robin across host controllers from
a persisted hint cache (~/.cache/tinyusb-hil/ctrl_cache.json), learned and
merge-on-write refreshed each run (concurrent HIL jobs keep each other's
entries). Only the cached PCI address is consumed - dispatch order and
first-flash budgeting, never battery serialization (batteries resolve live
or fail closed to an all-slot permit).
- HIL_FLASH_PARALLEL (8) and HIL_USBTEST_PARALLEL (4) are budgeted per
controller via lock slots assigned on first sight.
- re-runs: a failed run writes <report dir>/<config>.failed with the exact
re-run spec (--accumulate -b <failed board> -bt <board>:<its failed
tests>) instead of the inverted --skip-board list of everything that
passed; --skip-board is gone, --flasher/--exclude-flasher scope a config
across CI jobs by flasher type (no board names hardcoded in workflows),
and -a/--accumulate merges a re-run into the existing report. The spec is
stamped with GITHUB_RUN_ID and cleared on fresh runs, so a retry can never
consume a spec left behind by a different run's dead or skipped attempt.
- CI: esp-idf firmware builds move out of hil-build into hil-build-esp, and
the esptool-flashed boards run in their own hil-tinyusb-esp job, so the
main hil-tinyusb run starts as soon as the fast toolchains finish instead
of waiting on the slow esp-idf build (an esp toolchain flake previously
skipped the whole rig run). Artifacts are namespaced per toolchain so the
esp job downloads only esp-idf binaries.
- HIL_PROFILE=1: timestamped log lines, per-flash durations, permit-wait
logging, uid->controller map dump for analysis.
- hil_report: per-variant test duration as a dedicated trailing column,
recorded only by full runs.
Validated on the ci rig (fixed seeds 20260716/777, full fleet at 8/4):
738s/780s walls with only known-flake failures and no controller deaths,
vs 1134-1211s serialized-battery baseline.
stm32h533nucleo could never link with IAR: family.cmake points LD_FILE_IAR
at linker/stm32h533xx_flash.icf, which did not exist (every sibling H5
variant has one). Surfaced by CircleCI's one-random job picking
stm32h533nucleo+IAR (Fatal error[Lc002]). H533 and H523 have identical
memory maps (512K flash / 272K RAM; their GCC .ld files differ only in a
comment), so the icf is a copy of the H523 one.
The MosChip MCS9990 card is physically removed from the rig: delete its
cases-11/25 SKIP workaround (and the now-orphaned SKIP accounting) from
usbtest.py and refuse to run outright if a DUT ever sits behind one again.
usb_recover.sh gains `hub-cycle <busport>`: uhubctl VBUS cycle of the port
feeding the device, walking upstream (parent hub -> root port) until it
re-enumerates. Verified on the rig: leaf-level recovery (13-4.4 usbtest
device) and full walk to the root port on a dead branch. SKILL.md updated
for the action and the two-Renesas topology (root-port ppps is real; leaf
1a40:0201 hubs fake their "ganged" switching).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxUeX4Yn26KibfjvDg2pN9
edpt_schedule_packets() advanced xfer->buffer past each armed EP0
chunk, so the OUT-complete handler invalidated the cache at the
ADVANCED pointer: one line past the received data. The CPU then read
stale cached bytes instead of the DMA'd packet, and the misplaced
invalidate discarded a dirty line of whatever variable follows the
buffer - random neighbor corruption on every control-OUT data stage.
Found by usbtest ctrl_out (cases 14/21) on espressif_p4_function_ev
with DMA enabled, the first DWC2 target combining buffer DMA with a
data cache: usbd control state wedged after the first control write
(every later request stalled), and one build layout panicked in the
usbd memcpy with a wild pointer.
Rework the EP0 chunk bookkeeping so xfer->buffer always points at the
un-consumed position: the arm no longer advances it; instead the EP0
re-arm paths advance past each completed (full) chunk, invalidating it
first on the OUT side. The final OUT completion invalidates exactly
the received bytes of its last chunk, taken from DOEPDMA ("incremented
on every AHB transaction", databook 7.1.83 - the same semantics the
SETUP path relies on) before dma_setup_prepare() re-targets it. EP0
chunking state (ep0_pending) is now also dropped on bus reset and on
a new SETUP, so a stale latched completion can no longer re-arm EP0
DMA from dead state. No behavior change for targets without dcache.
While root-causing, the FIFO layout was cross-checked against the
DWC2 databook/programming guide v4.20a: the existing GDFIFOCFG
programming (EPInfoBaseAddr = otg_dfifo_depth - 2*ep_count, one SPRAM
word per endpoint direction for buffer DMA) is conformant and needs
no change; the P4 HS instance's reset GDFIFOCFG (0x03800400) merely
reflects a scatter/gather-sized EP_LOC_CNT of 128 that buffer DMA
does not need.
With the fix in place, enable the usbtest battery on the espressif
fleet: tools/build.py allowlists device/usbtest (a plain IDF component
like board_test/video_capture) and both espressif boards' only-lists
gain device/usbtest. Also re-enable device/usbtest on mimxrt1015_evk:
its skip predated the dcd_ci_hs stale-ACTIVE-overlay fix (already on
this branch), which cured the battery that previously killed the
uPD720201 host controller twice (2026-07-11 ROM fw, 2026-07-13 case 27
on fw 2.0.2.6); rig-validated 30/30 three consecutive runs.
Validated on rig (all 30/30): espressif_p4_function_ev(-DMA) (was
22/30 under DMA), espressif_s3_devkitm(-DMA), stm32f723disco(-DMA),
mimxrt1015_evk; p4/s3 slave-mode unaffected (DMA-only code path);
compile-checked stm32h743nucleo +TUD DMA, stm32f407disco,
stm32l476disco (device ports currently on the dead hub).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TQZrFfU3K4Y198aLsUpBC
- dcd_edpt_stall flushes the primed buffer (ENDPTFLUSH), but the aborted
transfer's dQH overlay can be left ACTIVE with mid-transfer state; the
next prime after clear-halt then resumes the stale overlay instead of
loading the fresh qtd, so post-halt IN reads return mid-buffer data
(usbtest case 13 'buf[32] = 56 (not 0)', with case 18 failing
downstream of the same corruption in the full battery). qhd_start_xfer
now clears overlay.active alongside overlay.halted before linking the
new qtd.
- test/hil(hfp): drop lpcxpresso43s67's device/usbtest skip - the
historical first-case wedge no longer reproduces on this branch, and
with the overlay fix the board runs 30/30 on its Fresco xHCI host
(previously 28/30 with deterministic case 13/18 failures).
mimxrt1064_evk (imxrt dcache path) 30/30 regression-clean.
- docs(hil skill): document the external hifiphile rig - pool
test/hil/hfp.json, SSH-reachable from htpc/ci with no outbound SSH,
exercised by the CI hil-tinyusb (hfp.json) job; never run HIL against
it during development unless the user explicitly asks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TQZrFfU3K4Y198aLsUpBC
Pool/config:
- record real uids (ra8m1_ek), enable usbtest for espressif s3/p4, then
park ra6m5_ek and ra8m1_ek in boards-skip (ra6m5's usbtest/MSC traffic
can kill the uPD720201 host on its ROM firmware; ra8m1 USBHS bring-up
pending); max32666/nrf54lm20 stay enabled - their MosChip flakiness
never wedges
- re-enable device/usbtest on HS boards (mimxrt1064, ch32v307) now that
uPD720201 firmware 2.0.2.6 fixes the command-ring death; mimxrt1015
stays skipped - its HS battery killed the controller on both ROM and
2.0.2.6 firmware (board-specific); match the moved host-test bundles
(f723 <-> rt1064); skip never-passing tests on the new
nrf5340dk/nrf54lm20dk boards and the detached pico host bundle, each
documented with a comment
Host-controller quirk gating in usbtest.py (auto-skip, self-heals on a
healthy xHCI):
- MosChip MCS9990 EHCI: case 25 (int-OUT never scheduled, FRINDEX bug)
and case 11 (unlinked reads complete short/EREMOTEIO)
- Renesas uPD720201 xHCI: firmware-gated. The card must run firmware
>= 2.0.2.6 (RAM-uploaded - it reverts to ROM on every power cycle):
on older firmware the command ring dies under unlink stress (a
Configure Endpoint command stops completing; the hub worker deadlocks
holding the device lock; only a host power cycle recovers; three
boards reproduced it). usbtest.py reads the FW version register (PCI
config 0x6c) and refuses to run at all on older firmware - hil_test
surfaces that as a failed test with the reason. On current firmware
the full 30-case battery runs (validated FS+HS: metro_m4, f723,
f723-DMA all 30/30).
Scheduling (hil_test.py):
- Shuffle each (board, variant)'s test order with a seeded RNG
(HIL_SHUFFLE_SEED to replay) so usbtest batteries and flash churn spread
across the timeline instead of convoying on one controller.
- Per-controller usbtest + flash semaphores: HIL_USBTEST_PARALLEL
(default 4) concurrent usbtest batteries and HIL_FLASH_PARALLEL
(default 8) concurrent flashes per host controller. Profiled on
uPD720201 firmware 2.0.2.6 across 8/1..12/8: wall time falls
22.2/14.3/12.5/10.8 min at usbtest width 1/2/3/4 and plateaus there;
zero controller errors everywhere; first battery case failures
(leaf-hub bandwidth stretch) appear at 12/8, and flash width 12 only
amplifies flasher-hub contention flakes - so 8/4 is the optimum. A
separate battery-window flash throttle was profiled and dropped.
- Give every example a unique hardcoded USB PID (0x4001-0x4022, usbtest
keeps 0x4010) instead of the PID_MAP interface bitmap: different
examples now always re-enumerate back-to-back, even on boards whose
CPU reset does not drop D+ (WCH CH58x), so the EXAMPLE_PID table and
same-PID adjacency reordering in hil_test.py are gone; only the
variant-boundary same-example repeat needs a swap.
- Report matrix: stable columns with the metric-bearing tests pinned
first (usbtest, cdc_msc_throughput, msc_file_explorer[_freertos]),
the rest alphabetical.
Fail fast:
- enum wait budget 8 s on the first attempt, 4 s on retries; dfu waits
are deadline-based so dfu-util's own runtime counts against the
budget.
A device-absent failure now costs ~3-5x a passing test (20-30 s)
instead of 10-30x (47-150 s).
- CI runs hil_test with --retry 1 and no in-run second pass: a broken
fixture fails the job fast instead of holding the self-hosted runner
for hours and blocking other PRs' HIL jobs. hil_test still writes the
.skip sidecar, so a manual re-run attempt only retests what failed.
Review fixes (multi-agent adversarial review of this commit):
- tinyusb_win_usbser.inf: the PID rework moved five CDC examples onto
even PIDs the INF's odd-only DeviceList never matched (legacy-Windows
usbser binding) - appended 0x4006/4008/400a/4020/4022 to both lists.
- usbtest example: USBTEST_TIER is now overridable and the descriptors
and pumps are tier-conditional, so a board whose DCD cannot serve a
tier lowers it instead of skipping the whole example - RA2A1 (RUSB2
with no isochronous pipe) builds at tier 3 via its BOARD_ define; the
host battery follows the tier advertised in bcdDevice. Tier-4 output
verified byte-identical after the refactor.
- dynamic_configuration's second config derived USB_PID + 11 = 0x4018,
colliding with net_lwip_webserver - now USB_PID + 0x0100, outside the
per-example space. tools/check_example_pids.py (pre-commit hook)
enforces PID uniqueness incl. derived and literal idProduct values.
- usbtest.py firmware gate: matched by device ID (uPD720201/720202,
both use the 0x6c FW register), and an unreadable version (setpci
missing/denied) now refuses with its own message instead of
masquerading as "firmware 0x00000000"; noted the gate is necessary
but not sufficient (board-specific kills stay per-board skips).
- hil_test: deadline waits use time.monotonic(); multiprocessing
context pinned to fork (raw semaphores in Pool initargs); flash and
usbtest permits unified into one fail-closed, exception-safe
ctrl_permit (unknown controller takes every slot and logs a warning
instead of silently borrowing slot 0); an all-skipped battery
reports as skip, not "0/0" failure; slow-body polls (mtp, printer,
disk read) go through a shared deadline-based wait_until so their
bodies count against the enum budget; throughput's FS detection
compares serials case-insensitively like every other walk; a missing
MSC read-speed line now fails the host msc_file_explorer test
instead of passing with an empty metric.
Hardening:
- fail fast (15 s) when a driver-registry sysfs write blocks: a wedged
device otherwise turns every subsequent battery into an unkillable
D-state writer and silently hangs the whole run
- usb-recover skill: a VM reboot is not a reliable cure (MosChip hubs
latch up across the PCIe reset); full host power cycle is
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg
Confirmed by a 10-finder / 28-verifier adversarial review pass:
board_lock.py — the flock is now the sole authority: drop cmd_hold's
pid-liveness pre-gate (a live hil_test.py pool worker's stale record no
longer blocks a genuinely free board); cmd_release probes the flock and
only signals a verified holder, refuses to kill hil_test.py holders
(CI mid-test), handles PermissionError; the holder daemon truncates its
lock records on SIGTERM and keeps the success pipe clear of fds 0-2
(closed-stdio hold used to leave an orphan holder while reporting
failure); --config default resolves beside the script.
hil_test.py — truncate the lock record on per-board release (pool
workers outlive their flocks); warn instead of silently failing open
when the lock dir is unusable; error out on -b names absent from the
config (was a silent zero-test exit 0, readable as a green HIL run);
drop an emptied board row in accumulate_report (variant boards left a
blank ghost row).
workflows — remove the stray positional arg that made the validate size
stage exit 2 on every run; wrap JSON.parse(args) in all six scripts;
factor pr-babysit's drifted reply recipe into postReplyRecipe and dedup
refutation replies across cycles; validate args.pr and maxCycles;
driver-review rejects an empty dimensions list; hil-validate drops a
dead guard clause and retries diagnostics with -v -r 1.
agents/docs — port-dev scopes git clang-format to its own files
(concurrent workers reformatted each other in shared checkouts);
hil-operator/hil skill wording matches actual fail-fast output; the
implementation plan is now a DO-NOT-EXECUTE historical record (banner +
checked boxes) so plan-executing agents cannot revert shipped files.
Verified: lock storm 1-winner-in-10, stale-record hold, closed-stdio
hold, dead-pid cleanup, CI-holder refusal, ghost-row 4-scenario merge,
unknown-board exit 1, py_compile + check.sh on all six workflows,
pre-commit clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Upj4hta5TNoAbidqeC1zZ6
- EP0 OUT: park a back-to-back data-stage packet the DCP accepted before
PID could go NAK and deliver it into the next armed chunk; flow-control
the single-buffer control pipe between chunks (usbtest ctrl_out
corruption); discard a packet parked while an OUT pipe was halted so
BOT reset recovery's fresh CBW read can't receive stale WRITE data
- HS UTMI PHY power-up per the FSP sequence, shared by dcd/hcd: CLKSEL
programmed from the board XTAL (EK-RA8M1 runs 20 MHz; the 24 MHz reset
default never locks) while DIRPD holds the PHY down, then timed release
- hw/bsp(ra8m1_ek): fix U60CK divider macro - BSP_CFG_U60CK_DIV used the
generic USB_CLOCK_DIV_8 encoding (7), which USB60CKDIVCR rejects,
leaving the USBHS link domain at 480 MHz; the USB60-specific
BSP_CLOCKS_USB60_CLOCK_DIV_8 (4) sticks and yields the required 60 MHz
from PLL1P
- support FS-only builds on the high-speed port: gate SYSCFG.HSE on
TUD_OPT_HIGH_SPEED (RHPORT_DEVICE_SPEED=OPT_MODE_FULL_SPEED was a
silent no-op) and always compile both hwfifo access widths - the FIFO
width belongs to the module, not the link speed (FS builds corrupted
odd-length tails: 16-bit access against MBW-32)
- iso activate: reset stale pipe bookkeeping so a BRDY firing before the
class re-arms can't replay a pre-SET_INTERFACE transfer; write PIPEBUF
after PIPESEL selects the pipe (PIPESEL-windowed register)
- clear-halt: re-assert BUF on a still-armed OUT pipe (usbtest case 29)
- bound the D0FIFO ready spin so an undrained double-buffered IN pipe
can't freeze the stack with the IRQ masked
- usbtest example: cap interrupt mps at 64 on RUSB2 high speed (pipes
6-9 have a fixed 64-byte buffer, RA6M5 UM 29.1)
Verified: usbtest 30/30 on ra6m5_ek (HS), ra4m1_ek (FS) and ra8m1_ek
(FS-forced build on the HS port).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg
The no-op activate left a transfer armed before SET_INTERFACE valid in
data.xfer, letting the ISR complete it against the old buffer. Drop the
descriptor and NAK the endpoint (mirrors the nrf5x fix).
Verified: usbtest 30/30 on ch32v103r, nanoch32v203, ch582m_evt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg
EPSKIP raises a transfer completion, so using it on the stall path let
the class re-arm the endpoint and Active+Stall never actually stalled
(usbtest case 13); write bare Active=0 instead, and retire skipped
transfers on endpoint reopen where the completion is wanted.
Verified: usbtest 30/30 on lpcxpresso11u37.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg
Review-fix batch (owner-confirmed) on the multi-agent harness:
- board_lock: detach holder stdio so a captured `hold` cannot hang on the
daemon's inherited pipe; probe locks by holder-pid liveness instead of a
momentary flock, which could spuriously fail a concurrent acquirer
(storm-tested: 1 winner in 10, 0/15 acquire failures under probe storm)
- hil_test: locked board renders a visible board-locked fail row so the
report matches the exit code; stale marker cleared on a real re-run
- pr-babysit: autoPush now opt-in (default dry run); resolve recipe
paginates reviewThreads; post-push resolve gets issue-comment fallback
- validate: size stage honors non-default base via --base-branch; pvs
stage delegated to the new agent
- new static-analyzer agent (sonnet): PVS-Studio SAST+MISRA for one
board, structured findings gated on files changed vs base
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
nrf_clock_is_running() was added in nrfx 2.1.0 (MDK 8.30.2), but the
version autodetect puts 2.0.0 (MDK 8.29.0) in the v2 bucket, so
hfclk_running() fails to compile on that nrfx.
Route v2 through nrf_clock_hf_is_running(), which all of 2.x provides.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add worker agents (builder, port-dev, driver-reviewer, hil-operator,
pr-monitor), deterministic workflows (validate, fanout-dev, driver-review,
hil-validate, full-check, pr-babysit) and a /pre-pr gate skill, so sessions
can fan build/test/review/PR-triage work out to tiered subagents. pr-babysit
drives a PR to green: triage CI + bot reviews, fix validated findings, verify,
push, and reply-to + resolve each inline review thread (fixed or refuted).
Replace the stop-the-runner HIL discipline with per-board flock locks:
test/hil/board_lock.py plus a fail-open guard in hil_test.py let CI and dev
sessions share the rig per board (locked boards fail fast and re-run;
HIL_NO_BOARD_LOCK=1 is a user-authorized bypass). The actions-runner is
never stopped.
Design spec, implementation plan, and real-rig smoke evidence under
docs/superpowers/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
Direct agents to work in a git worktree rather than switching the shared
primary checkout's branch. Concurrent sessions share that checkout, so a
mid-flight branch switch disrupts the other sessions and can silently
point a review, build, or commit at the wrong diff. CLAUDE.md is a
symlink to AGENTS.md, so both pick this up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg
add a three-level TUSB_VALIDATION_NONE/BASIC/STRICT knob and default
CFG_TUH_VALIDATION_LEVEL to BASIC, then make acm_open the first user so
the enumeration bounds checks compile out at NONE for trusted-device
setups and stay on by default.
