ERR050101: while an isochronous IN endpoint is active, an IN token addressed to
that same endpoint number on ANOTHER device sharing the host can silently
unprime one of this device's OUT endpoints - control, bulk, interrupt or
isochronous alike. NXP states it cannot be detected by software and raises no
interrupt, so the endpoint simply stops answering and the transfer never
completes.
The workaround is a uniqueness requirement rather than a particular number: the
isochronous IN endpoint must not share its number with any IN endpoint in use on
the bus. One family-wide constant therefore defeats it, since two affected
boards on the same hub then pick the same number and each becomes the other's
aggressor. CFG_TUSB_MIMXRT1XXX_ERRATA_ERR050101 is set only for the parts whose
errata list it - RT1015, RT1020, RT1024 and RT1050, where it is marked no fix
scheduled, plus RT1060 and RT1064 rev A - so RT1010 and the RT11xx family keep
the ordinary number and cannot collide with an affected board beside them.
Several affected boards on one hub can still be given distinct numbers with
-DEPNUM_ISO_IN.
The guard covers every example that has an isochronous IN endpoint: audio_test,
audio_4_channel_mic, uac2_headset, cdc_uac2, usbtest, video_capture and
video_capture_2ch. The video examples move the endpoint only when streaming
isochronously, since the bulk configuration is unaffected, and video_capture_2ch
takes two numbers because it has two streams.
The macro name follows CFG_TUSB_RP2_ERRATA_E2/E4/E15 already in tree, and its
is fixed, and which cannot be told apart at compile time - a way to define it to
0.
device_issues.rst records ERR050101 against every affected part with a link to
each errata sheet, and adds the LPC55S2x USB.3 speed-detection and USB.5
isochronous IN entries, neither of which TinyUSB works around. The branch's
design notes are included under docs/superpowers.
Verified: 340 wedge-free runs on mimxrt1064_evk, which previously wedged within
hours, and the macro resolving to endpoint 0x87 on mimxrt1064_evk against 0x83
on mimxrt1010_evk and stm32f407disco.
Records the convention in CLAUDE.md -- deferred work is a SEPARATE scope that deserves
its own PR, written by another session, so it is handed off as a writing-plans doc in
docs/superpowers/followup/pr<NNN>-<topic>.md rather than accumulated in the PR that
found it.
Five handoffs from #3803: flasher_recover (convoy-safe recovery for J-Link boards, seven
validated on the rig), the blindness reporting gaps, the usbtest recovery reserve, the
IAR re-run spec, and the pci-rebind stranding question. Each carries what is already
established with its citations and measurements, what remains, and why it was split out.
One doc per follow-up, not one per PR: a per-PR file invites unrelated work into the
same document and rots as a unit.
Two things the rig taught us that the old guidance got wrong.
A usbfs ioctl wedged in D state cannot be freed on a running kernel. It holds
the device lock, so usb_disconnect() blocks behind it; reboot(2) walks
device_shutdown() and takes the same lock, so every userspace reboot stalls too.
Only sysrq b (emergency_restart, which skips device_shutdown) or hypervisor
action clears it -- all cited to the kernel source. The recovery ladder is
generic across rigs now (ci.lan, hifiphile, a bench PC) and ends at hypervisor
escalation only where host access exists. Two claims are corrected outright:
JLinkExe is NOT convoy-safe, and a park-flash cannot free a device-lock owner.
The hil skill's banner list is what an operator agent matches a report against,
so it enumerates the banners that actually exist, including the D-state note --
which is explicitly NOT a wedge, since a healthy in-flight testusb is
uninterruptible for most of every case and a concurrent CI battery would
otherwise turn a clean run red.
A wedged USB device used to take the whole HIL run with it. Every worker that
touched the poisoned node blocked uninterruptibly, the pool could not be joined,
map_async discarded every board's result, and the job ran to the GitHub ceiling
with no report at all -- while the self-hosted runner's single job slot stayed
occupied and every queued job waited behind it.
Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all
answer within a wall clock; read_sysfs distinguishes "absent" from "unknown",
because a blocked read is not evidence of absence, and caps stranded readers at
four (each costs a thread and an fd for the life of the process) after which the
worker declares itself blind. mtype, the gio unmount, the libmtp session and the
arecord/iperf reaps go through those bounds; the MTP session runs in a disposable
subprocess, since libmtp's ctypes calls block unkillably in D state.
Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job
ceiling and still writes a report. When the pool will not shut down, the sweep
kills what the workers spawned -- descendants, not just direct children, since
flashers run in their own session -- confirms each kill actually landed, and
exits early so the runner is freed. Whatever survived is named in the report.
Deliberately shallow past that point. We do not re-scan process groups, prove
pid ownership, or escalate through sudo: a root-owned survivor is reported, not
force-killed, because signalling a pid we cannot prove is ours is the worse
failure, and the job ceiling backstops whatever this misses. A D-state holder
was never killable anyway.
Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT
through its roster flasher, but only where the flasher can reach its probe past
a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it
cannot, the run says so rather than reserving budget for a path that cannot fire.
Raise the CI ceilings above the pool guard so the guard fires first and still
writes its report, and pin --retry 1 on every HIL leg: the guard is a flat
constant and does not scale with max_retry, so argparse's default of 3 would
triple the serialized usbtest tail against an unchanged guard.
Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in
helper/ (locking, health, selection, shared bounded IO), and the two matrix
generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where
GitHub treats every file as a workflow definition. 193 tests cover the bounded
paths, the kill ladder, the guard and the selector against synthetic /proc trees
and PATH-injected fakes; a real wedge cannot be manufactured on demand.
hil, ci: scope HIL builds and tests to the boards a PR affects
Add test/hil/hil_select.py, a stdlib-only selector that maps a PR diff to the
rig boards, tests and BSP families a change can affect, and wire it into CI so
pull requests build and run only those. A port change picks its families' boards,
a class change picks the examples enabling that class, and device/host changes
prune the other role. Anything unclassified — infra, an unmapped port, a selector
error — falls back to the full matrix, and push/schedule runs are untouched.
Move the shared example lists to hil_examples.py; 54 hardware-free tests cover
the rules.
test/hil: add board-pool health check, split hil_test into focused modules (#3794)
Add test/hil/hil_pool_check.py: per-board rig health scan — probe presence,
light-example flash (dfu_runtime; device_info + serial check for host-only
boards), uid re-enumeration, safe recovery (probe authorized-toggle, board
reset), verified board_test re-park, USB topology report, and a markdown
summary table. Missing firmware is built on the spot (tools/build.py, idf.py
for espressif, one get_deps retry); row statuses: ok, flash-failed, failed,
locked. Board locks are always respected, never bypassed.
Refactor hil_test.py into hil_lock.py (flock protocol, controller permits,
hold/release/status CLI; replaces board_lock.py) and hil_flash.py (flashers,
find_firmware, run_cmd). Update WCH probe uids and the board roster in
tinyusb.json; add the hil-pool-check skill.
The plan doc records why the fork exists and how each vendor source was
ported; the interim handoff it superseded is dropped.
CLAUDE.md: a new worktree should symlink the dependency dirs to the
primary checkout rather than re-fetching them, replacing a single
symlink only when the branch needs a different dep revision. Also allow
'linke' in codespell - WCH-LinkE is a product name.
- target-debug: fault frame lives on PSP when EXC_RETURN bit2 set (FreeRTOS
tasks) — decode LR before choosing $msp/$psp (Codex; valid, our verify
happened to fault on MSP)
- esp-target-debug: show the Xtensa gdb for S3 in the attach recipe; clarify
adapter serial = USB-SJ iSerial (colon MAC, hardware-verified) vs the
CP2102N flasher uids in tinyusb.json (Codex; second half of the finding
had the identifiers inverted)
- esp plan: align serial form with the verified command; record the real
console-gate outcome (UART0, USB-SJ console untested) (Copilot)
- target-debug plan: Task 4 now consistently $JB/ARMv7-M matching the
executed JLinkExe path (Copilot)
- drop IDE-local .idea files swept in by the rename commit (Copilot)
Spec (brainstormed): own-skill backend decision, PHY-conflict map, six
verification gates, external-JTAG TODO. Plan executed same-day: all gates
run on the rig; apptrace resolved per its own gate rule as (untested).
- Vector catch + Cortex-M fault autopsy, verified with a deliberate bad-load
on stm32f407disco: CFSR=0x8200 (BFARVALID|PRECISERR), BFAR = exact bad
address, stacked pc addr2lined to the faulting line; gotchas recorded
(stale FPB comparators fire phantom SIGTRAPs — scrub first; arm DEMCR
after reset; loads precise / stores imprecise; ARMv6-M has no CFSR/BFAR)
- SWO exception trace + hw PC sampling gate PASSED on F407: 680 KB of
packets in 3 s (0x17 PC samples in flash range, 0x0E SysTick enter/exit);
JLinkSWOViewerCL decodes stimulus only — raw SWORead is the recipe;
SWOStart needs an explicit speed headless
- verifybin 'Verify successful.'; FreeRTOS -rtos plugin lists all 6
cdc_msc_freertos tasks after a run->stop cycle (plain attach = 0xDEAD
placeholder); semihosting anti-note; monitor-mode pointer (untested)
- Intrusiveness table gains the new rows; agent playbook bullet updated;
retrieval gate 5/5 with a fresh reader; executed plan committed
Opus-tier agent charter for backgrounding a long hardware debug session:
instrument -> build -> flash under one held board lock -> dual-side
capture -> correlate -> refine, strictly one instance, skills as source
of truth (usb-target-debug, usbmon, usb-debug, usb-sniffer, usb-recover,
hil). The charter encodes what dogfooding established:
- diagnosis standard: evidence must show the mechanism, or a fix must
flip the ORIGINAL failing case on hardware; stop after two
evidence-free cycles and hand back a partial diagnosis
- lock cadence: hold for the whole session, release around hil_test.py
runs (it self-locks per board)
- revert semantics: "fix stays, probe goes, re-verify clean" —
instrumentation reverted, candidate fix left uncommitted and
re-verified on a clean build, pristine firmware reflashed before
lock release
Returns a machine-parseable diagnosis report including ruledOut[] —
disproven hypotheses are deliverables. Spec roster updated (opus/xhigh,
effort requested per agent() call).
Completes the debugging toolset (usbmon = what the host exchanged,
usb-debug = why the host acted, usb-sniffer = what crossed the wire):
TU_LOG/RTT capture, per-probe GDB autopsy without reset, RAM ring-buffer
event trace, J-Link DWT_PCSR PC-sampling, dual-side capture posture, and
board-lock rig discipline. Includes the implementation plan it executes.
Hard-won warnings baked in from real bring-up sessions: volatile ring
buffers vs -Os dead-store elimination, RTT NO_BLOCK_SKIP post-mortem
limits (no overwrite mode exists), DHCSR validity anchors for register
snapshots, release-lock-before-hil_test, and that a marginal just-recabled
link can fake a deterministic firmware bug.
Also ignore .claude/worktrees/.
Confirmed by a 10-finder / 28-verifier adversarial review pass:
board_lock.py — the flock is now the sole authority: drop cmd_hold's
pid-liveness pre-gate (a live hil_test.py pool worker's stale record no
longer blocks a genuinely free board); cmd_release probes the flock and
only signals a verified holder, refuses to kill hil_test.py holders
(CI mid-test), handles PermissionError; the holder daemon truncates its
lock records on SIGTERM and keeps the success pipe clear of fds 0-2
(closed-stdio hold used to leave an orphan holder while reporting
failure); --config default resolves beside the script.
hil_test.py — truncate the lock record on per-board release (pool
workers outlive their flocks); warn instead of silently failing open
when the lock dir is unusable; error out on -b names absent from the
config (was a silent zero-test exit 0, readable as a green HIL run);
drop an emptied board row in accumulate_report (variant boards left a
blank ghost row).
workflows — remove the stray positional arg that made the validate size
stage exit 2 on every run; wrap JSON.parse(args) in all six scripts;
factor pr-babysit's drifted reply recipe into postReplyRecipe and dedup
refutation replies across cycles; validate args.pr and maxCycles;
driver-review rejects an empty dimensions list; hil-validate drops a
dead guard clause and retries diagnostics with -v -r 1.
agents/docs — port-dev scopes git clang-format to its own files
(concurrent workers reformatted each other in shared checkouts);
hil-operator/hil skill wording matches actual fail-fast output; the
implementation plan is now a DO-NOT-EXECUTE historical record (banner +
checked boxes) so plan-executing agents cannot revert shipped files.
Verified: lock storm 1-winner-in-10, stale-record hold, closed-stdio
hold, dead-pid cleanup, CI-holder refusal, ghost-row 4-scenario merge,
unknown-board exit 1, py_compile + check.sh on all six workflows,
pre-commit clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Upj4hta5TNoAbidqeC1zZ6
Review-fix batch (owner-confirmed) on the multi-agent harness:
- board_lock: detach holder stdio so a captured `hold` cannot hang on the
daemon's inherited pipe; probe locks by holder-pid liveness instead of a
momentary flock, which could spuriously fail a concurrent acquirer
(storm-tested: 1 winner in 10, 0/15 acquire failures under probe storm)
- hil_test: locked board renders a visible board-locked fail row so the
report matches the exit code; stale marker cleared on a real re-run
- pr-babysit: autoPush now opt-in (default dry run); resolve recipe
paginates reviewThreads; post-push resolve gets issue-comment fallback
- validate: size stage honors non-default base via --base-branch; pvs
stage delegated to the new agent
- new static-analyzer agent (sonnet): PVS-Studio SAST+MISRA for one
board, structured findings gated on files changed vs base
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
Add worker agents (builder, port-dev, driver-reviewer, hil-operator,
pr-monitor), deterministic workflows (validate, fanout-dev, driver-review,
hil-validate, full-check, pr-babysit) and a /pre-pr gate skill, so sessions
can fan build/test/review/PR-triage work out to tiered subagents. pr-babysit
drives a PR to green: triage CI + bot reviews, fix validated findings, verify,
push, and reply-to + resolve each inline review thread (fixed or refuted).
Replace the stop-the-runner HIL discipline with per-board flock locks:
test/hil/board_lock.py plus a fail-open guard in hil_test.py let CI and dev
sessions share the rig per board (locked boards fail fast and re-run;
HIL_NO_BOARD_LOCK=1 is a user-authorized bypass). The actions-runner is
never stopped.
Design spec, implementation plan, and real-rig smoke evidence under
docs/superpowers/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
