mirror/tinyusb
3
0
Fork 0
mirror of https://github.com/hathach/tinyusb.git synced 2026-08-18 11:02:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

skills, docs: rewrite USB recovery from the live incidents

Browse Source 
Two things the rig taught us that the old guidance got wrong.

A usbfs ioctl wedged in D state cannot be freed on a running kernel. It holds
the device lock, so usb_disconnect() blocks behind it; reboot(2) walks
device_shutdown() and takes the same lock, so every userspace reboot stalls too.
Only sysrq b (emergency_restart, which skips device_shutdown) or hypervisor
action clears it -- all cited to the kernel source. The recovery ladder is
generic across rigs now (ci.lan, hifiphile, a bench PC) and ends at hypervisor
escalation only where host access exists. Two claims are corrected outright:
JLinkExe is NOT convoy-safe, and a park-flash cannot free a device-lock owner.

The hil skill's banner list is what an operator agent matches a report against,
so it enumerates the banners that actually exist, including the D-state note --
which is explicitly NOT a wedge, since a healthy in-flight testusb is
uninterruptible for most of every case and a concurrent CI battery would
otherwise turn a clean run red.
This commit is contained in:
hathach
2026-08-14 01:08:50 +07:00
parent 3963a1b70a
commit f822f69a98
15 changed files with 597 additions and 204 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
.claude/agents/hil-operator.md
View File

@ -18,19 +18,23 @@ The GitHub Actions runner keeps running during your work. Per-board flock locks
- `python3 test/hil/hil_test.py ...` runs: do NOT pre-hold those boards — `hil_test.py` self-locks each board for its flash+test and would fail fast with `board locked` against your own hold.
- ANY other hardware action (JLinkExe/openocd/GDB, manual flash, usbtest.py, serial poking): hold first, release when done — release is mandatory cleanup (a crashed holder auto-releases via kernel flock, but do not rely on it):
```bash
python3 test/hil/hil_lock.py hold <board...> --reason "<task>"
python3 test/hil/helper/hil_lock.py hold <board...> --reason "<task>"
# ... hardware work ...
python3 test/hil/hil_lock.py release <board...>
python3 test/hil/helper/hil_lock.py release <board...>
```
- Rig-wide operations (uhubctl power cycling, pci-rebind — they renumber buses): `python3 test/hil/hil_lock.py hold --all --reason "<why>"` first.
- Rig-wide operations (uhubctl power cycling, controller resets — they renumber buses): `python3 test/hil/helper/hil_lock.py hold --all --reason "<why>"` first.
- If a lock is already held by someone else: report holder/reason (`hil_lock.py status`) — never force, never kill the holder. If the holder's reason is `hil_test.py`, that is a concurrent CI job mid-test on the board: waiting a few minutes and retrying once is appropriate when your task allows; otherwise return the holder info so the orchestrator can ask the user.
- You cannot ask the user anything. Bypassing a lock (`HIL_NO_BOARD_LOCK=1`, or proceeding with manual hardware work despite a held lock) is allowed ONLY when your prompt explicitly states the user authorized forcing.
## Hard rules
- HIL runs take 2–5 min per board: use Bash timeouts >= 20 min (1200000 ms) and NEVER cancel early.
- HIL runs take 2-5 min per board, but a stuck fleet runs to `HIL_POOL_TIMEOUT` — 60 min
unless the env pins it; the run logs its guard in the startup line. That far exceeds the
Bash tool's 10 min foreground cap: run it in the background and wait
for the completion notification. A foreground timeout kills the run before hil_test.py
can write its report. NEVER cancel early.
- One hardware action at a time. You are never run concurrently with another hil-operator.
- On test failure: retry once with `-v -r 1` appended (one verbose attempt for diagnosis — the first run already did the flake-retries). If a board/fixture stops enumerating or tools hang in D state, consult usb-kernel-recover and capture `dmesg | tail -50` into `detail`; set `wedged` true.
- On test failure: retry once with `-v -r 1` appended (one verbose attempt for diagnosis; a usbtest battery that produced per-case verdicts is NOT auto-retried, so its result already stands). If a board/fixture stops enumerating, or a tool of YOURS hangs in D state, consult usb-kernel-recover and capture `dmesg | tail -50` into `detail`; set `wedged` true. A `> **Rig note.**` banner reporting someone else's D-state process is not that — see the hil skill's banner list.
## Output contract

2
.claude/agents/pr-monitor.md
View File

@ -9,7 +9,7 @@ You triage exactly one PR (number given in your prompt) using `gh`. You never mo
## CI triage
1. `gh pr checks <N>`. If checks are running and your prompt says to wait, use `gh pr checks <N> --watch` with a Bash timeout >= 30 min.
1. `gh pr checks <N>`. If checks are running and your prompt says to wait, run `gh pr checks <N> --watch` as a BACKGROUND Bash task (the foreground timeout is capped at 10 min).
2. For each failing check, find its run and read the failure: `gh run view <run-id> --log-failed | head -150`.
3. Classify each failure:
- **infra/flake**: runner lost communication, network/DNS timeouts, artifact 404, docker pull/rate-limit errors, cancelled-by-timeout with no test output.

4
.claude/agents/target-debugger.md
View File

@ -46,8 +46,8 @@ the next technique you would try.
## Lock discipline
- Hold the board lock for the WHOLE session (`hil_lock.py hold <board>
--reason "target debug: <bug>"`). Multi-hour holds are fine; never stop the
- Hold the board lock for the WHOLE session (`python3 test/hil/helper/hil_lock.py
hold <board> --reason "target debug: <bug>"`). Multi-hour holds are fine; never stop the
actions-runner. Locks held by others: report holder/reason, never force
unless your prompt states the user authorized it.
- `hil_test.py` self-locks: release your hold before any `hil_test.py` run,

2
.claude/skills/etm-trace/SKILL.md
View File

@ -43,7 +43,7 @@ this skill for exact counts, coverage, or instruction-by-instruction history.
capture script uses automation port **19201**, never an interactive Ozone's
19200.
- Hold the board lock (see the `hil` skill):
`python3 test/hil/hil_lock.py hold <board> --reason "etm capture"`.
`python3 test/hil/helper/hil_lock.py hold <board> --reason "etm capture"`.
- Committed `hw/bsp/**/ozone/*.jdebug` are the maintainer's interactive
projects — automation never opens them (Ozone rewrites project files); the
script generates a throwaway project.

17
.claude/skills/hil-pool-check/SKILL.md
View File

@ -5,7 +5,7 @@ description: Use when asked for a pool check or board/probe health scan on a Tin
# HIL Pool Check (board/probe health)
Health-scan the HIL board pool with `test/hil/hil_pool_check.py`: per board it checks the flash
Health-scan the HIL board pool with `test/hil/helper/hil_pool_check.py`: per board it checks the flash
probe is on the USB bus, flashes a light example (`device/dfu_runtime`; host-only boards get
`host/device_info`, verified by serial output), waits for the board's uid to re-enumerate,
applies safe per-device recovery (probe authorized-toggle, board reset), re-parks with
@ -21,19 +21,19 @@ pool check holds fails it as "board locked" — prefer running between CI runs.
A request for a "pool check" means the DEFAULT full check below. Use `--scan-only` only when the
user explicitly asks for a quick look, or when you have VERIFIED a CI sweep is mid-run right now
(`python3 test/hil/hil_lock.py status` shows `hil_test.py` holders) — "CI might be running" is not
(`python3 test/hil/helper/hil_lock.py status` shows `hil_test.py` holders) — "CI might be running" is not
that predicate: the full check is already lock-safe (CI-held boards report 🔒 locked and are never
touched), so an unconfirmed suspicion is no reason to downgrade. In either scan case say which
mode ran and why; never silently substitute the scan for the full check.
```bash
python3 test/hil/hil_pool_check.py # full check: ~10 s + ~1-2 s/board with firmware built;
python3 test/hil/helper/hil_pool_check.py # full check: ~10 s + ~1-2 s/board with firmware built;
# first run on an unbuilt tree takes minutes (it builds)
python3 test/hil/hil_pool_check.py --scan-only # USB presence only, <1 s, no locks/flashing/building
python3 test/hil/hil_pool_check.py -b BOARD [-b …] # subset; may name boards-skip (parked) entries
python3 test/hil/helper/hil_pool_check.py --scan-only # USB presence only, <1 s, no locks/flashing/building
python3 test/hil/helper/hil_pool_check.py -b BOARD [-b …] # subset; may name boards-skip (parked) entries
# from a dev PC, against the ci rig (bash -lc: flashers like STM32_Programmer_CLI live in ~/bin):
ssh ci.lan 'bash -lc "cd ~/code/tinyusb && python3 test/hil/hil_pool_check.py"'
ssh ci.lan 'bash -lc "cd ~/code/tinyusb && python3 test/hil/helper/hil_pool_check.py"'
```
## Notes
@ -45,8 +45,9 @@ ESP-IDF env (`get-idf`) for espressif — which also needs `esptool` on PATH (pi
`~/.local/bin/esptool`; a non-login shell may lack it — run via `bash -lc`). An explicit `-B` is
searched exclusively for *existing* firmware; builds still land in `cmake-build/` and are noted
`built <example>`. Espressif boards park too when the IDF env is present. A first run on an
unbuilt tree builds for many minutes: use a command timeout ≥ 30 min and NEVER cancel early — a
killed run leaves detached cmake/ninja children still writing to `cmake-build/`.
unbuilt tree builds for many minutes: the Bash tool caps a foreground timeout at 10 min, so run
it in the BACKGROUND and NEVER cancel early — a killed run leaves detached cmake/ninja children
still writing to `cmake-build/` with the board locks held under a protected reason.
Statuses: `ok` (flashed and verified; in `--scan-only` it only means the probe is present),
`flash-failed` (firmware delivery failed: probe missing, build failed, flasher error, silent

68
.claude/skills/hil/SKILL.md
View File

@ -1,6 +1,6 @@
---
name: hil
description: Use when running TinyUSB Hardware-in-the-Loop (HIL) tests on physical boards, debugging HIL failures, or copying firmware to the ci.lan test rig. Covers per-host config selection (infra rigs ci/tusb use tinyusb.json/hfp.json, any dev PC uses local.json), local and remote execution, the board-lock protocol, and debugging tips. For board/probe health scans ("pool check") use the hil-pool-check skill.
description: Use when running TinyUSB Hardware-in-the-Loop (HIL) tests on physical boards, when a HIL run fails, hangs, reports a board locked, or produces a report you need to interpret, or when copying firmware to a test rig (ci.lan, hifiphile/tusb, or a dev PC). For board/probe health scans ("pool check") use the hil-pool-check skill instead.
---
# Hardware-in-the-Loop (HIL) Testing
@ -26,28 +26,28 @@ The `ci` rig also hosts a GitHub Actions runner that flashes boards and runs HIL
- For hardware work outside `hil_test.py` (JLink/GDB, manual flashing, `usbtest.py`, serial poking), hold the lock first:
```bash
python3 test/hil/hil_lock.py hold BOARD [BOARD...] --reason "why"
python3 test/hil/helper/hil_lock.py hold BOARD [BOARD...] --reason "why"
# ... hardware work ...
python3 test/hil/hil_lock.py release BOARD [BOARD...]
python3 test/hil/helper/hil_lock.py release BOARD [BOARD...]
```
- Never pre-hold boards you are about to run `hil_test.py` on — it self-locks and would treat your own hold as a conflict.
- Rig-wide operations (uhubctl power cycling, pci-rebind — bus renumbering) affect every board: `hil_lock.py hold --all --reason "..."` first.
- Rig-wide operations (uhubctl power cycling, controller resets — bus renumbering) affect every board: `hil_lock.py hold --all --reason "..."` first.
- `hil_lock.py status` lists holders. Locks auto-release when the holder process dies (kernel flock); `/tmp` clears on reboot.
- Forcing past a lock: `HIL_NO_BOARD_LOCK=1 python3 test/hil/hil_test.py ...` bypasses the guard without killing the holder. Only with the user's explicit go-ahead — they accept the risk of colliding with whatever holds the board.
## Pool check (board/probe health)
Board/probe health scanning (`test/hil/hil_pool_check.py`) has its own skill: **hil-pool-check**.
Board/probe health scanning (`test/hil/helper/hil_pool_check.py`) has its own skill: **hil-pool-check**.
Use it before a HIL campaign, after rig maintenance/reboot, or when boards fail to flash.
## PR-scoped selection
`test/hil/hil_select.py` maps a diff to affected boards/tests (used by CI on PRs; fail-open
`test/hil/helper/hil_select.py` maps a diff to affected boards/tests (used by CI on PRs; fail-open
to the full matrix). Manual use:
```bash
SEL=$(python3 test/hil/hil_select.py --base master test/hil/tinyusb.json)
SEL=$(python3 test/hil/helper/hil_select.py --base master test/hil/tinyusb.json)
FULL=$(printf '%s' "$SEL" | python3 -c "import json,sys; print(json.load(sys.stdin)['full'])")
ARGS=$(printf '%s' "$SEL" | python3 -c "import json,sys; print(json.load(sys.stdin)['args']['tinyusb.json'])")
if [ "$FULL" = "True" ] || [ -n "$ARGS" ]; then
@ -60,7 +60,20 @@ fi
Read `full`, never `args` alone: `args` is empty for BOTH `full: true` (run the whole matrix — a broad or
unclassified change) and "nothing selected" (skip). Skip only when `full` is false AND `args` is empty.
Unit suite: `python3 test/hil/test_hil_select.py` (no hardware).
Unit suites (no hardware), all four run by the `hil-test`/`hil-select-test` pre-commit
hooks: `test_hil_select.py` covers only board selection. The containment work --- bounded
reads, the kill ladders, the build and pool guards --- lives in `test_hil_bounded.py`,
`test_hil_health.py` and `test_hil_util.py`, so run all four when changing `test/hil`:
`for f in test/hil/test/test_*.py; do python3 "$f"; done` (~55s).
## Pre-flight rig health check
`hil_test.py` notes any process already in D state when the run starts, as one line above
the table. It never aborts, and it is a hint rather than a diagnosis. What bounds a stuck
run is `HIL_POOL_TIMEOUT` plus the job's `timeout-minutes`; what diagnoses a wedged rig is
the `hil-pool-check` skill.
See the `usb-kernel-recover` skill for what a real wedge looks like and how to clear it.
## Prerequisites
@ -103,11 +116,44 @@ Env overrides: `REMOTE`, `REMOTE_DIR`, `CONFIG`. Fails fast if the build dir/rep
## Timing
Runs take 2-5 min. Use a timeout ≥ 20 min (1200000 ms). NEVER cancel early.
Runs take 2-5 min per board, but a stuck fleet runs to `HIL_POOL_TIMEOUT` — 60 min
unless the env pins it. The run logs its guard in the startup line; never declare a run
stuck before THAT value has elapsed.
The Bash tool caps a foreground timeout at 10 min, so **run it in the background** and
wait for the completion notification -- never a foreground timeout, which would kill
the run before its own guard can write a report. NEVER cancel early.
## Reporting
The user-facing answer to a HIL run IS the tool's summary table: paste the complete per-board
table (and footer counts) verbatim — never truncate rows or reduce it to a prose digest; at most
one line of commentary below it. On failure, retry with `-v`; if that's not enough, add temporary
debug prints to `hil_test.py`.
one line of commentary below it.
**First check what sits above the table.** Seven banners can appear there; match on a
PREFIX, since each carries trailing detail and one is a blockquote:
- `**HIL run abandoned: worker pool timed out after …s.**` — no results were collected this
attempt, so any table below is a PREVIOUS attempt's. Report the abandonment, never those
rows, and never `"pass": true`.
- `**HIL run aborted: a worker raised …**` — same rule: a worker crashed before results
were collected; any table below is stale. Report the abort, never the rows.
- `**HIL run abandoned: the worker pool would not shut down.**` — DIFFERENT: the table
below IS this run's, but the pool could not be shut down afterwards (the job exits
non-zero even if every board passed). Report the results AND the abandonment; never
`"pass": true`.
- `**HIL run selected no boards.**` — the filters intersected to nothing, so there is no
table at all. Report that (and the filter shown), never `"pass": true`.
- `> **Rig note.**` — a process was in D state when the run started. This is NOT a wedge:
a healthy in-flight testusb is uninterruptible for most of every case, and the rig
supports a dev run alongside CI. On its own it is never `wedged: true` and never turns a
green table into `"pass": false`. Mention it only when a board below failed, as the first
thing to check.
- `> **Rig dirty.**` — a process survived SIGKILL and still holds a probe or usbfs node
into the NEXT job. The table below is this run's and can be reported, but say the rig is
dirty: the next job starts degraded and nothing in the harness can clear it.
- `> **Not all verdicts are evidence.**` — one or more workers went blind on sysfs, so
"device not found" from the named boards means "could not tell". Do NOT report their red
cells as broken boards.
On failure, retry with `-v`; if that's not enough, add temporary debug prints to
`hil_test.py`.

2
.claude/skills/pre-pr/SKILL.md
View File

@ -15,7 +15,7 @@ Run the software + hardware gate for the current branch. The user invoking this
## 2. Map changes to boards
- `python3 test/hil/hil_select.py --base $BASE test/hil/tinyusb.json` → JSON with the affected
- `python3 test/hil/helper/hil_select.py --base $BASE test/hil/tinyusb.json` → JSON with the affected
bsp `families`, the affected rig `boards`, and per-file `reasons`. `full: true` means a
broad/infra change.
- Affected families = `families` ∪ the family of every name in `boards`. Neither half is

4
.claude/skills/target-debug/SKILL.md
View File

@ -30,9 +30,9 @@ Hold the board lock for the WHOLE manual session; never stop the
actions-runner (see the `hil` skill for the full lock protocol):
```bash
python3 test/hil/hil_lock.py hold <board> --reason "target debug: <bug>"
python3 test/hil/helper/hil_lock.py hold <board> --reason "target debug: <bug>"
# ... instrument / build / flash / capture / GDB ...
python3 test/hil/hil_lock.py release <board>
python3 test/hil/helper/hil_lock.py release <board>
```
Board → probe mapping: `test/hil/tinyusb.json` — `flasher.name` is the probe

286
.claude/skills/usb-kernel-recover/SKILL.md
View File

@ -1,136 +1,206 @@
---
name: usb-kernel-recover
description: Use when a USB device or fixture attached to the ci HIL rig's Linux host is stuck, hung, not enumerating, or wedged after a failed flash or test, or when processes touching USB (testusb, JLinkExe, uhubctl, libusb tools) start hanging in D state. Linux-kernel-side only — a bus owned by a TinyUSB host is out of reach (reset the target / cycle its VBUS instead); the rig's probes and serial fixtures always remain in scope.
description: Use when a USB device or fixture on a HIL rig's Linux host (ci.lan, hifiphile/tusb, a bench PC) is wedged, not enumerating, or when processes touching USB (testusb, JLinkExe, uhubctl, openocd, libusb tools) hang in D state. Linux-host side only — a bus owned by a TinyUSB host is out of reach.
---
# USB Recovery on the HIL Rig (Linux kernel side)
Run this skill's `scripts/usb_recover.sh` with `sudo`. It wraps the sysfs reset
actions, a uhubctl power-cycle escalator, and a resolver:
**The rule:** a wedged usbfs ioctl holds that device's `device_lock`
(`usbdev_do_ioctl` takes `usb_lock_device`, the uninterruptible variant —
v6.12.96 devio.c:2609) and the driver under it waits in a plain
`wait_for_completion()` with no timeout (usbtest.c:1404; `usb_sg_wait`,
message.c:765). Nothing that also takes that lock can help. Only two levers
don't: **failing the URB at the device** (rung 1) and **the port-side data-line
drop** (rung 2).
## 1. Triage: find the holder
```bash
# all examples below abbreviate: sudo .claude/skills/usb-kernel-recover/scripts/usb_recover.sh
sudo usb_recover.sh resolve /dev/ttyACM3 # /dev node -> busport (e.g. 3-4.7); also ttyUSB*, sg*
sudo usb_recover.sh authorized <busport> # deauthorize+reauthorize: re-enumerate, no VBUS cut
sudo usb_recover.sh rebind <busport> # usb driver unbind+bind: re-probe
sudo usb_recover.sh hub-cycle <busport> # uhubctl VBUS cycle of the feeding port, walking parent hub
# -> root port until the device re-enumerates
sudo usb_recover.sh root-cycle <busport> [serial] # uhubctl VBUS cut straight at the ROOT port (real ppps), no
# leaf walk, no device-lock touch: the D-state cure.
# [serial] is checked and a mismatch refused.
sudo usb_recover.sh pci-rebind <pciaddr> # whole HCD controller unbind+bind, e.g. 0000:02:00.0
sudo usb_recover.sh pci-bind <pciaddr> [drv] # re-bind a DRIVERLESS controller (auto-tries xHCI drivers)
ps -eo pid,stat,etimes,wchan:22,args | awk '$2 ~ /D/'
sudo cat /proc/<pid>/stack # never opens the node, so it cannot block
```
`hub-cycle` caveats: leaf hubs that gang (or fake) port power switching bounce
**all siblings** on that hub when cycled; a **self-powered** leaf hub keeps
downstream VBUS up, so cycling it only resets its uplink — that's why the walk
escalates to the root port, where the Renesas cards' per-port power (ppps) is
real. A device that is wedged but bus-powered from a switching hub gets a true
power cycle; one on a self-powered hub may only get a re-enumeration.
- **`S` = victim.** Lock-taking sysfs *reads* use `usb_lock_device_interruptible`
(sysfs.c:124-139, 11 sites), so readers are killable and `timeout` bounds them.
Ignore them; they unwind by themselves.
- **`D` = the holder, or a writer that took the uninterruptible path.**
## Decide first: is anything stuck in D state?
| Stack shows | Meaning | Go to |
|---|---|---|
| `usbdev_ioctl` + a driver module (`[usbtest]`) | **owner, holds the lock** | rung 3 — terminal |
| `usbdev_ioctl`, no driver frames | owner waiting on a URB | rung 1 (DUT) / rung 2 (probe) |
| `usbdev_open`, sysfs reads | victim | ignore |
| `tee .../usbtest/new_id`, `bind`, `unbind` | **victim that SPREADS it** | stop issuing them |
| `hub_event` in a kworker | teardown stuck behind an owner | rung 3 |
Driver-bind writes are not passive: `__device_driver_lock` (drivers/base/dd.c)
takes `device_lock()` uninterruptibly **and `device_lock(parent)`**, because
`usb_bus_type` sets `.need_parent_lock = true` (driver.c:2048) — each one holds
the HUB's lock, which is how one wedged port takes a whole bus down.
Map the holder to a busport with **lock-free attrs only** (`devnum`, `idVendor`,
`idProduct` are `usb_descriptor_attr*`, plain `sysfs_emit`, sysfs.c:688-705):
```bash
ps -eo pid,stat,wchan:30,cmd | awk '$2 ~ /D/'
for d in /sys/bus/usb/devices/<bus>-*/; do
[ "$(cat $d/devnum)" = "<devnum>" ] && echo "$d $(cat $d/idVendor):$(cat $d/idProduct)"
done
grep -l <SERIAL> /sys/bus/usb/devices/*/serial # only on a HEALTHY device
```
**If yes** (uninterruptible sleep, typically a usbfs ioctl — e.g. testusb inside
`usb_sg_wait`): cut VBUS at the root port, and nothing else.
## 2. Shield first (prerequisite for anything using libusb)
A wedged device blocks every enumerator that reads its locking attributes —
JLinkExe, uhubctl, openocd's HID fallback. `chmod 000` makes the VFS reject the
read before `->show()` runs, so they skip it and keep enumerating:
```bash
sudo usb_recover.sh root-cycle <busport> # e.g. 11-3.7 -> cycles bus 11 root port 3
for f in bNumInterfaces bmAttributes bMaxPower configuration bConfigurationValue \
product manufacturer serial avoid_reset_quirk; do
sudo chmod 000 /sys/bus/usb/devices/<busport>/$f
done
```
This drops power to the wedged device, so its in-flight URB fails and the ioctl
returns. It targets the *root hub* — a different USB device from the wedged one —
and never *writes* the wedged device's sysfs. It reads a few attributes from it —
`idVendor`/`idProduct`/`serial`/`product` to report and check the target, and the
directory inode plus `devnum` afterwards — none of which take the device lock, so
it does not join the convoy the way `authorized`/`rebind`/`pci-rebind` do.
Recovery is proven by that inode changing — a real disconnect destroys the
kobject and reconnecting creates a new one, whereas a disconnect blocked on the
device lock leaves it untouched. It exits non-zero if the device does not come
back; a **zero exit only means it re-enumerated**, so still confirm the D-state
process actually let go. Pass the expected serial as a third argument and it
refuses a busport that now names a different device.
- Shield the **leaf, its parent hub, and the root hub** (`usb<N>`) — a stuck
uhubctl locks the root hub too.
- **Run the recovery tool as NON-root**: root has `CAP_DAC_OVERRIDE`, ignores the
`000`, and blocks anyway.
- **Only those nine.** `descriptors`, `busnum`, `devnum`, `speed`, `idVendor`,
`idProduct` are lock-free and libusb needs them; a blanket `chmod` breaks
enumeration instead of fixing it.
- `chmod` never blocks (inode setattr, no `show()`), so it works on a fully
wedged device.
- **Not needed for openocd pinned with `vid_pid`** — it matches the cached
descriptor and skips a foreign device before `libusb_open`
(cmsis_dap_usb_bulk.c:107, bulk backend; the HID fallback ignores the pin).
- Leaf shields vanish on re-enumeration; **the root hub's must be restored**:
`sudo chmod "$(stat -c %a /sys/bus/usb/devices/usb<healthy>/$f)" …/usb<N>/$f`
It bounces **every fixture under that root port** — on ci that is up to 25
devices. Hold the affected boards' locks first if you can, but note
`hil_lock.py` uses `LOCK_EX | LOCK_NB` and so fails immediately when CI already
holds them; there is no wait-for-lock. When CI is mid-run you are choosing
between bouncing its fixtures and leaving the bus wedged for everything. The
automated path in `usbtest.py` takes no locks at all and accepts that collateral
deliberately: by the time a D-state wedge exists the convoy will take the bus
down anyway.
## 3. The rungs — go straight to the one triage names
(The VBUS mechanism is verified on the ci rig — the leaf hubs report
`bmAttributes=e0`, "self-powered", but are physically bus-powered with no adapter,
so a root-port cut really does kill downstream power. Do not re-derive this from
the descriptor; it lies. Not yet confirmed against a live D-state wedge. If
`uhubctl` itself hangs, the convoy has already spread — escalate.)
If `root-cycle` does not free the D-state process, there is no software cure
left: ask the operator for a full PVE **host** power cycle. A VM reboot is NOT
reliable (downstream hubs can latch up across the PCIe reset and need a physical
replug), and a graceful reboot stalls on the D-state process anyway. Do NOT fall
through to `pci-rebind` (see next).
**`pci-rebind` can strand the controller driverless.** Its unbind succeeds but,
with a D-state process still holding a URB, the *re-bind* hangs — leaving the
PCI device with **no driver** (`/sys/bus/pci/devices/<addr>/driver` gone) and the
whole controller's fixtures offline. A second `pci-rebind` then dies with "no
driver bound". Recover with `pci-bind <addr>` (re-attaches the xHCI driver);
if that also hangs because the D-state URB is unkillable, only a full PVE host
power cycle (operator action) recovers. The Renesas binds via `xhci-pci-renesas` (firmware loader), others via
`xhci_hcd` — `pci-bind` auto-tries both, or pass the driver explicitly.
**Ordering is critical.** `authorized`/`rebind`/`pci-rebind` all take the
per-device lock the stuck ioctl holds — they block and join the convoy, and
soon every libusb tool (uhubctl, JLinkExe) hangs too. Worse, a blocked
`pci-rebind` grabs the PCI device lock on its way in and can wedge the whole
function, after which **only a full PVE host power cycle recovers**. `root-cycle`
first, and never `pci-rebind` a D-state wedge.
**If no** (device merely dead or silent), escalate gently:
1. `authorized <busport>` — re-enumerates just that device
2. `rebind <busport>` — re-probe; also worth trying on the parent hub's busport
3. `hub-cycle <busport>` — VBUS cycle of the feeding port, walking up to the
root port; may bounce sibling fixtures on ganged hubs
4. `pci-rebind <pciaddr>` — last resort: bounces every fixture on that controller
## Finding targets
**Rung 1 — wedged DUT: reset it through its own probe.**
```bash
grep -l <SERIAL> /sys/bus/usb/devices/*/serial # serial -> busport (dir name)
readlink -f /sys/bus/usb/devices/usb<N> # bus N -> its PCI addr in the path
printf "r\ng\nq\n" > /tmp/rec.jlink
JLinkExe -device <DEV> -if SWD -speed 4000 -SelectEmuBySN <probe-sn> \
-autoconnect 1 -nogui 1 -CommandFile /tmp/rec.jlink
```
Rig layout (2026-07-15, two Renesas uPD720201 cards; bus numbers renumber every
boot — re-derive with `readlink`): AMD `0000:02:00.0` = the debug-probe tree
(J-Links, ST-Links, WCH-Links), no port power switching; Renesas `0000:01:00.0`
and `0000:03:00.0` = DUT device hubs + serial fixtures, and ALL their root-hub
ports have real per-port power (`ppps`, 4+4 each) — `sudo uhubctl -l <bus> -p
<port> -a cycle` cuts VBUS to the leaf hub on that port. The 1a40:0201 leaf
hubs themselves claim "ganged" switching but do not actually cut power.
Reset **before** park-flash: non-destructive (the firmware under test survives
for autopsy), no flash wear, and no bad park image — a `wfe`/`wfi` park has
bricked SWD on mimxrt1064_evk and max32666fthr through a power cycle.
`ResetTarget` measures 128-129 ms; cleared 57 → 0, 26 → 0 and 5 → 0 D-state
processes, single shot each. Mechanism: chip reset drops the pull-up →
`usb_hcd_flush_endpoint` unlinks the URB `-ESHUTDOWN` (hcd.c:1783) → the
completion fires → the ioctl returns → the lock releases.
Works on i.MX RT (`USBCMD.RS` = 0 detaches, RT1050 RM Rev 3 p.2453) **and on
DWC2** — measured 2026-08-16 on stm32f407disco: `r; g` gave
`usb 13-2.2: USB disconnect, device number 107`, re-enumerating 325 ms later.
(A bare **halt** does not: the core keeps running with the pull-up asserted.)
**Park-flash** (`--recover-board`/`--recover-fw`, what `usbtest.py` automates) is
the fallback where the reset cannot reach the peripheral. Delivery must be
convoy-safe: **openocd pinned with `vid_pid`**, or esptool (`-p <ttyACM>`).
JLinkExe selects by serial, which needs `libusb_open`, so it needs the shield.
**Rung 2 — wedged PROBE: `root-cycle`.** A probe has no probe to reset it, so the
port-side drop is the only lock-free lever left. It commands the ROOT hub and
never touches the wedged device's lock.
```bash
sudo usb_recover.sh root-cycle <busport> [expected-serial]
```
Bounces **every fixture under that root port** (up to 25 here). Renesas `ppps`
disables D+/D− only — VBUS stays up, so it is a forced re-enumeration, not a
power cycle. Success is the sysfs inode changing, not uhubctl's exit code.
**Rung 3 — terminal case: a driver ioctl that OWNS the lock.** No software cure:
the task is uninterruptible and SIGKILL is queued, not delivered. Reboot with
**sysrq**, never `reboot(2)` — a graceful reboot runs `device_shutdown()`, which
takes every device lock and stalls on the wedged one.
```bash
echo b | sudo tee /proc/sysrq-trigger # after: sync; sudo umount -a
```
**Rung 4 — hypervisor.** ci.lan only, and never needed in eight recorded wedges:
`qm stop <vmid> && qm start <vmid>` from the PVE host. A VM *reboot* is not
reliable — hubs can latch across the PCIe reset.
## 3b. If the CONTROLLER is dead, not a device
Signature: `xhci-pci-renesas <addr>: Timeout while waiting for setup device
command`, devices on that controller failing to enumerate, or its buses gone —
as opposed to ONE device wedged. The rungs above cannot help; the controller
itself needs re-initialising.
```bash
sudo usb_recover.sh pci-rebind <pciaddr> # unbind + bind the whole xHCI
sudo usb_recover.sh pci-bind <pciaddr> # only if it ends up driverless
```
Measured on ci.lan 2026-08-17 02:34:41 after a `hub-cycle` failed to take: unbind
deregistered buses 17 and 18, the re-bind registered new buses **1 and 2** one
second later, and every fixture re-enumerated. **It renumbers every bus that
controller owns**, so hold all affected boards' locks first (`hil_lock.py hold
--all`) and re-derive busports afterwards.
Do NOT reach for it while a device-lock convoy is live — see Common mistakes.
## 4. If nothing is in D state
The device is dead or silent, not wedged. `sudo usb_recover.sh authorized
<busport>` unconfigures and reconfigures it (`usb_set_configuration(dev, -1)`
then re-choose, hub.c) — it fixes stale driver/interface state, does **not**
replug: the `usb_device` survives, so most probes keep their sysfs node. If that
does not take, the device is wedged rather than silent — go to rung 1 or 2.
`resolve <dev-node>` maps `/dev/ttyACM3` → busport.
**It takes `usb_lock_device` uninterruptibly** (hub.c `usb_deauthorize_device`),
so it is safe only while nothing is in D state.
## 5. Before declaring the rig healthy
```bash
ps -eo stat,args | awk '$1 ~ /D/' | wc -l # must be 0
timeout 15 lsusb # rc 0 and a sane device count
sudo uhubctl -l <bus> -p <port> # "0000 off" = never came back
sudo uhubctl -l <bus> -p <port> -a on
```
Observed: 5 boards missing with a completely clean D-state list, because
`usb17-port2` sat at `disable=1`.
## Common mistakes
- `resolve` takes a **/dev node**, not a busport or serial ("no such device node").
- `authorized`/`rebind`/`hub-cycle`/`root-cycle` take a **busport** (`3-4.7`);
`pci-rebind`/`pci-bind` take a **PCI addr**.
- Command produces no output and doesn't return → it is blocked on the device
lock: a D-state holder exists; see above.
- Trying `pci-rebind` on a D-state hang — its re-bind hangs and strands the
controller **driverless**; recover with `pci-bind <addr>`, or a PVE host power
cycle if the D-state URB is unkillable. Use `root-cycle` for D-state, never
`pci-rebind`.
- Writing `/sys/bus/pci/devices/<addr>/reset` because the attribute is there. No
rig controller has FLR, so it becomes a PCIe bus reset that resets the xHCI
behind its live driver — the write succeeds, the card is halted for good, and
only a PVE host power cycle brings it back. Use `root-cycle`.
- `root-cycle` bounces **every** fixture under that root port, not just the target
— hold the sibling boards' locks first.
- A J-Link reset (`r; go`) does not disconnect a wedged DUT from the host: the
DWC2 soft-connect pullup stays up through a core halt, so stuck URBs stay stuck.
- **`uhubctl -a cycle` on a root port without `-S`.** It writes sysfs
`disable`, and `disable_store` takes `usb_lock_device(hdev)` uninterruptibly
then calls `usb_disconnect(child)` inside it (port.c) — against a wedged child
that blocks while holding the root hub's lock, poisoning the bus.
`usb_recover.sh` passes `-S`.
- **`echo 1 > .../remove`** to make a wedged device "go away": `remove_store` is
the one attribute in sysfs.c taking the uninterruptible `usb_lock_device`
(sysfs.c:765). It joins the convoy instead of clearing it.
- **`authorized` on anything wedged** — same uninterruptible lock. Driver
unbind/bind (`/sys/bus/usb/drivers/usb/{unbind,bind}`) does the same
unconfigure/reconfigure via `usb_generic_driver_disconnect` (generic.c) but ALSO
takes the parent hub's lock (`need_parent_lock`), so it is strictly worse; it was
removed from `usb_recover.sh` for that reason.
- **`pci-rebind` for a wedged DEVICE.** It is the cure for a dead CONTROLLER (see
below), not for a device-lock convoy: with a live D-state URB the re-bind can
hang and leave the controller with **no driver** and every fixture offline
(observed once). Recover that with `pci-bind <addr>`.
- **Writing `/sys/bus/pci/devices/<addr>/reset`** — no rig controller has FLR, so
it becomes a bus reset behind a live driver: card halted, host power cycle.
- **Resetting a victim's board.** Two boards were reset innocently before anyone
found the holder. Map by `devnum`, not by which board "should" be running.
- **Assuming one controller.** Observed: 26 D-state processes across three xHCI
controllers, all cleared by one probe reset on one device.
## Rig layout (ci.lan, bus numbers renumber every boot)
`readlink -f /sys/bus/usb/devices/usb<N>` → its PCI address. AMD `0000:02:00.0`
has no port-power switching; Renesas `0000:01:00.0` (probe tree) and
`0000:03:00.0`/`0000:05:00.0` (DUT hubs) have real per-port `ppps`.

122
.claude/skills/usb-kernel-recover/scripts/usb_recover.sh
View File

@ -4,21 +4,15 @@
#
# Usage:
# sudo usb_recover.sh authorized <busport> # e.g. 3-2 -> deauthorize+reauthorize (re-enumerate, NO VBUS cut)
# sudo usb_recover.sh rebind <busport> # e.g. 3-2 -> usb driver unbind+bind (re-probe)
# sudo usb_recover.sh pci-rebind <pciaddr> # e.g. 0000:01:00.0 -> HCD unbind+bind (WHOLE controller)
# sudo usb_recover.sh pci-bind <pciaddr> [driver] # bind a DRIVERLESS controller (e.g. after a pci-rebind
# # whose re-bind hung and left it unbound). Auto-tries the xHCI
# # drivers (xhci-pci-renesas, xhci_hcd) unless one is named.
# sudo usb_recover.sh hub-cycle <busport> # e.g. 13-1.6 -> uhubctl power-cycle of the port feeding it,
# # walking upstream (parent hub -> root port) until the device
# # re-enumerates. Ganged/fake-switching hubs may bounce ALL
# # siblings; self-powered hubs only reset their uplink, which
# # is why the walk ends at the root port (real xHCI ppps).
# sudo usb_recover.sh root-cycle <busport> [serial] # e.g. 13-1.6 -> uhubctl VBUS cut at the ROOT port feeding
# sudo usb_recover.sh root-cycle <busport> [serial] # e.g. 13-1.6 -> uhubctl port-off/on at the ROOT port feeding
# # it; [serial] is verified against the device and refused on mismatch,
# # skipping the leaf hubs (which fake ganged switching and do not
# # actually cut power). Bounces every sibling under that root port.
# # The D-state escape: no device lock, so it cannot convoy.
# sudo usb_recover.sh pci-rebind <pciaddr> # e.g. 0000:05:00.0 -> unbind+bind the whole xHCI
# # controller. For a DEAD CONTROLLER, not a wedged
# # device: it renumbers every bus it owns.
# sudo usb_recover.sh pci-bind <pciaddr> [drv] # re-attach a driver to a DRIVERLESS controller
# sudo usb_recover.sh resolve <devnode> # e.g. /dev/ttyACM3 -> print its <busport> (no privilege needed)
set -euo pipefail
@ -28,6 +22,34 @@ DRIVER_RE='^[A-Za-z0-9_-]+$'
die() { echo "usb_recover: $*" >&2; exit 1; }
lock_read() {
# Read an attribute served under the device lock (serial, product) with a 2s bound.
# Prints the value, '' when the attribute is absent, or '?' when it did not answer.
#
# Bounding these is load-bearing, not defensive: they are the FIRST thing root-cycle
# does, so on a real wedge an unbounded read blocks before reaching uhubctl at all
# (observed live: one attempt sat 3h; three concurrent invocations all frozen there).
# The operator then reads that as "recovery didn't work" and escalates to a bare
# `uhubctl -a cycle`, which tears the subtree down and blocks holding the ROOT HUB
# lock -- taking the whole bus with it. That is how one wedge becomes an incident.
#
# `timeout` is enough, though this said for a while that it was not (claiming the read
# sits in D state, where SIGKILL is not delivered, so timeout waitpid()s forever). It
# does not: v6.12.101 drivers/usb/core/sysfs.c takes the lock for every READ through
# usb_lock_device_interruptible -> device_lock_interruptible -> mutex_lock_interruptible,
# so the waiter sleeps INTERRUPTIBLY and SIGTERM ends it. Uninterruptible is the usbfs
# ioctl HOLDER, not us. The abandon-a-background-reader dance that claim justified is
# gone, and with it a fail-open where an absent attribute answered '?' -- the wedge
# signature, which root-cycle reads as "cannot confirm serial, proceed".
local v rc=0
# `|| rc=$?`, never a bare assignment: under this script's `set -e` a command
# substitution that FAILS (an absent attribute -- most hubs and probes have no
# iSerialNumber, and `product` is often missing) exits the whole recovery script.
v=$(timeout 2 cat "$1" 2>/dev/null) || rc=$?
[ "$rc" -eq 124 ] && { echo '?'; return; } # timed out: nobody answered
printf '%s\n' "$v"
}
# Generation marker for "did this device actually re-enumerate". A real disconnect destroys the
# usb_device and its sysfs kobject; reconnecting creates a new one, and kernfs hands out inode
# numbers monotonically, so the directory inode changes. Verified on the rig: ports re-enumerated
@ -49,9 +71,6 @@ die() { echo "usb_recover: $*" >&2; exit 1; }
# The trailing slash is load-bearing: /sys/bus/usb/devices/<busport> is a SYMLINK with its own
# separate inode, so without it stat reports the link rather than the device it points at, and the
# value would never change. Do not "tidy" it away.
sysfs_gen() { stat -c %i "/sys/bus/usb/devices/$1/" 2>/dev/null || echo none; }
usage() { grep -E '^# sudo usb_recover' "$0" >&2; exit 2; }
# Refuse to touch a PCI function that is not a USB controller (class 0x0c03xx), so a stray or
# mistyped BDF can't unbind/reset an unrelated device (storage, NIC) on a shared HIL host.
require_usb_controller() {
@ -60,6 +79,9 @@ require_usb_controller() {
[[ "$cls" =~ ^0x0c03 ]] || die "$addr is not a USB controller (class $cls); refusing"
}
sysfs_gen() { stat -c %i "/sys/bus/usb/devices/$1/" 2>/dev/null || echo none; }
usage() { grep -E '^# sudo usb_recover' "$0" >&2; exit 2; }
# Resolve a /dev node (ttyACMx, ttyUSBx, sgN, ...) up to its USB device busport.
resolve() {
local node=$1 syspath dev
@ -89,13 +111,6 @@ case "$action" in
echo 0 > "$d/authorized"; sleep 1; echo 1 > "$d/authorized"
echo "re-authorized $target"
;;
rebind)
[[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target"
[ -e "/sys/bus/usb/devices/$target" ] || die "no such usb device: $target"
echo "$target" > /sys/bus/usb/drivers/usb/unbind; sleep 1
echo "$target" > /sys/bus/usb/drivers/usb/bind
echo "rebound $target"
;;
pci-rebind)
[[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target"
require_usb_controller "$target"
@ -128,39 +143,10 @@ case "$action" in
die "could not bind $target with a known xHCI driver; pass the driver explicitly"
fi
;;
hub-cycle)
[[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target"
UHUBCTL=$(command -v uhubctl || echo /sbin/uhubctl)
[ -x "$UHUBCTL" ] || die "uhubctl not installed"
# sysfs generation, not node existence: a disconnect blocked on the device lock leaves the
# old node (and its idVendor) in place, so an existence check reports success without anything
# having happened -- and the walk to the root port, which is the part that actually cuts power
# on these fake-ganged leaf hubs, would never run.
gen=$(sysfs_gen "$target")
dev="$target"
while :; do
if [[ "$dev" =~ ^([0-9]+)-([0-9]+)$ ]]; then # parent is the root hub
loc="${BASH_REMATCH[1]}"; port="${BASH_REMATCH[2]}"; up=""
else # parent is a downstream hub
loc="${dev%.*}"; port="${dev##*.}"; up="$loc"
fi
echo "hub-cycle: power-cycling hub $loc port $port (feeds $dev)"
"$UHUBCTL" -l "$loc" -p "$port" -a cycle -d 5 -f || echo " (uhubctl failed at $loc; walking up)"
for _ in $(seq 1 10); do
sleep 1
now=$(sysfs_gen "$target")
if [ "$now" != none ] && [ "$now" != "$gen" ]; then
echo "recovered: $target re-enumerated (gen $gen -> $now)"; exit 0
fi
done
[ -n "$up" ] || break
dev="$up"
done
die "hub-cycle: $target still not enumerated after cycling up to the root port"
;;
root-cycle)
# VBUS cut at the ROOT port, where xHCI ppps is real. Unlike hub-cycle this does not walk up
# from the leaf (the 1a40:0201 hubs claim ganged switching but never cut power) and never
# Port-off/on at the ROOT port. NOTE: the Renesas ppps only disables D+/D- (VBUS stays up),
# so this is a forced re-enumeration, not a power cycle. It goes straight at the root port --
# no leaf walk (the 1a40:0201 hubs claim ganged switching but never cut power) -- and never
# writes the wedged device's sysfs or takes its lock, so it cannot join a D-state convoy.
# uhubctl exits 0 even when it does nothing ("No compatible devices detected" still returns
# 0), so its status proves nothing -- the sysfs_gen check below is the only real verdict.
@ -174,21 +160,33 @@ case "$action" in
# wrong target is at least visible.
[ -e "/sys/bus/usb/devices/$target" ] || die "no such usb device: $target"
idf="/sys/bus/usb/devices/$target"
serial=$(cat "$idf/serial" 2>/dev/null || echo -)
serial=$(lock_read "$idf/serial")
expect=${3:-}
[ -z "$expect" ] || [ "$expect" = "$serial" ] || \
if [ -n "$expect" ] && [ "$serial" = '?' ]; then
# Warn and PROCEED: an unreadable serial is the wedge signature itself, so refusing
# here would block the cure on exactly the condition it exists for. The identity
# guard is lost for this call -- say so, because the cost of a wrong target is the
# whole subtree.
echo "root-cycle: WARNING $target's serial did not answer (it is wedged), so '$expect'" \
"could NOT be confirmed; proceeding, but verify the busport if siblings drop" >&2
elif [ -n "$expect" ] && [ "$expect" != "$serial" ]; then
die "root-cycle: $target has serial '$serial', expected '$expect' — stale busport, refusing"
fi
# idVendor/idProduct are usb_descriptor_attr_le16: served WITHOUT the device lock, so
# a plain cat is safe on a wedged device. serial/product are usb_string_attr and are not.
echo "root-cycle: target $target is $(cat "$idf/idVendor" 2>/dev/null || echo -):$(cat "$idf/idProduct" 2>/dev/null || echo -)" \
"serial=$serial product=$(cat "$idf/product" 2>/dev/null || echo -)"
"serial=$serial product=$(lock_read "$idf/product")"
bus=${target%%-*}; rest=${target#*-}; rootport=${rest%%.*}
gen=$(sysfs_gen "$target")
echo "root-cycle: cutting VBUS on bus $bus root port $rootport (feeds $target, bounces its siblings)"
# -S is load-bearing. By default uhubctl writes /sys/.../usb<bus>-port<n>/disable (verified:
echo "root-cycle: disabling D+/D- on bus $bus root port $rootport (no VBUS cut; feeds $target, bounces its siblings)"
# -S is load-bearing. By default uhubctl writes /sys/.../usb<bus>-port<n>/disable (observed:
# two O_WRONLY opens per cycle), and the kernel's disable_store() takes the ROOT HUB's lock and
# synchronously usb_disconnect()s the child BEFORE cutting power -- against a wedged device that
# blocks on the lock we are trying to free, so power would never drop and uhubctl would D-state
# holding the root hub's lock, poisoning the whole bus. -S forces the libusb path, which sends
# the power-off control transfer straight to the root hub with no child-disconnect in front.
# synchronously usb_disconnect()s the child BEFORE cutting power -- confirmed in v6.12.96
# drivers/usb/core/port.c: usb_lock_device(hdev), the UNINTERRUPTIBLE variant, then
# usb_disconnect(&port_dev->child) inside it. Against a wedged device that disconnect blocks on
# the very lock we are trying to free, so power never drops and uhubctl D-states holding the
# root hub's lock, poisoning the whole bus. -S forces the libusb path, which sends the
# power-off control transfer straight to the root hub with no child-disconnect in front.
"$UHUBCTL" -S -l "$bus" -p "$rootport" -a cycle -d 5 \
|| die "uhubctl failed to cycle bus $bus port $rootport"
for _ in $(seq 1 10); do

30
.claude/skills/usbtest/SKILL.md
View File

@ -29,6 +29,11 @@ python3 test/hil/usbtest.py --serial <uid> --keep-binding --tests 29 # one case
```
- **Always `--keep-binding`**: the cleanup unbind path has wedged host xHCIs (`usb_hcd_alloc_bandwidth`).
- CI (`hil_test.py`) additionally passes `--budget`, `--outer-timeout` and
`--recover-board`/`--recover-fw`: on a HUNG case the battery aborts, RESETS the DUT
through its roster probe (non-destructive, ~130 ms) and reflashes only if that does not
clear the wedge (see usb-kernel-recover). Manual runs without those flags leave a HUNG
device wedged and skip cleanup — expected; reset or reflash it yourself.
- Always settle a few seconds after flashing — enumeration can bounce once; testusb into the gap sees
the device drop mid-case.
- On a CI rig: stop the actions runner before touching hardware; restart after. Never run two
@ -87,6 +92,28 @@ python3 test/hil/usbtest.py --serial <uid> --keep-binding --tests 29 # one case
| 5 | EIO — iso packet errors (check `dmesg`: "N errors out of M") |
| 71 | EPROTO — device answered wrong / too slow (after HC retries) |
**Step 0 — read what the case actually does.** The kernel module is ground truth;
the table above is a summary. Do this before theorising, and always before deciding
whether a hung case is recoverable. Fetch the rig's exact version (`uname -r`):
```bash
curl -sO "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/plain/drivers/usb/misc/usbtest.c?h=v6.12.96"
# case N lives under `case N:` in usbtest_do_ioctl(); tools/usb/testusb.c maps the flags:
# -c = param.iterations, -s = param.length, -g = param.sglen (NOT what they read like)
```
- **Real traffic and pass criteria.** Case 24 at `-c 256 -s 1024 -g 8` is 256 rounds
of 8 bulk-OUT URBs, unlinking `urbs[num-4]`/`urbs[num-2]` and requiring
`-ECONNRESET` on those two plus normal completion on the other 6 — not the
"256 URBs" the flags suggest.
- **Whether the wait is bounded** — decisive for recovery. `simple_io` uses
`wait_for_completion_timeout` (:481); the unlink paths use a bare
`wait_for_completion` (:1502, :1615). A device stalling there wedges the ioctl in
**D state permanently** — it holds the device lock, so nothing recovers it
(usb-kernel-recover, "The terminal case"). Knowing this first stops you burning
the rig on attempts that cannot work.
- **Which DCD path is implicated**, precisely rather than by category.
1. `usbtest.py` per-case output + its captured `dmesg` (`TEST n` markers bracket each case).
2. **usbmon** (`usbmon` skill): URB-level ground truth. **It cannot show data toggles or NAKs** —
a toggle desync and a dead endpoint look identical (Submits without Completes); distinguish
@ -121,3 +148,6 @@ python3 test/hil/usbtest.py --serial <uid> --keep-binding --tests 29 # one case
- "It works on gcc" → clang/IAR/LTO/make still pending.
- "Fixed iso IN" → apply the same exemption to iso OUT (toggle logic is symmetric).
- A clean single-board run does not validate concurrent/fleet behavior — batteries serialize.
- Reasoning about a case from its name or table row → open `usbtest.c` (step 0). The
flags don't mean what they look like, and recoverability is a property of that
case's wait, not of the rig.

4
.claude/workflows/hil-validate.js
View File

@ -26,8 +26,8 @@ const runBoard = (b) => agent(
? 'THE USER HAS EXPLICITLY AUTHORIZED FORCING: run hil_test.py with HIL_NO_BOARD_LOCK=1 in the environment (bypasses the board lock check; do NOT release or kill the existing holder). '
: 'If the run fails because the board lock is held (a dev session or concurrent CI job), report pass=false and set detail to start EXACTLY with "board locked:" followed by the holder JSON verbatim — never force the lock. ') +
'Reserve the phrase "board locked" strictly for lock contention; describe a frozen or non-enumerating board as "unresponsive" instead. ' +
`Firmware is in examples/cmake-build-${b}. Use the config for this host (hostname first), single-board flag -b ${b}, Bash timeout >= 20 min, never cancel early. ` +
'On non-lock failures retry once with -v -r 1 (one verbose attempt for diagnosis — the first run already did the flake-retries). wedged=true if the board/fixture is unresponsive after the run (capture dmesg | tail -50 into detail).',
`Firmware is in examples/cmake-build-${b}. Use the config for this host (hostname first), single-board flag -b ${b}. Run hil_test.py as a BACKGROUND Bash task and wait for it (a stuck fleet runs to its pool guard, 60 min by default — beyond any foreground timeout); never cancel it early. ` +
'On non-lock failures retry once with -v -r 1 (one verbose attempt for diagnosis; note a usbtest battery that produced per-case verdicts is NOT auto-retried, so its result already stands). wedged=true if the board/fixture is unresponsive after the run (capture dmesg | tail -50 into detail).',
{ label: `hil:${b}`, phase: 'HIL', agentType: 'hil-operator', schema: HIL },
)

2
.claude/workflows/pr-babysit.js
View File

@ -109,7 +109,7 @@ const history = []
const repliedIds = new Set() // issue comments can't be thread-resolved, so they re-harvest every cycle — never reply twice
for (let cycle = 1; cycle <= maxCycles; cycle++) {
const t = await agent(
`Triage PR #${args.pr}. If checks are still running, wait for them first (gh pr checks ${args.pr} --watch, Bash timeout >= 30 min). ` +
`Triage PR #${args.pr}. If checks are still running, wait for them first (gh pr checks ${args.pr} --watch as a BACKGROUND Bash task; the foreground timeout is capped at 10 min). ` +
'Then follow your triage procedure: classify CI failures, re-run infra ones, harvest and adversarially validate bot review findings, draft replies for invalid/stale ones.',
{ label: `triage#${cycle}`, phase: 'Triage', agentType: 'pr-monitor', schema: TRIAGE },
)

8
CLAUDE.md
View File

@ -118,6 +118,14 @@ Cutting a release — version bump, regenerated files, the per-release changelog
## References
- MCU reference manuals, datasheets, schematics: before answering register/bitfield/pinout/errata/timing questions from memory or the web — or changing a specific dcd/hcd driver — use the `read-doc` skill (`.claude/skills/read-doc/SKILL.md`) to cross-check against docs in `$HOME/Documents/calibre-library`; tell the user if the needed document is missing (skill no-ops if the library is absent).
- Linux kernel behaviour (usbfs, usbtest, sysfs attributes, device locks, D state): never
infer it from symptoms — read the source for the *running* version. It refutes as often
as it confirms: it has killed two plausible dcd theories and corrected a recovery skill's
own attribute list.
```bash
V=$(uname -r | grep -oE '^[0-9]+\.[0-9]+\.[0-9]+') # on the rig: ssh ci.lan uname -r
curl -fsSL "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/plain/drivers/usb/core/sysfs.c?h=v$V"
```
- Supported MCUs/boards: `hw/bsp/` and `docs/reference/boards.rst`.
- USB classes: `src/class/{cdc,hid,msc,audio,…}/` — each has `*_device.c` and `*_host.c`.
- Key files: `src/tusb.h`, `src/tusb_config.h`, `tools/get_deps.py`, `tools/build.py`, `test/unit-test/project.yml`.

236
docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md Normal file
View File

@ -0,0 +1,236 @@
# HIL fleet-wedge containment
Date: 2026-07-30
Status: implemented, then superseded in part — addendum last checked 2026-08-12
against the shipped code; where they disagree the CODE and the usb-kernel-recover
skill win, never this document.
- **Pool guard.** A single constant, not the flat 4200s below and not a derivation:
`POOL_TIMEOUT = pos_int_env('HIL_POOL_TIMEOUT', 3600)`. A per-controller model briefly
lived here and was removed -- it under-modelled the flash phase and could INVERT
(adding a usbtest board lowered the guard, because the derived value fell below the
baseline it was meant to raise). The guard's only job is to stop a wedged pool short
of the job ceiling so the report still gets written; predicting a healthy run's
duration is a different problem. `pos_int_env` warns only on a non-integer or a value
<= 0: there is NO upper clamp and no warning above any threshold, so a pin larger than
a job ceiling silently restores the inversion this work removed.
- **Job ceilings.** 90/90/120 min (build.yml), not 60/60/90 and not the 85/115 below.
They must clear the 3600s guard plus the pre-pool checkout/artifact merge and the
post-guard sweep and report upload. No job pins `HIL_POOL_TIMEOUT`.
- **Battery budgets.** `USBTEST_BATTERY_BUDGET` 260s, `USBTEST_RECOVERY_BUDGET` 250s.
The 200s-with-a-197s-floor derivation recorded here was never shipped; the floor
assertion was removed with it.
- **HUNG recovery.** Reflash of the DUT through its roster flasher
(`usbtest.py --recover-board/--recover-fw`), not the root-cycle-first recovery in
section 1d — replaced after the 2026-08-11 ppps measurement (uhubctl never cuts
VBUS; root-cycle is probe-only). Since 2026-08-12 the reflash is SKIPPED
when `hil_flash.convoy_safe(board['flasher'])` is false (usbtest.py:675): the flasher
would enumerate by opening usbfs nodes, block on the same convoy, and become a second
stray rather than clear the first. A holder that owns the device lock inside a driver
ioctl is terminal either way -- a reflash only produces a disconnect, and
`usb_disconnect()` needs that same lock -- and that state needs a reboot.
Step 0 done — the host was rebooted 2026-07-30 14:11 and the rig
came back clean. The device that triggered this incident was removed from the rig, so
only the containment work remains relevant.
Rig: `ci.lan` (Proxmox guest on `pve.lan`)
## Problem
On 2026-07-29/30 every board in the `ci.lan` usbtest fleet failed, `openocd` processes
landed in uninterruptible sleep, and no subsequent HIL run could start. Two GitHub
Actions runs were stranded: `30484641269` sat `in_progress` for over eight hours
(past GitHub's own 360-minute default), and `30485082274` sat `queued` behind it from
2026-07-29 19:35 UTC onward. Both report directories were written empty.
A reboot of the `ci` guest at 10:48 did not clear the condition: the same kernel state
re-formed at 10:52:23.
## Root cause
Five layers, each independently observable.
### 1. A permanently wedged hub worker holds a root-hub device lock
A device that repeatedly re-asserts connect while failing to enumerate keeps
`hub_event()` busy, and `hub_event()` holds `usb_lock_device(hdev)` on its hub for its
whole run (hub.c:5896/5989). The `usb_hub_wq` worker sits in `hub_port_reset`, so that
hub's `device_lock` is effectively never released:
```
kworker/14:6+usb_hub_wq (state D, 400+ s)
msleep+0x2b
hub_port_reset+0x1a4 [usbcore]
hub_event+0x727 [usbcore]
```
`usb usbN-portM: Cannot enable. Maybe the USB cable is bad?` is logged every four seconds
for as long as it lasts.
Verified against hub.c v6.12.96 rather than inferred: the kernel does **not** retry
without bound, and root and downstream ports are bounded identically —
`hub_port_reset()` tries `PORT_RESET_TRIES` then logs that message (hub.c:3149),
`hub_port_connect()` wraps it in `PORT_INIT_TRIES` = 4 and disables the port on give-up
(hub.c:5455/5619). A count in the thousands is therefore that many separate connect
events, not one runaway loop, and it indicts the device rather than the port.
### 2. A parked board storms the second controller
`ra6m5_ek` (`test/hil/tinyusb.json`, uid `8419032D32363657364EF4622D294B4E`, at
`13-3.3`) runs dfu firmware (`cafe:400b`) and re-enumerates every 1-2 seconds
continuously, wrapping the entire bus-13 devnum space (`...120 -> 127 -> 4 -> 6 -> 10`).
This is standing `hub_event` and Address-Device pressure on controller `03:00.0`,
concurrent with parallel usbtest batteries on the same silicon.
The board is already listed in `boards-skip`, which is precisely why it storms:
`boards-skip` stops testing a board but never parks it, so it keeps running whatever
firmware it last received. Park-flash only runs as teardown of a board that actually
executed tests.
### 3. The kernel `usbtest` control-queue case waits without a timeout
`test_ctrl_queue` blocks on an untimed `wait_for_completion()` while `usbdev_ioctl`
holds the DUT's `device_lock`:
```
wait_for_completion+0x8a <- no _timeout variant
test_ctrl_queue+0x4ab [usbtest]
usbtest_do_ioctl+0x501 [usbtest]
usbdev_ioctl+0x6b8 [usbcore]
```
`--timeout 60` in `test/hil/usbtest.py` is a subprocess timeout only. `SIGKILL` is not
delivered to a task in uninterruptible sleep. `usbtest.py` already recognises this and
reports `HUNG`, then calls `usb_recover.sh root-cycle`.
### 4. openocd inherits the convoy and the whole fleet dies
Once a device lock is stuck, `port_event()` takes a child device's lock to warm-reset
it and blocks while still holding its hub's lock. Any later
`open("/dev/bus/usb/BBB/DDD")` against such a device blocks uninterruptibly:
```
usbdev_open+0xdc [usbcore] -> __mutex_lock
chrdev_open -> do_sys_openat2 -> __x64_sys_openat
```
That is the state of the three `openocd` processes at 04:16:51 (pids 207921, 207987,
208034) — the flasher, unkillable. Because one controller carries two buses, a single
convoy takes out every board on both, which is why the failure presents as the entire
fleet.
The existing `HUNG` recovery cannot help here. A root-port VBUS cycle frees a
*device-lock* holder; it cannot free a lock held by a stuck *hub worker*, and on this
rig the cycle lands on the controller that is already wedged.
### 5. Nothing bounds the damage, so one bad run becomes a CI outage
- `hil-tinyusb` and `hil-tinyusb-esp` in `.github/workflows/build.yml` carry no
`timeout-minutes`. Only `hil-hfp-iar` does.
- `ci.lan` runs a single runner service, so there is one job slot.
- `test/hil/hil_test.py` bounds the pool with `POOL_TIMEOUT` (4200 s), and that guard
fires correctly — but the recovery path does not survive a D-state worker:
```python
with Pool(processes=os.cpu_count() or 1, initializer=init_worker, initargs=initargs) as pool:
async_ret = pool.map_async(test_board, config_boards)
try:
mret = async_ret.get(timeout=POOL_TIMEOUT)
except MpTimeoutError:
pool.terminate()
pool.join() # blocks forever: a D-state worker never reaps
raise RuntimeError(f'HIL worker pool timed out after {POOL_TIMEOUT}s')
```
`multiprocessing` joins workers unbounded, so both `pool.terminate()` and
`pool.join()` hang, as does the `with Pool(...)` exit on the success path. Normal
`hil-tinyusb (tinyusb.json)` runs take 10-20 minutes; one recent run took 71.3
minutes, which is the 70-minute guard firing and succeeding. The eight-hour run is the
pathological case.
## Design
### Step 0 — recovery (manual prerequisite)
Power-cycle the PVE **host**, not the `ci` guest. A guest reboot is not sufficient;
hubs latch up across the PCIe reset, which the 10:48 reboot demonstrated. Nothing
below can be verified until the rig is clean.
### Section 1 — CI containment
**1a. Two layered timers.** An inner guard inside `hil_test.py` (`POOL_TIMEOUT`, 70 min)
that fails gracefully -- it writes a report naming the timeout and the dispatched boards,
shuts the pool down and exits -- and an outer `timeout-minutes` per rig job (85 for the
hil-tinyusb jobs; 115 for hil-hfp-iar, which also builds four boards with IAR in the same
job) as the backstop for when even exiting cannot free the runner. The ceiling must stay
ABOVE the inner guard, or GitHub kills the job before the report is written.
> **Corrected after measurement.** An earlier revision cut the guard to 30 min on the
> reading that real runs take 9-17 min and everything longer was the old guard firing.
> That was wrong. `hil_lock.py` records 22.2/14.3/12.5/10.8 min at usbtest width 1/2/3/4,
> and raising the per-battery budget to 380s made hung boards cost more again. The 30 min
> guard then fired on 5 of the last 8 HIL job executions across both rigs, and because
> `map_async` is all-or-nothing each of those runs published a banner instead of any
> per-board result. Restored to 4200s, the value whose original rationale -- usbtest
> batteries are serialized fleet-wide, lengthening the tail -- was correct.
**1b. Bound the pool shutdown.** Add a helper to `test/hil/hil_test.py`:
```python
def _shutdown_pool(pool, grace=30):
"""terminate() a Pool without ever blocking forever: multiprocessing joins its
workers unbounded, and a worker in uninterruptible sleep (wedged usbfs) never
reaps -- which would hold the runner's only job slot indefinitely."""
t = threading.Thread(target=pool.terminate, daemon=True)
t.start()
t.join(grace)
return not t.is_alive()
```
On the `MpTimeoutError` path: write the report first, recording the boards that never
reported so the run stops producing an empty report directory; then `_shutdown_pool`;
then `os._exit(1)` if it did not return. The hard exit is the point — it is the only
way past a kernel-side unkillable child. Use the same helper for the `with Pool(...)`
exit path.
**1c. Pre-flight rig health check.** `check_rig_health()` runs before the build and
**never aborts**. It probes `/proc` unprivileged (dmesg is restricted on the rig) for a
wedged `usb_hub_wq` worker, and reports a `/proc` too restricted to trust as its own
distinct cause rather than as a diagnosed fault.
It is deliberately non-fatal: the rig is unattended and every remedy for a real wedge is
manual, so aborting would not fix anything -- it would discard the per-board results the
run can still collect and leave CI red until a human noticed. It emits a GitHub
`::error::` annotation and continues. The automatic containment is 1a and 1b, which bound
a stuck run and explain it without anyone touching the rig.
**1d. Order the recovery correctly.** In `test/hil/usbtest.py`, attempt
`usb_recover.sh root-cycle` FIRST on a `HUNG` case, and only check for a wedged hub worker
*afterwards*.
> **Corrected during implementation.** This section originally said to check for a wedged
> worker *before* the cycle and skip it on a hit. That is backwards. Our own stuck
> `testusb` holds the DUT's device lock, so any port event drives a hub worker into
> `usb_lock_device()` on it -- uninterruptible, so it reads `D` in ~100% of samples and the
> confirmation window makes the wrong verdict *more* confident, not less. Cutting VBUS is
> precisely what completes the in-flight URB, returns the ioctl and frees that worker, so
> gating on that signature would suppress the recovery in the exact ordering it exists for.
> A worker still wedged after the cycle is the genuinely unrecoverable case, and that is
> what the code now reports.
## Verification
- Unit-test `shutdown_pool` and the `hil_health` detectors against a synthetic `/proc`.
A real wedge cannot be manufactured on demand, so they are tested against fabricated
inputs rather than live hardware.
- Confirm the detectors flag a genuinely wedged rig, and return clean on a healthy one.
- One clean full-fleet `hil_test.py` run to prove `check_rig_health` does not
false-abort.
## Out of scope
- **`ra6m5_ek` park and its dfu reset loop.** Dropped by decision. Consequence: the
layer-2 devnum storm remains as standing pressure on controller `03:00.0`. Unplugging
the board or flashing `board_test` by hand resolves it without any code change.
- **An unattended PVE watchdog** that detects the wedge and power-cycles the host.
Declined: more moving parts, and it can cut a running CI job.