mirror of
https://github.com/hathach/tinyusb.git
synced 2026-08-18 02:53:35 +00:00
test/hil: replace PCI reset with root-port VBUS cycle for D-state recovery (#3789)
test/hil: replace PCI reset with root-port VBUS cycle for D-state recovery pci-reset was documented as an FLR, but no controller on either rig has FLR, so it issued a PCIe secondary bus reset on a live, driver-bound xHCI -- halting the card until the PVE host was power-cycled, and returning success so the caller could not tell. It destroyed the ci controller twice. Replace it with root-cycle, which cuts VBUS at the xHCI root port and touches only the root hub, so it never takes the per-device lock the wedged ioctl holds. uhubctl needs -S, or its sysfs backend disconnects the child before cutting power and blocks on that same lock. Success is proven by the device's sysfs directory inode changing: node existence proves nothing, and devnum is reused once the per-bus map wraps. usbtest.py's hang path invokes it, then confirms via /proc that nothing still holds the device node. Skill scripts now run from the repo; the drifted /usr/local/sbin copies are deleted.
This commit is contained in:
Ha Thach
GitHub
@ -12,11 +12,11 @@ sits in the link — the rig PC when it is the host, or a Linux gadget peer
|
||||
(dwc2/UDC + gadget modules) when TinyUSB is the host. It cannot see inside
|
||||
the TinyUSB MCU — that is the `target-debug` skill.
|
||||
|
||||
Run this skill's `scripts/usb_dyndbg.sh` with `sudo` (abbreviated to
|
||||
`usb_dyndbg.sh` in the examples below). It flips the dynamic-debug print flag
|
||||
for an allowlisted set of USB modules only:
|
||||
Run this skill's `scripts/usb_dyndbg.sh` with `sudo`. It flips the dynamic-debug
|
||||
print flag for an allowlisted set of USB modules only:
|
||||
|
||||
```bash
|
||||
# all examples below abbreviate: sudo .claude/skills/usb-kernel-debug/scripts/usb_dyndbg.sh
|
||||
sudo usb_dyndbg.sh on usbcore xhci_hcd # enable +p; pick modules from `lsusb -t` Driver=
|
||||
sudo usb_dyndbg.sh status [module] # list enabled print sites
|
||||
sudo usb_dyndbg.sh off usbcore xhci_hcd # ALWAYS turn off when done — very noisy
|
||||
|
||||
@ -5,18 +5,20 @@ description: Use when a USB device or fixture attached to the ci HIL rig's Linux
|
||||
|
||||
# USB Recovery on the HIL Rig (Linux kernel side)
|
||||
|
||||
Run this skill's `scripts/usb_recover.sh` with `sudo` (abbreviated to
|
||||
`usb_recover.sh` in the examples below). It wraps the sysfs reset actions, a
|
||||
uhubctl power-cycle escalator, and a resolver:
|
||||
Run this skill's `scripts/usb_recover.sh` with `sudo`. It wraps the sysfs reset
|
||||
actions, a uhubctl power-cycle escalator, and a resolver:
|
||||
|
||||
```bash
|
||||
# all examples below abbreviate: sudo .claude/skills/usb-kernel-recover/scripts/usb_recover.sh
|
||||
sudo usb_recover.sh resolve /dev/ttyACM3 # /dev node -> busport (e.g. 3-4.7); also ttyUSB*, sg*
|
||||
sudo usb_recover.sh authorized <busport> # deauthorize+reauthorize: re-enumerate, no VBUS cut
|
||||
sudo usb_recover.sh rebind <busport> # usb driver unbind+bind: re-probe
|
||||
sudo usb_recover.sh hub-cycle <busport> # uhubctl VBUS cycle of the feeding port, walking parent hub
|
||||
# -> root port until the device re-enumerates
|
||||
sudo usb_recover.sh root-cycle <busport> [serial] # uhubctl VBUS cut straight at the ROOT port (real ppps), no
|
||||
# leaf walk, no device-lock touch: the D-state cure.
|
||||
# [serial] is checked and a mismatch refused.
|
||||
sudo usb_recover.sh pci-rebind <pciaddr> # whole HCD controller unbind+bind, e.g. 0000:02:00.0
|
||||
sudo usb_recover.sh pci-reset <pciaddr> # PCI function-level reset: kills URBs at HW level, no device lock
|
||||
sudo usb_recover.sh pci-bind <pciaddr> [drv] # re-bind a DRIVERLESS controller (auto-tries xHCI drivers)
|
||||
```
|
||||
|
||||
@ -34,21 +36,45 @@ ps -eo pid,stat,wchan:30,cmd | awk '$2 ~ /D/'
|
||||
```
|
||||
|
||||
**If yes** (uninterruptible sleep, typically a usbfs ioctl — e.g. testusb inside
|
||||
`usb_sg_wait`): run `pci-reset` and NOTHING ELSE first:
|
||||
`usb_sg_wait`): cut VBUS at the root port, and nothing else.
|
||||
|
||||
```bash
|
||||
sudo usb_recover.sh pci-reset <pciaddr>
|
||||
sudo usb_recover.sh root-cycle <busport> # e.g. 11-3.7 -> cycles bus 11 root port 3
|
||||
```
|
||||
|
||||
FLR kills the URBs at the hardware level without taking the per-device lock;
|
||||
the ioctl then returns and the convoy unwinds on its own.
|
||||
This drops power to the wedged device, so its in-flight URB fails and the ioctl
|
||||
returns. It targets the *root hub* — a different USB device from the wedged one —
|
||||
and never *writes* the wedged device's sysfs. It reads a few attributes from it —
|
||||
`idVendor`/`idProduct`/`serial`/`product` to report and check the target, and the
|
||||
directory inode plus `devnum` afterwards — none of which take the device lock, so
|
||||
it does not join the convoy the way `authorized`/`rebind`/`pci-rebind` do.
|
||||
Recovery is proven by that inode changing — a real disconnect destroys the
|
||||
kobject and reconnecting creates a new one, whereas a disconnect blocked on the
|
||||
device lock leaves it untouched. It exits non-zero if the device does not come
|
||||
back; a **zero exit only means it re-enumerated**, so still confirm the D-state
|
||||
process actually let go. Pass the expected serial as a third argument and it
|
||||
refuses a busport that now names a different device.
|
||||
|
||||
**Not every controller supports FLR.** The Renesas uPD720201 (`0000:01:00.0`)
|
||||
has no reset method — `pci-reset` fails with `Inappropriate ioctl for device`
|
||||
(ENOTTY). On those, there is no clean software D-state cure — a VM reboot is NOT
|
||||
It bounces **every fixture under that root port** — on ci that is up to 25
|
||||
devices. Hold the affected boards' locks first if you can, but note
|
||||
`board_lock.py` uses `LOCK_EX | LOCK_NB` and so fails immediately when CI already
|
||||
holds them; there is no wait-for-lock. When CI is mid-run you are choosing
|
||||
between bouncing its fixtures and leaving the bus wedged for everything. The
|
||||
automated path in `usbtest.py` takes no locks at all and accepts that collateral
|
||||
deliberately: by the time a D-state wedge exists the convoy will take the bus
|
||||
down anyway.
|
||||
|
||||
(The VBUS mechanism is verified on the ci rig — the leaf hubs report
|
||||
`bmAttributes=e0`, "self-powered", but are physically bus-powered with no adapter,
|
||||
so a root-port cut really does kill downstream power. Do not re-derive this from
|
||||
the descriptor; it lies. Not yet confirmed against a live D-state wedge. If
|
||||
`uhubctl` itself hangs, the convoy has already spread — escalate.)
|
||||
|
||||
If `root-cycle` does not free the D-state process, there is no software cure
|
||||
left: ask the operator for a full PVE **host** power cycle. A VM reboot is NOT
|
||||
reliable (downstream hubs can latch up across the PCIe reset and need a physical
|
||||
replug); ask the operator for a full PVE host power cycle instead. Do NOT
|
||||
fall through to `pci-rebind` (see next).
|
||||
replug), and a graceful reboot stalls on the D-state process anyway. Do NOT fall
|
||||
through to `pci-rebind` (see next).
|
||||
|
||||
**`pci-rebind` can strand the controller driverless.** Its unbind succeeds but,
|
||||
with a D-state process still holding a URB, the *re-bind* hangs — leaving the
|
||||
@ -62,10 +88,9 @@ power cycle (operator action) recovers. The Renesas binds via `xhci-pci-renesas`
|
||||
**Ordering is critical.** `authorized`/`rebind`/`pci-rebind` all take the
|
||||
per-device lock the stuck ioctl holds — they block and join the convoy, and
|
||||
soon every libusb tool (uhubctl, JLinkExe) hangs too. Worse, a blocked
|
||||
`pci-rebind` grabs the PCI device lock on its way in, which `pci-reset` also
|
||||
needs: once a rebind has been attempted and is stuck, even FLR deadlocks and
|
||||
**only a full PVE host power cycle recovers**. pci-reset first (if supported), and never
|
||||
`pci-rebind` a D-state wedge.
|
||||
`pci-rebind` grabs the PCI device lock on its way in and can wedge the whole
|
||||
function, after which **only a full PVE host power cycle recovers**. `root-cycle`
|
||||
first, and never `pci-rebind` a D-state wedge.
|
||||
|
||||
**If no** (device merely dead or silent), escalate gently:
|
||||
|
||||
@ -93,15 +118,19 @@ hubs themselves claim "ganged" switching but do not actually cut power.
|
||||
## Common mistakes
|
||||
|
||||
- `resolve` takes a **/dev node**, not a busport or serial ("no such device node").
|
||||
- `authorized`/`rebind` take a **busport** (`3-4.7`); `pci-rebind`/`pci-reset`
|
||||
take a **PCI addr**.
|
||||
- `authorized`/`rebind`/`hub-cycle`/`root-cycle` take a **busport** (`3-4.7`);
|
||||
`pci-rebind`/`pci-bind` take a **PCI addr**.
|
||||
- Command produces no output and doesn't return → it is blocked on the device
|
||||
lock: a D-state holder exists; see above.
|
||||
- Trying `pci-rebind` on a D-state hang — its re-bind hangs and strands the
|
||||
controller **driverless**; recover with `pci-bind <addr>`, or a PVE host power
|
||||
cycle if the D-state URB is unkillable. Use `pci-reset` (if supported) for D-state, never
|
||||
cycle if the D-state URB is unkillable. Use `root-cycle` for D-state, never
|
||||
`pci-rebind`.
|
||||
- Running `pci-reset` on a controller without FLR support (Renesas) → ENOTTY;
|
||||
no software recovery — needs a PVE host power cycle.
|
||||
- Writing `/sys/bus/pci/devices/<addr>/reset` because the attribute is there. No
|
||||
rig controller has FLR, so it becomes a PCIe bus reset that resets the xHCI
|
||||
behind its live driver — the write succeeds, the card is halted for good, and
|
||||
only a PVE host power cycle brings it back. Use `root-cycle`.
|
||||
- `root-cycle` bounces **every** fixture under that root port, not just the target
|
||||
— hold the sibling boards' locks first.
|
||||
- A J-Link reset (`r; go`) does not disconnect a wedged DUT from the host: the
|
||||
DWC2 soft-connect pullup stays up through a core halt, so stuck URBs stay stuck.
|
||||
|
||||
@ -6,9 +6,6 @@
|
||||
# sudo usb_recover.sh authorized <busport> # e.g. 3-2 -> deauthorize+reauthorize (re-enumerate, NO VBUS cut)
|
||||
# sudo usb_recover.sh rebind <busport> # e.g. 3-2 -> usb driver unbind+bind (re-probe)
|
||||
# sudo usb_recover.sh pci-rebind <pciaddr> # e.g. 0000:01:00.0 -> HCD unbind+bind (WHOLE controller)
|
||||
# sudo usb_recover.sh pci-reset <pciaddr> # e.g. 0000:01:00.0 -> PCI function-level reset: kills URBs at
|
||||
# # HW level WITHOUT the device lock; the only cure when a process
|
||||
# # is stuck in D state (usbfs ioctl) and unbind paths would convoy
|
||||
# sudo usb_recover.sh pci-bind <pciaddr> [driver] # bind a DRIVERLESS controller (e.g. after a pci-rebind
|
||||
# # whose re-bind hung and left it unbound). Auto-tries the xHCI
|
||||
# # drivers (xhci-pci-renesas, xhci_hcd) unless one is named.
|
||||
@ -17,6 +14,11 @@
|
||||
# # re-enumerates. Ganged/fake-switching hubs may bounce ALL
|
||||
# # siblings; self-powered hubs only reset their uplink, which
|
||||
# # is why the walk ends at the root port (real xHCI ppps).
|
||||
# sudo usb_recover.sh root-cycle <busport> [serial] # e.g. 13-1.6 -> uhubctl VBUS cut at the ROOT port feeding
|
||||
# # it; [serial] is verified against the device and refused on mismatch,
|
||||
# # skipping the leaf hubs (which fake ganged switching and do not
|
||||
# # actually cut power). Bounces every sibling under that root port.
|
||||
# # The D-state escape: no device lock, so it cannot convoy.
|
||||
# sudo usb_recover.sh resolve <devnode> # e.g. /dev/ttyACM3 -> print its <busport> (no privilege needed)
|
||||
set -euo pipefail
|
||||
|
||||
@ -25,6 +27,29 @@ PCI_RE='^[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}\.[0-9]$'
|
||||
DRIVER_RE='^[A-Za-z0-9_-]+$'
|
||||
|
||||
die() { echo "usb_recover: $*" >&2; exit 1; }
|
||||
|
||||
# Generation marker for "did this device actually re-enumerate". A real disconnect destroys the
|
||||
# usb_device and its sysfs kobject; reconnecting creates a new one, and kernfs hands out inode
|
||||
# numbers monotonically, so the directory inode changes. Verified on the rig: ports re-enumerated
|
||||
# minutes ago carry inodes in the millions while ports untouched since boot are still in the tens
|
||||
# of thousands, ranking identically to their mtimes.
|
||||
#
|
||||
# This beats comparing devnum, which Linux reuses once the per-bus map wraps (observed live: a
|
||||
# single cycle moved one device 123 -> 113). It also beats watching for the node to vanish, since
|
||||
# `uhubctl -a cycle` holds the whole power-off window inside itself and a poll afterwards can
|
||||
# never witness the gap. The inode survives the gap, so no observation window is needed.
|
||||
#
|
||||
# Crucially, if the disconnect is blocked on the wedged device's lock the kobject is never
|
||||
# recreated -- same inode -- which is exactly the case that must be reported as a failure. Verified
|
||||
# against kernfs: __kernfs_new_node() allocates via idr_alloc_cyclic() but kernfs_id_ino() exposes
|
||||
# the full 64-bit (id_highbits<<32 | lowbits) as st_ino on 64-bit ino_t, so a repeat needs ~2^64
|
||||
# node creations. authorized-toggle, set_configuration and suspend/resume all leave the parent
|
||||
# device kobject alone, so none of them can move the marker and fake a success.
|
||||
#
|
||||
# The trailing slash is load-bearing: /sys/bus/usb/devices/<busport> is a SYMLINK with its own
|
||||
# separate inode, so without it stat reports the link rather than the device it points at, and the
|
||||
# value would never change. Do not "tidy" it away.
|
||||
sysfs_gen() { stat -c %i "/sys/bus/usb/devices/$1/" 2>/dev/null || echo none; }
|
||||
usage() { grep -E '^# sudo usb_recover' "$0" >&2; exit 2; }
|
||||
|
||||
# Refuse to touch a PCI function that is not a USB controller (class 0x0c03xx), so a stray or
|
||||
@ -107,6 +132,11 @@ case "$action" in
|
||||
[[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target"
|
||||
UHUBCTL=$(command -v uhubctl || echo /sbin/uhubctl)
|
||||
[ -x "$UHUBCTL" ] || die "uhubctl not installed"
|
||||
# sysfs generation, not node existence: a disconnect blocked on the device lock leaves the
|
||||
# old node (and its idVendor) in place, so an existence check reports success without anything
|
||||
# having happened -- and the walk to the root port, which is the part that actually cuts power
|
||||
# on these fake-ganged leaf hubs, would never run.
|
||||
gen=$(sysfs_gen "$target")
|
||||
dev="$target"
|
||||
while :; do
|
||||
if [[ "$dev" =~ ^([0-9]+)-([0-9]+)$ ]]; then # parent is the root hub
|
||||
@ -118,8 +148,9 @@ case "$action" in
|
||||
"$UHUBCTL" -l "$loc" -p "$port" -a cycle -d 5 -f || echo " (uhubctl failed at $loc; walking up)"
|
||||
for _ in $(seq 1 10); do
|
||||
sleep 1
|
||||
if [ -e "/sys/bus/usb/devices/$target/idVendor" ]; then
|
||||
echo "recovered: $target re-enumerated"; exit 0
|
||||
now=$(sysfs_gen "$target")
|
||||
if [ "$now" != none ] && [ "$now" != "$gen" ]; then
|
||||
echo "recovered: $target re-enumerated (gen $gen -> $now)"; exit 0
|
||||
fi
|
||||
done
|
||||
[ -n "$up" ] || break
|
||||
@ -127,12 +158,48 @@ case "$action" in
|
||||
done
|
||||
die "hub-cycle: $target still not enumerated after cycling up to the root port"
|
||||
;;
|
||||
pci-reset)
|
||||
[[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target"
|
||||
require_usb_controller "$target"
|
||||
[ -e "/sys/bus/pci/devices/$target/reset" ] || die "no reset support on $target"
|
||||
echo 1 > "/sys/bus/pci/devices/$target/reset"
|
||||
echo "flr-reset pci $target"
|
||||
root-cycle)
|
||||
# VBUS cut at the ROOT port, where xHCI ppps is real. Unlike hub-cycle this does not walk up
|
||||
# from the leaf (the 1a40:0201 hubs claim ganged switching but never cut power) and never
|
||||
# writes the wedged device's sysfs or takes its lock, so it cannot join a D-state convoy.
|
||||
# uhubctl exits 0 even when it does nothing ("No compatible devices detected" still returns
|
||||
# 0), so its status proves nothing -- the sysfs_gen check below is the only real verdict.
|
||||
[[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target"
|
||||
UHUBCTL=$(command -v uhubctl || echo /sbin/uhubctl)
|
||||
[ -x "$UHUBCTL" ] || die "uhubctl not installed"
|
||||
# Existence alone only proves *something* occupies that path -- bus numbers renumber every
|
||||
# boot, so a stale busport can name a different device entirely and we would cut power to its
|
||||
# whole subtree (up to 25 fixtures on this rig). Callers that know what they expect pass the
|
||||
# serial as a third argument and we refuse on mismatch; otherwise print the identity so a
|
||||
# wrong target is at least visible.
|
||||
[ -e "/sys/bus/usb/devices/$target" ] || die "no such usb device: $target"
|
||||
idf="/sys/bus/usb/devices/$target"
|
||||
serial=$(cat "$idf/serial" 2>/dev/null || echo -)
|
||||
expect=${3:-}
|
||||
[ -z "$expect" ] || [ "$expect" = "$serial" ] || \
|
||||
die "root-cycle: $target has serial '$serial', expected '$expect' — stale busport, refusing"
|
||||
echo "root-cycle: target $target is $(cat "$idf/idVendor" 2>/dev/null || echo -):$(cat "$idf/idProduct" 2>/dev/null || echo -)" \
|
||||
"serial=$serial product=$(cat "$idf/product" 2>/dev/null || echo -)"
|
||||
bus=${target%%-*}; rest=${target#*-}; rootport=${rest%%.*}
|
||||
gen=$(sysfs_gen "$target")
|
||||
echo "root-cycle: cutting VBUS on bus $bus root port $rootport (feeds $target, bounces its siblings)"
|
||||
# -S is load-bearing. By default uhubctl writes /sys/.../usb<bus>-port<n>/disable (verified:
|
||||
# two O_WRONLY opens per cycle), and the kernel's disable_store() takes the ROOT HUB's lock and
|
||||
# synchronously usb_disconnect()s the child BEFORE cutting power -- against a wedged device that
|
||||
# blocks on the lock we are trying to free, so power would never drop and uhubctl would D-state
|
||||
# holding the root hub's lock, poisoning the whole bus. -S forces the libusb path, which sends
|
||||
# the power-off control transfer straight to the root hub with no child-disconnect in front.
|
||||
"$UHUBCTL" -S -l "$bus" -p "$rootport" -a cycle -d 5 \
|
||||
|| die "uhubctl failed to cycle bus $bus port $rootport"
|
||||
for _ in $(seq 1 10); do
|
||||
sleep 1
|
||||
now=$(sysfs_gen "$target")
|
||||
if [ "$now" != none ] && [ "$now" != "$gen" ]; then
|
||||
echo "root-cycled $bus port $rootport: $target re-enumerated"\
|
||||
"(devnum $(cat "/sys/bus/usb/devices/$target/devnum" 2>/dev/null || echo ?), gen $gen -> $now)"; exit 0
|
||||
fi
|
||||
done
|
||||
die "root-cycle: $target did not re-enumerate after cycling bus $bus port $rootport (sysfs generation still $gen: no disconnect happened)"
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
|
||||
@ -254,11 +254,46 @@ def dmesg_tail():
|
||||
return '\n'.join(lines[-8:])
|
||||
|
||||
|
||||
def pci_addr_of_bus(busnum):
|
||||
"""Return the PCI B:D.F backing a USB bus, or None for a non-PCI (SoC/platform) controller."""
|
||||
m = re.search(r'([0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-9])/usb\d+$',
|
||||
os.path.realpath(f'/sys/bus/usb/devices/usb{int(busnum)}'))
|
||||
return m.group(1) if m else None
|
||||
def wedged_pids(devnode):
|
||||
"""Return (pids, complete): PIDs in uninterruptible sleep whose cmdline names devnode, i.e.
|
||||
still holding its usbfs device lock, and whether every /proc entry could actually be read.
|
||||
|
||||
Matched by device node rather than by our child's pid because run_case() may wrap testusb in
|
||||
sudo, in which case the Popen pid is the wrapper and the blocked process is its child --
|
||||
killing the wrapper would make a pid-based check look clean while the real holder is stuck.
|
||||
|
||||
complete is False when a PermissionError hid an entry (a hidepid/ProtectProc mount, or the
|
||||
root-owned child of that same sudo). An entry we could not read might be the holder, so the
|
||||
caller must treat that as unrecovered rather than as an all-clear."""
|
||||
stuck, complete = [], True
|
||||
# hidepid=2 and systemd's ProtectProc=invisible omit other users' processes from iterdir()
|
||||
# entirely -- no entry at all, so no PermissionError to catch -- and testusb runs under sudo
|
||||
# whenever the device node is not writable. The scan would then look clean while hiding the
|
||||
# very holder it exists to find. pid 1 is always root-owned, so being unable to read it means
|
||||
# enumeration is restricted and no result from this scan can be trusted as complete.
|
||||
if os.geteuid() != 0 and not os.access('/proc/1/cmdline', os.R_OK):
|
||||
complete = False
|
||||
for entry in Path('/proc').iterdir():
|
||||
if not entry.name.isdigit():
|
||||
continue
|
||||
try:
|
||||
cmdline = (entry / 'cmdline').read_bytes()
|
||||
except PermissionError:
|
||||
complete = False # cannot rule this pid out
|
||||
continue
|
||||
except OSError:
|
||||
continue # raced with process exit: genuinely gone, not hidden
|
||||
if devnode.encode() not in cmdline:
|
||||
continue
|
||||
try:
|
||||
stat = (entry / 'stat').read_text()
|
||||
if stat[stat.rindex(')') + 2] == 'D': # comm may contain ')', so scan from the right
|
||||
stuck.append(int(entry.name))
|
||||
except PermissionError:
|
||||
complete = False
|
||||
except (OSError, ValueError, IndexError):
|
||||
continue
|
||||
return stuck, complete
|
||||
|
||||
|
||||
def run_case(num, dev, testusb, quick, timeout):
|
||||
@ -387,20 +422,68 @@ def main():
|
||||
extra += f" {r['mbps']} MB/s" if 'mbps' in r else ''
|
||||
print(f"test {num:2d} {r['name']:22s} {r['status']:6s}{extra}")
|
||||
if r['status'] == 'HUNG':
|
||||
pci = pci_addr_of_bus(dev['node'].split('/')[-2])
|
||||
if pci:
|
||||
print(f'aborting battery: kernel-side hang, device wedged mid-transfer.\n'
|
||||
f'auto-recovering: sudo {USB_RECOVER} pci-reset {pci} '
|
||||
f'(see .claude/skills/usb-kernel-recover)', file=sys.stderr)
|
||||
# FLR frees the D-state ioctl without the device lock; must run BEFORE
|
||||
# any unbind/remove_id, which would deadlock the bus otherwise
|
||||
if sudo([str(USB_RECOVER), 'pci-reset', pci]).returncode != 0:
|
||||
unrecovered_hang = True
|
||||
time.sleep(5) # let the bus re-enumerate before cleanup touches sysfs
|
||||
else:
|
||||
unrecovered_hang = True
|
||||
print('aborting battery: kernel-side hang, and the controller has no PCI address '
|
||||
'for FLR recovery — manual intervention (reboot) required', file=sys.stderr)
|
||||
print(f'aborting battery: kernel-side hang, device wedged mid-transfer.\n'
|
||||
f'auto-recovering: {USB_RECOVER.name} root-cycle {dev["sysname"]} '
|
||||
f'(see .claude/skills/usb-kernel-recover)', file=sys.stderr)
|
||||
# Cutting VBUS at the root port fails the in-flight URB so the usbfs ioctl returns.
|
||||
# Must run BEFORE any unbind/remove_id, which would take the device lock the stuck
|
||||
# ioctl holds and deadlock the bus.
|
||||
#
|
||||
# Assume unrecovered until proven otherwise, so that any early exit from this block
|
||||
# -- an OSError spawning the helper, a KeyboardInterrupt, a sudo prompt killing the
|
||||
# run -- still reaches the finally cleanup with the flag set, instead of running
|
||||
# the remove_id/unbind the comments there forbid while a device lock is held.
|
||||
unrecovered_hang = True
|
||||
# Pass the serial so the helper refuses a stale busport rather than cutting power
|
||||
# to whatever else now occupies that path. Popen rather than sudo()/subprocess.run:
|
||||
# run() would kill() then wait() unbounded on timeout, which never returns if
|
||||
# uhubctl is itself in D state -- the case the timeout exists for. Merge stderr
|
||||
# into stdout so the helper's target-identity and action lines are not lost.
|
||||
# Only pass the serial when we actually have one: an empty third argument reads as
|
||||
# "no expectation" and would silently disable the helper's stale-busport guard.
|
||||
cmd = [str(USB_RECOVER), 'root-cycle', dev['sysname']]
|
||||
if dev['serial']:
|
||||
cmd.append(dev['serial'])
|
||||
if os.geteuid() != 0:
|
||||
cmd = ['sudo', '-n'] + cmd
|
||||
try:
|
||||
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
|
||||
text=True)
|
||||
except OSError as e:
|
||||
# helper missing or not executable, or sudo unavailable. unrecovered_hang is
|
||||
# already True so the finally block still skips the unsafe cleanup -- this only
|
||||
# replaces a traceback with a message that says what to fix.
|
||||
print(f'cannot run {USB_RECOVER}: {e}', file=sys.stderr)
|
||||
break
|
||||
rc = None
|
||||
try:
|
||||
out, _ = p.communicate(timeout=60) # normal run is ~8s
|
||||
rc = p.returncode
|
||||
except subprocess.TimeoutExpired:
|
||||
p.kill()
|
||||
try:
|
||||
out, _ = p.communicate(timeout=5)
|
||||
rc = p.returncode
|
||||
except subprocess.TimeoutExpired:
|
||||
out = ('root-cycle abandoned after 60s: uhubctl did not die to SIGKILL, so '
|
||||
'it is wedged too and the convoy has spread beyond this device')
|
||||
if out:
|
||||
print(out.strip(), file=sys.stderr)
|
||||
if rc is not None:
|
||||
time.sleep(5) # let the bus settle and the freed ioctl unwind
|
||||
# Authoritative either way. A non-zero exit only means the device did not come
|
||||
# back within the poll (a slow bootloader will do that) -- if nothing still
|
||||
# holds the lock, the bus is usable and cleanup is safe. Conversely a zero exit
|
||||
# only proves re-enumeration, not that the D-state holder let go.
|
||||
stuck, complete = wedged_pids(dev['node'])
|
||||
if stuck:
|
||||
print(f'{dev["sysname"]}: pid(s) {stuck} still in D state on '
|
||||
f'{dev["node"]} — the device lock was never released', file=sys.stderr)
|
||||
elif not complete:
|
||||
print('cannot confirm recovery: /proc is only partly readable, so a '
|
||||
'hidden D-state holder cannot be ruled out', file=sys.stderr)
|
||||
else:
|
||||
unrecovered_hang = False
|
||||
break
|
||||
# re-resolve: after a mid-battery re-enumeration the devnum (and thus the node
|
||||
# path) changes; keep testing the live node instead of the stale one. Match on the
|
||||
@ -419,7 +502,8 @@ def main():
|
||||
if unrecovered_hang:
|
||||
# testusb is still stuck in a usbfs ioctl holding the device lock; remove_id/unbind
|
||||
# would join the convoy and deadlock the bus (see usb-kernel-recover skill) — leave it be
|
||||
print('skipping cleanup after unrecovered hang: reboot required to release the bus',
|
||||
print('skipping cleanup after unrecovered hang: ask the operator for a full PVE host '
|
||||
'power cycle (a VM reboot is not reliable — hubs latch up across the PCIe reset)',
|
||||
file=sys.stderr)
|
||||
elif not args.keep_binding:
|
||||
sysfs_write(DRIVER / 'remove_id', f'{VID} {PID}', check=False)
|
||||
|
||||
Reference in New Issue
Block a user