diff --git a/.claude/agents/hil-operator.md b/.claude/agents/hil-operator.md index c48ceb8bd..d19eca047 100644 --- a/.claude/agents/hil-operator.md +++ b/.claude/agents/hil-operator.md @@ -8,8 +8,8 @@ model: sonnet You operate physical USB test hardware. These repo skills are your source of truth — read the relevant one BEFORE acting: - `.claude/skills/hil/SKILL.md` — run `hostname` first (host `ci` = local mode with `test/hil/tinyusb.json`; host `htpc` = local `local.json` or remote via `test/hil/hil_ci.sh`); the board lock protocol; exact `hil_test.py` invocations. -- `.claude/skills/usb-recover/SKILL.md` — only when a device/fixture is wedged or processes hang in D state. -- `.claude/skills/usb-debug/SKILL.md` — only when you need to explain WHY the host rejected a device (dmesg analysis). +- `.claude/skills/usb-kernel-recover/SKILL.md` — only when a device/fixture on the rig's Linux host is wedged or processes hang in D state. +- `.claude/skills/usb-kernel-debug/SKILL.md` — only when you need to explain WHY the Linux kernel rejected a device (dmesg analysis). ## Board lock protocol (CI runs concurrently — NEVER stop the actions-runner) @@ -30,7 +30,7 @@ The GitHub Actions runner keeps running during your work. Per-board flock locks - HIL runs take 2–5 min per board: use Bash timeouts >= 20 min (1200000 ms) and NEVER cancel early. - One hardware action at a time. You are never run concurrently with another hil-operator. -- On test failure: retry once with `-v -r 1` appended (one verbose attempt for diagnosis — the first run already did the flake-retries). If a board/fixture stops enumerating or tools hang in D state, consult usb-recover and capture `dmesg | tail -50` into `detail`; set `wedged` true. +- On test failure: retry once with `-v -r 1` appended (one verbose attempt for diagnosis — the first run already did the flake-retries). If a board/fixture stops enumerating or tools hang in D state, consult usb-kernel-recover and capture `dmesg | tail -50` into `detail`; set `wedged` true. ## Output contract diff --git a/.claude/agents/target-debugger.md b/.claude/agents/target-debugger.md index c1df47cb2..e25ffa7f1 100644 --- a/.claude/agents/target-debugger.md +++ b/.claude/agents/target-debugger.md @@ -1,25 +1,29 @@ --- name: target-debugger -description: Root-cause one USB misbehavior on real HIL hardware by instrumenting the TinyUSB device side — TU_LOG/RTT, RAM ring-buffer trace, GDB autopsy, J-Link PC-sampling — correlated with host-side and wire-level capture. Long serial debug loop under one held board lock; strictly one instance. Produces a diagnosis with on-target evidence (plus a candidate fix when one emerges), never a merged patch. +description: Root-cause one USB misbehavior on real HIL hardware by instrumenting the TinyUSB target — device or host stack — with TU_LOG/RTT, RAM ring-buffer trace, GDB autopsy, J-Link PC-sampling, correlated with capture from the link's other end (Linux PC host, another TinyUSB board, or a Linux gadget peer) and the wire. Long serial debug loop under one held board lock; strictly one instance. Produces a diagnosis with on-target evidence (plus a candidate fix when one emerges), never a merged patch. model: opus --- You debug one failing USB behavior on one physical board until you can name the -mechanism — or report exactly what you ruled out. These repo skills are your -source of truth; read the relevant SKILL.md BEFORE acting: +mechanism — or report exactly what you ruled out. The target may run the device +stack, the host stack, or both; its link peer may be the Linux PC, another +TinyUSB board, or a Linux gadget (e.g. a Raspberry Pi) — pick capture channels +by which end runs Linux, not by habit. Resolve the board's family first +(`ls -d hw/bsp/*/boards/`): Espressif boards are a different backend +entirely — esp-target-debug is your primary playbook there; every other +family uses target-debug's probe recipes directly. These repo skills (each at +`.claude/skills//SKILL.md`) are your source of truth; read the relevant +one BEFORE acting: -- `.claude/skills/usb-target-debug/SKILL.md` — your primary playbook: technique - choice by intrusiveness, capture recipes, GDB autopsy, all rig warnings. -- `.claude/skills/hil/SKILL.md` — host/config selection, board lock protocol, - `hil_test.py` invocation. -- `.claude/skills/usbmon/SKILL.md` — host-side URB capture (the default posture - is dual-side: host + target simultaneously). -- `.claude/skills/usb-sniffer/SKILL.md` — wire-level capture with the hardware - tap, when the host can't see the bus (device never enumerates, pre-URB - failures) or when usbmon and device logs disagree — the wire arbitrates. -- `.claude/skills/usb-debug/SKILL.md` — why the host acted (dmesg/dynamic debug). -- `.claude/skills/usb-recover/SKILL.md` — only when the DUT or fixture wedges - the host stack. +| Skill | Use for | +|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| target-debug | primary playbook — technique choice by intrusiveness, channel choice by link topology, capture recipes, bp/wp budget + cost model, vector catch + fault autopsy, SWO trace, GDB autopsy, rig warnings | +| hil | host/config selection, board lock protocol, `hil_test.py` invocation | +| esp-target-debug | PRIMARY playbook for Espressif boards — built-in USB-Serial-JTAG attach, the PHY map that decides whether JTAG exists, FreeRTOS threads via ESP_RTOS; target-debug still supplies the methodology | +| usbmon | Linux-host URB capture; only when a Linux PC is the link's host (default posture: dual-side, both ends simultaneously) | +| usb-sniffer | wire-level capture (hardware tap): host can't see the bus, usbmon vs target logs disagree, or TinyUSB is the host (no usbmon anywhere) | +| usb-kernel-debug | why the Linux kernel acted (dmesg/dynamic debug); PC host or a Linux gadget peer's device side | +| usb-kernel-recover | only when the DUT or fixture wedges the rig PC's Linux host stack | ## The loop (deliberately serial — no fan-out) diff --git a/.claude/skills/esp-target-debug/SKILL.md b/.claude/skills/esp-target-debug/SKILL.md new file mode 100644 index 000000000..1af9fcf7f --- /dev/null +++ b/.claude/skills/esp-target-debug/SKILL.md @@ -0,0 +1,110 @@ +--- +name: esp-target-debug +description: Use when debugging TinyUSB firmware on Espressif boards (ESP32-S3/P4 on the rig — dcd_dwc2 examples, idf.py builds) with the chips' built-in USB-Serial-JTAG — attach/halt/backtrace, breakpoints, FreeRTOS task lists, console capture — or when JTAG "could not find or open device", the 303a:1001 port vanishes, or the S3's debug port turns into the TinyUSB device. +--- + +# esp-target-debug — Espressif built-in USB-JTAG backend + +Methodology — intrusiveness ladder, board locks, dual-side capture, diagnosis +standards — lives in `target-debug`; this skill is the Espressif backend: a +different gdb, a different openocd (fork), no probe serial (the debugger IS a +USB device), and a PHY story that decides whether JTAG exists at all. +Built-in USB-Serial-JTAG only; external JTAG is a TODO (no rig adapter). + +## The PHY map — decides everything (verified on the rig) + +| Board | USB-SJ vs TinyUSB OTG | JTAG while USB device runs? | +|--------------------------|------------------------------------------------------|-----------------------------| +| espressif_p4_function_ev | separate pins: USB-SJ GPIO24/25 (FS), OTG own HS PHY | **yes — coexist** (verified: 303a:1001 + cafe:4008 simultaneously, gdb attach during live CDC traffic) | +| espressif_s3_devkitm | ONE shared PHY/port | **no** — the same hub port flips 303a:1001 → cafe:4008 as the app boots; openocd fails `esp_usb_jtag: could not find or open device!` | + +Flashing works in ANY PHY state: the rig flashes via the boards' CP2102N UART +bridges (hence `tinyusb.json` esptool uids are CP210x serials, not MACs). The +UART side is also the remote reset: `esptool.py --after hard_reset read_mac`. + +### P4 (Function-EV) notes + +- The board has **no USB-SJ connector** — GPIO24 (D−, white) / GPIO25 (D+, + green) / GND are broken out from header J1 to a rig hub port. Swapped + D+/D− enumerates as `new low-speed USB device` + error -71; correct shows + `new full-speed`. + +### S3 (DevKitM) notes + +- Debugging windows: non-USB firmware (`board_test` — attach/halt/symbol + resolution verified; `usb_new_phy` is absent from the ELF when + `CFG_TUD/TUH_ENABLED` are 0), bootloader/ROM (always stable), or external + JTAG (TODO). +- **Keep-alive quirk (verified)**: with app firmware running and nothing + attached, USB-SJ drops ~4 s after boot (device-side disconnect, then + half-dead `-71` setup failures until reset). Attach a client inside the + window — once it survives the window it stays up. Recover via the UART + reset above. +- PHY mux reference: `RTC_CNTL_RTC_USB_CONF_REG` (0x60008120) bits + `SW_HW_USB_PHY_SEL`/`SW_USB_PHY_SEL` (TRM 10.56); 0 = eFuse/hardware + control (default). `esptool.py read_mem/write_mem` peeks and pokes + registers over plain UART with the chip in download mode. + +## Attach + +```bash +. $HOME/code/esp-idf/export.sh # openocd-esp32, riscv32-/xtensa-esp32s3-elf-gdb, esptool +openocd -c 'set ESP_RTOS FreeRTOS' -f board/esp32p4-builtin.cfg \ + -c 'adapter serial ' & # S3: board/esp32s3-builtin.cfg +riscv32-esp-elf-gdb -batch -ex 'target extended-remote :3333' \ + -ex 'tbreak tud_task_ext' -ex continue -ex bt -ex 'info threads' -ex detach +# S3 is Xtensa: use xtensa-esp32s3-elf-gdb with the same arguments +``` + +- `adapter serial` = the chip MAC **with colons** — the USB-SJ device's + iSerial exactly as `lsusb -v -d 303a:1001` or + `/dev/serial/by-id/usb-Espressif_USB_JTAG_serial_debug_unit_-if00` + prints it. (The `tinyusb.json` esptool uids are the CP2102N *flasher* + serials — a different port; never pass those to openocd.) +- `set ESP_RTOS FreeRTOS` must precede the board cfg: with it, `info threads` + lists every task with name/state/CPU (verified: usbd Running @CPU0, IDLE1 + @CPU1, ...); without it, one bare "Remote target". +- The ELF: `idf.py -B -DBOARD= build` under the example + (CLAUDE.md Espressif notes) — symbolized app backtraces verified + (`tud_task_ext` ← `usb_device_task` ← `vPortTaskWrapper`). +- **Attach may reset the target** (a boot-fresh FreeRTOS tick observed on a + minutes-old session). Until pinned down, do NOT trust built-in-JTAG attach + for post-mortem autopsy of a wedged board (`target-debug`'s + attach-and-halt-only rule); capture state via console or treat the reset + as part of the reproduce cycle. +- Halting still stops USB service: the host may drop the DUT during long + halts; after detach the device may need the UART reset to re-enumerate. + +## Scripted-session gotchas (verified) + +- xtensa-gdb batch `continue`/`interrupt` is async-flaky — for scripted + state reads, halt via openocd telnet :4444 first, then attach gdb to the + stopped target. Interactive sessions are unaffected. +- cpu1 debug-logic examination can fail (`OCD_ID = 00000000`) — + `-c 'set ESP_ONLYCPU 1'` degrades to cpu0-only debugging. +- ROM-frame backtraces (`0x4004xxxx` on S3, `0x4fc0xxxx` on P4, all `??`) + mean the core idles in ROM — break in app code (`tbreak tud_task_ext`) + for symbolized frames. + +## Technique mapping (vs the `target-debug` arsenal) + +| target-debug technique | Espressif backend | +|------------------------|-------------------| +| GDB autopsy, bp/wp | same flow via openocd-esp32 :3333; RISC-V triggers (P4) / Xtensa 2 bp + 2 wp (S3) | +| Vector catch | none — breakpoint the panic handler; `mcause`/`mepc`/`mtval` on P4 | +| SWO / DWT data trace | none — apptrace over JTAG is the analog (untested: needs CONFIG_APPTRACE + app init) | +| RTT / TU_LOG | console on **UART0 = the CP2102 flasher tty** by default (verified); USB-SJ console needs sdkconfig `ESP_CONSOLE_USB_SERIAL_JTAG` (untested) | +| FreeRTOS threads | native — `set ESP_RTOS FreeRTOS` (see Attach) | +| verifybin | `esptool.py verify_flash` (untested) | + +## Rig deltas + +- Locks per `hil` skill; reflash-pristine before release applies unchanged. +- One client per USB-SJ: openocd and a terminal on the USB-SJ CDC side + conflict the same way J-Link clients do. + +## TODO — external JTAG (needs hardware) + +S3 JTAG pins GPIO39–42 (MTCK/MTDO/MTDI/MTMS) + any adapter openocd-esp32 +supports (ESP-Prog/FT2232-class); would give S3 debugging under live USB +traffic. Mind `EFUSE_DIS_PAD_JTAG` / JTAG-source strapping. Unverified. diff --git a/.claude/skills/hil/SKILL.md b/.claude/skills/hil/SKILL.md index 7b76bf1b8..2aeaa10af 100644 --- a/.claude/skills/hil/SKILL.md +++ b/.claude/skills/hil/SKILL.md @@ -7,11 +7,11 @@ description: Use when running TinyUSB Hardware-in-the-Loop (HIL) tests on physic Run TinyUSB HIL tests on real boards. **Run `hostname` first** — it tells you which host you are on, which determines the default config and whether remote mode is possible. -| Host | Local config | Remote (SSH → ci.lan)? | -|------|--------------|------------------------| -| `htpc` (dev PC) | `test/hil/local.json` | yes (large pool, `test/hil/tinyusb.json`) | -| `ci` (the rig) | `test/hil/tinyusb.json` (large pool) | no — can't SSH to htpc, and boards are already local | -| `hifiphile` (external rig) | `test/hil/hfp.json` | no outbound SSH to htpc/ci; SSH-reachable FROM both | +| Host | Local config | Remote (SSH → ci.lan)? | +|----------------------------|--------------------------------------|------------------------------------------------------| +| `htpc` (dev PC) | `test/hil/local.json` | yes (large pool, `test/hil/tinyusb.json`) | +| `ci` (the rig) | `test/hil/tinyusb.json` (large pool) | no — can't SSH to htpc, and boards are already local | +| `hifiphile` (external rig) | `test/hil/hfp.json` | no outbound SSH to htpc/ci; SSH-reachable FROM both | Default to **local**. Use **remote** only when on `htpc` and the user says `remote`/`ci.lan`. Never attempt remote on `ci`. diff --git a/.claude/skills/pre-pr/SKILL.md b/.claude/skills/pre-pr/SKILL.md index d4c35f7e5..3f062db78 100644 --- a/.claude/skills/pre-pr/SKILL.md +++ b/.claude/skills/pre-pr/SKILL.md @@ -38,5 +38,5 @@ Invoke the Workflow tool: - Per-stage table: unit / build: / size / pvs, then HIL per board — pass/fail with the first error for each failure. - If the hardware result has non-empty `locked` (a CI job held those boards): ask the user with AskUserQuestion — **Force now** (re-invoke `hil-validate` with `force: true` for those boards; user accepts the risk of colliding with a mid-test CI job), **Keep waiting** (re-invoke `hil-validate` for them after a few minutes; ask again if still locked), or **Accept** the partial verdict. Never force without the user's answer. -- Wedged boards: point at `.claude/skills/usb-recover/SKILL.md`. +- Wedged boards: point at `.claude/skills/usb-kernel-recover/SKILL.md`. - End with a clear ship / no-ship verdict and what to fix first. diff --git a/.claude/skills/target-debug/SKILL.md b/.claude/skills/target-debug/SKILL.md new file mode 100644 index 000000000..1dc55440f --- /dev/null +++ b/.claude/skills/target-debug/SKILL.md @@ -0,0 +1,385 @@ +--- +name: target-debug +description: Use when TinyUSB firmware — device or host stack — misbehaves on real hardware and capture from the other end can't explain it: a HIL test fails but usbmon shows only Submits with no Completes, the device silently NAKs, wedges, STALLs, babbles, or drops data, EP0 starves, tuh_ enumeration of an attached device fails, an ISR or DCD/HCD state bug is suspected — and you need target-side evidence: TU_LOG/RTT logs, GDB state dumps, a RAM ring-buffer event trace, or PC-sampling of where the core spins. +--- + +# target-debug — target-side capture & debugging on the HIL rig + +The **target** is whichever MCU runs TinyUSB — device stack (`dcd_*`), host +stack (`hcd_*`/`tuh_*`), or both. Its link peer is not always a Linux PC: a +TinyUSB host may face another TinyUSB board or a Linux gadget (e.g. a +Raspberry Pi). Pick capture channels by which end runs Linux, not by habit: + +| Skill | Answers | Exists when | +|--------------------|----------------------------------------------------|---------------------------------------------------| +| `usbmon` | what the Linux host exchanged (URBs) | a Linux PC is the link's host | +| `usb-kernel-debug` | why the Linux kernel acted (dmesg / dynamic debug) | Linux on either end: PC host or Linux gadget peer | +| **`target-debug`** | **what the target did** (logs, driver state, PC) | always — either role, needs a debug probe | +| `usb-sniffer` | what crossed the wire (PIDs, handshakes, resets) | hardware tap cabled in — role-agnostic | + +For enumeration/transfer bugs the default posture is **dual-side capture** — +both ends simultaneously: usbmon + a target +channel when a Linux PC is the host; TinyUSB-as-host has no usbmon on either +end — target channel + the wire (`usb-sniffer`), plus `usb-kernel-debug` on a +Linux gadget peer. + +## Rig discipline — lock first, always + +Hold the board lock for the WHOLE manual session; never stop the +actions-runner (see the `hil` skill for the full lock protocol): + +```bash +python3 test/hil/board_lock.py hold --reason "target debug: " +# ... instrument / build / flash / capture / GDB ... +python3 test/hil/board_lock.py release +``` + +Board → probe mapping: `test/hil/tinyusb.json` — `flasher.name` is the probe +family, `flasher.uid` the **probe serial** (many identical probes on the rig): + +- Select the probe by serial: J-Link `-SelectEmuBySN `, its GDB server + `-select usb=`, OpenOCD `-c 'adapter serial '`. +- `JLINK_DEVICE` / `OPENOCD_OPTION`: from + `hw/bsp//boards//board.cmake` (or `board.mk`); family via + `ls -d hw/bsp/*/boards/`. +- Run on the host that owns the probe — config `test/hil/tinyusb.json` on ci, + `local.json` on htpc (`hil` skill). +- Espressif boards (S3/P4): different toolchain, probe model, and PHY + constraints entirely — read `esp-target-debug` first. + +## Pick the least intrusive technique that can answer the question + +Observation can mask the bug — the ch32v307 Heisenbug changed behavior under +logging *and* under the debugger. If the bug disappears when instrumented, +that IS a finding (timing-sensitive): move down in intrusiveness, not up. + +| Technique | Intrusiveness | Reach for it when | +|------------------------------------|--------------------------------|---------------------------------------------------------| +| PC-sampling | none — no halt, no code change | core wedged/spinning somewhere unknown (rusb2 FRDY) | +| SWO exception trace / hw PC-sample | none — needs SWO pin wired | ISR ordering/timing with zero code change | +| DWT data trace | none — needs SWO pin wired | stream one address's accesses: value + accessor PC | +| Vector catch | none until a fault fires | crash-shaped wedges — autopsy AT the faulting pc | +| RAM ring-buffer | ~tens of cycles per event | ISR ordering/timing bugs (musb babble) | +| TU_LOG (RTT) | µs per line | logic bugs that survive logging (J-Link or OpenOCD rtt) | +| TU_LOG (UART) | ms per line — blocking write | same, when no debug-probe RTT path | +| dprintf / conditional breakpoint | halt+resume per hit (~ms) | low-rate probes post-wedge; never ISR-rate events | +| GDB halt / breakpoints | stops USB service entirely | post-mortem state autopsy once wedged | + +## PC-sampling, SWO trace & DWT data trace — watch without halting + +`DWT_PCSR` (0xE000101C) returns the current PC on every read, target running +(Cortex-M3+; optional on M0+, reads 0 if absent; 0xFFFFFFFF = core halted or +WFI-asleep — `mem32 E000EDF0, 1`, DHCSR bit 17 S_HALT, tells which). One +probe serves one client: quit JLinkExe before starting JLinkGDBServer on the +same probe. Nailed the rusb2 FRDY wedge: + +```bash +for i in $(seq 300); do echo 'mem32 E000101C, 1'; done \ + | JLinkExe -device $JLINK_DEVICE -SelectEmuBySN -if swd -speed 4000 -autoconnect 1 -nogui 1 \ + | awk '/E000101C = /{print $3}' | sort | uniq -c | sort -rn | head +arm-none-eabi-addr2line -e -f -a 0x ... # PCs → functions +``` + +OpenOCD variant: repeat `mdw 0xE000101C` over telnet :4444. The histogram's +top entries are the spin site; a flat histogram = core is servicing normally. + +### SWO — hardware-timed trace on one pin (J-Link; verified on F407) + +If SWO (TRACESWO) is wired, DWT emits packets with ZERO code change: +**exception trace** (DWT_CTRL bit16 — every IRQ enter/exit, timestamped) and +**hardware PC sampling** (bit12), better histograms than DWT_PCSR polling. +SWOViewer tools decode only ITM *stimulus* (TinyUSB emits none) — capture +raw: + +```bash +# JLinkExe -CommandFile: +w4 E0001000, 0x00011401 # EXCTRCENA|PCSAMPLENA|SYNCTAP|CYCCNTENA +SWOStart 4000000 # explicit speed — autodetect fails headless +Sleep 3000 +SWORead # hex: 0x17+4B LE = PC sample, 0x0E+2B = IRQ enter/exit +``` + +Verified: 680 KB in 3 s (flash-range PC samples + SysTick enter/exit). +SWORead stuck at 0 = SWO pin not wired (many boards route only SWDIO/SWCLK). +Restore DWT_CTRL when done. + +### DWT data trace — stream one variable's accesses (value + PC), zero code + +The watchpoint comparators' non-halting sibling (ARMv7-M ARM Table C1-21; +absent on ARMv6-M): emit a packet on every access to a watched address +instead of halting. Verified on F407 (J-Link) and H743 (OpenOCD/ST-Link) — +both streamed `system_ticks`' live value plus the accessor PC +(`tusb_time_millis_api`): + +```bash +w4 E0001020, <&variable> # DWT_COMP0 (JLinkExe shown; OpenOCD: same via mww) +w4 E0001024, 0 # DWT_MASK0 = exact address +w4 E0001028, 0x3 # FUNCTION 0b0011: value + accessor-PC packets (0b0010: value only) +# stream: 0x47+4B = accessor PC, 0x87+4B = value, 0x70 = timestamp +``` + +Caveats: traces reads AND writes (no write-only encoding) — a variable the +main loop polls floods the pipe with read packets and squeezes out value +packets (seen on F407); disarm (`FUNCTION=0`) when done; costs one of the +DWT comparators. + +### Enabling SWO — the chain, and the vendor part that bites + +DEMCR.TRCENA → ITM (TCR/TER) → SWO/TPIU (protocol + prescaler) → pin mux. +Tools set the first three (`SWOStart` on SEGGER; `swo`/`tpiu` object +`enable` + `itm ports on` on OpenOCD) — pin mux and trace clocks are +per-family: + +- STM32F4: debug pins default to trace — nothing to configure. +- STM32H7 (verified, ST-Link): DBGMCU trace clocks + **PB3 muxed to AF0 by + hand** + native `stlink-dap.cfg` (the hla transport's tpiu path silently + does nothing) + the cfg-provided `stm32h7x.swo` object (`stm32h7x.tpiu` + is the parallel port — rejects uart). traceclk = c_ck 400 MHz, not HCLK: + too-slow guesses give ratio-garbled bytes, too-fast gives silence. + +```bash +openocd -f interface/stlink-dap.cfg -c 'adapter serial ' -f target/stm32h7x.cfg -c init \ + -c "mww 0x5C001004 0x00700000" \ + -c 'set m [read_memory 0x58020400 32 1]; mww 0x58020400 [expr {([lindex $m 0] & ~0xC0) | 0x80}]' \ + -c 'set a [read_memory 0x58020420 32 1]; mww 0x58020420 [expr {[lindex $a 0] & ~0xF000}]' \ + -c "stm32h7x.swo configure -protocol uart -traceclk 400000000 -pin-freq 2000000 -output /tmp/swo.bin" \ + -c "stm32h7x.swo enable" -c "itm ports on" \ + -c "sleep 3000" -c "stm32h7x.swo disable" -c shutdown # then decode /tmp/swo.bin +``` + +## Vector catch + fault autopsy — catch the crash, not the wedge + +A wedge that is really a fault (HardFault loop, lockup) autopsies best AT +the faulting instruction. Two hardware-proven gotchas: **FPB/DWT comparators +survive reflash and dead sessions** — a stale one fires as a phantom SIGTRAP +at an unrelated line of NEW firmware — and J-Link's reset strategy manages +vector-catch bits: scrub first, arm AFTER reset: + +```gdb +# scrub: FP_COMP0..5 = 0xE0002008..201C, DWT_FUNCTIONn = 0xE0001028 + n*0x10 +set *(unsigned*)0xE0002008 = 0 +# ... (repeat per comparator; count = the GDB section's budget reads) +# arm (after monitor reset; tool-agnostic — works via JLinkExe w4 too): +set *(unsigned*)0xE000EDFC |= (1<<10)|(1<<9)|(1<<8)|(1<<7)|(1<<6)|(1<<5)|(1<<4) +# = VC_HARDERR|INTERR|BUSERR|STATERR|CHKERR|NOCPERR|MMERR; bit0 VC_CORERESET halts at reset +``` + +OpenOCD native: `cortex_m vector_catch hard_err bus_err state_err chk_err mm_err`. +It halts at exception ENTRY (pc = handler, LR = EXC_RETURN 0xFFFFFFFx); decode: + +```gdb +p/x *(unsigned*)0xE000ED28 # CFSR — low byte MemManage, byte1 BusFault, top half UsageFault +p/x *(unsigned*)0xE000ED2C # HFSR — bit30 FORCED = an escalated lower-priority fault +p/x *(unsigned*)0xE000ED38 # BFAR — faulting address (valid if CFSR bit15 BFARVALID) +x/8wx $msp # stacked frame: r0 r1 r2 r3 r12 lr pc xpsr — pc = culprit +# frame is on PSP when EXC_RETURN bit2 is set (LR = 0xFFFFFFFD — FreeRTOS +# tasks run on PSP): then x/8wx $psp instead. LR 0xFFFFFFF1/E9 = MSP. +``` + +`addr2line -e ` names the line (verified: CFSR 0x8200, +BFAR = the bad address, stacked pc = the faulting ldr). Loads fault +precisely; stores usually IMPRECISERR (BFAR invalid, pc late). ARMv6-M has no +CFSR/BFAR, only VC_HARDERR|VC_CORERESET — stacked frame alone. Still a halt +(host URB timeouts apply); clear DEMCR (`&= ~0x7F0`) before handing back; +RISC-V: breakpoint the trap handler; mcause/mepc/mtval are the CFSR/BFAR analogs. + +## RAM ring-buffer trace + +The zero-print instrument (cracked the musb babble): a small event ring in the +dcd/hcd, dumped over GDB after the failure. Single-writer (ISR) — no locking: + +```c +typedef struct { uint16_t ev; uint16_t a; uint32_t b; } dbg_ev_t; +#define DBG_N 512 // power of two +static volatile dbg_ev_t dbg_ring[DBG_N]; // volatile REQUIRED: -Os dead-store- +static volatile uint32_t dbg_wr; // eliminates a write-only static array +static inline void DBG_EV(uint16_t ev, uint16_t a, uint32_t b) { + uint32_t i = dbg_wr++; + dbg_ring[i & (DBG_N - 1)] = (dbg_ev_t){ ev, a, b }; +} +// call sites: DBG_EV(__LINE__, ep_addr, count); — __LINE__ as event id +``` + +After building, `nm` the ELF for `dbg_ring`/`dbg_wr` — if they're missing the +compiler deleted your instrument and the run will "reproduce" with an empty ring. + +Order is the index; if durations matter add a `uint32_t t = DWT->CYCCNT` field +(enable once: `CoreDebug->DEMCR |= CoreDebug_DEMCR_TRCENA_Msk; DWT->CTRL |= 1;` +RISC-V: read `mcycle`). Let the failure happen, halt, then: + +```gdb +p dbg_wr # total events; oldest slot = dbg_wr & (DBG_N-1) once wrapped +p dbg_ring +dump binary memory /tmp/ring.bin &dbg_ring[0] &dbg_ring[512] +``` + +## TU_LOG capture + +Build with `LOG=2` (`LOG=3` adds per-transfer noise and much more timing skew). +`LOGGER=rtt` routes it over the debug probe — no UART wiring. SEGGER's host +tools need a J-Link, but OpenOCD serves the same RTT buffer on ST-Link / +CMSIS-DAP / WCH-Link boards: + +```bash +# RTT: JLinkGDBServer from CLAUDE.md "GDB Debugging" + -RTTTelnetPort, then: +timeout 20s JLinkRTTClient > /tmp/rtt.log # non-interactive capture +# UART (board's debug serial, if wired): +stty -F /dev/ttyACM 115200 raw && timeout 20s cat /dev/ttyACM | tee /tmp/uart.log +``` + +```bash +# OpenOCD RTT (any probe OpenOCD drives) — in telnet :4444 (or -c equivalents): +rtt setup 0x20000000 0x8000 "SEGGER RTT" # RAM ORIGIN + LENGTH (from the .ld/map) +rtt start # after firmware booted; rerun after each reflash +rtt server start 19021 0 +# then: timeout 20s nc localhost 19021 > /tmp/rtt.log +``` + +OpenOCD polls — bursty logs can drop lines; prefer J-Link where both +exist. The drain-model warning below applies unchanged. + +An RTT-built firmware that has since wedged still holds a log tail in RAM — +but ONLY what fits the drain model: the default SEGGER mode (NO_BLOCK_SKIP) +**drops** writes once the ring fills with no reader, so an undrained target +holds the first KB after boot, not the wedge tail. There is no overwrite mode +in stock SEGGER RTT (only SKIP/TRIM/BLOCK): post-mortem RTT is evidence only +if a live drain was running — otherwise instrument with the RAM ring above. +Use `JLinkGDBServer -RTTTelnetPort 19021` + `JLinkRTTClient` for the drain +(proven; note the server briefly halts the core on connect). `JLinkRTTLogger` +fails to find the control block on some parts (LPC4088) even when it exists +and even given `-RTTAddress`; don't fight it — `nm` the ELF for `_SEGGER_RTT`, +read the aUp[0] descriptor (`mem32`), `savebin` the buffer — debug-AP RAM +reads don't halt the target. + +## GDB — state autopsy and watchpoints + +Connect/load recipes per probe family (J-Link, OpenOCD for ST-Link / +CMSIS-DAP / WCH-Link) are in CLAUDE.md "GDB Debugging"; script sessions with +JLinkGDBServer `-singlerun` — the server exits with the connection, and +back-to-back relaunches race the probe handle and hang. Release builds keep +DWARF (`MinSizeRel`), so `p`/struct access works on HIL firmware. + +**Autopsy of a wedged board: attach and halt ONLY** — skip CLAUDE.md's +`monitor reset halt` + `load` (those are for fresh starts; a reset destroys +the evidence). Symbolize with the ELF that is actually flashed — +`/cmake-build-//.elf` from the run that +wedged; do not rebuild while the wedge is still on the board. The debug-loop +specifics: + +```gdb +p/x _usbd_dev.ep_status # device stack: usbd [epnum][dir] (1=IN): busy/stalled/claimed +p _usbh_devices[0] # host stack: usbh per-device state (addr, enum/config) +p/x # per-port names — read the board's dcd_*.c first +x/32wx # raw EP/FIFO regs; base = the macro the dcd uses +watch xfer_status[2][1].total_len # HW watchpoint (Cortex-M: ~4); dwc2 names shown +break dcd_int_handler # works, but see warning below +``` + +**Hardware budget — read it off the chip** (verified: F407/M4 = 6 bp + 4 wp, +rp2040/M0+ = 4 + 2; M7 typically 8/4): + +```gdb +p ((*(unsigned*)0xE0002000)>>4) & 0xF # FPB NUM_CODE = hw breakpoints (M7 adds bits[14:12]) +p (*(unsigned*)0xE0001000)>>28 # DWT_CTRL NUMCOMP = watchpoint comparators +``` + +- `hbreak`/`thbreak` force a hardware breakpoint (software breaks in flash + need flash-breakpoint support — J-Link has it; OpenOCD: `bp 2 hw`); + `tbreak` = one-shot. +- `watch -l ` watches the address expr evaluates to once — almost + always what you want; `rwatch`/`awatch` trap reads/any access (hardware- + only — they error, never fall back). OpenOCD adds a data-VALUE match GDB + can't express: `wp 4 w [mask]` — catch who writes 0 into a + busy flag, ignoring writes of 1. +- **Demand the word "Hardware" in the confirmation.** `watch` silently falls + back to a software watchpoint when no DWT comparator fits — GDB then + single-steps the whole program, hundreds of times slower: certain USB + death. Plain `Watchpoint 2:` = delete it and narrow the expression + (`watch -l`, cast to a 4-byte int). +- Conditional breaks (`break ... if ep_addr==0x81`) and `dprintf + ,"fmt",args` (printf without recompiling; keep `dprintf-style gdb`) + are host-evaluated — no Cortex-M agent expressions in our stubs: every hit + halts+resumes (~ms) even when the condition is false. Post-wedge/cold + paths only; ISR-rate events belong in the RAM ring. +- `commands ... end` (start `silent`, end `continue`) auto-collects + evidence per hit — same halt-per-hit cost. +- Stepping with the USB ISR firing between steps is chaos: OpenOCD + `cortex_m maskisr steponly`. The bus runs either way — the host may still + reset a halted-looking device. +- Poking state while halted (`set var _usbd_dev.ep_status[2][1].busy = 0`) + tests a hypothesis but invalidates the post-mortem — dump first, poke after. +- FreeRTOS examples: `-rtos GDBServer/RTOSPlugin_FreeRTOS` (OpenOCD: `-rtos + FreeRTOS`) → `info threads` lists every task with state/prio/frame + (verified: 6 tasks). It populates only after a + run→stop cycle — plain attach shows one 0xDEAD placeholder. Semihosting is + never the answer (traps + halts per call — RTT instead). **Monitor-mode + debugging** (J-Link, M3+) keeps chosen IRQs serviced at a breakpoint — + needs SEGGER's JLINK_MONITOR files + `SetMonModeDebug=1`; not set up here: + (untested). + +While halted the device answers **nothing**: host control transfers time out +in ~5 s and the OS may reset/re-enumerate — after `continue`, the bus traffic +shows recovery, not the original bug. Prefer one halt for a post-mortem dump +over stepping through live USB traffic. + +## Dual-side capture — the default for enumeration/transfer bugs + +Start both channels, then trigger the failing test (Linux-PC-host shown; +TinyUSB-as-host: swap usbmon for `usb-sniffer`, + `usb-kernel-debug` on a +Linux gadget peer): + +```bash +.claude/skills/usbmon/scripts/usbcap.sh cafe: 30 /tmp/host.pcapng & # host URBs (usbmon skill) +timeout 30s JLinkRTTClient > /tmp/target.rtt & # target (or ring dump after) +wait +``` + +RTT lines and ring events carry no wall-clock: correlate on unambiguous +anchors — bus reset, SET_ADDRESS, the first transfer on the failing EP — then +lay device events between anchors in host-URB order. Logging the SOF/frame +number on the target gives a shared clock when you need finer alignment. +When host and target evidence disagree, or the host sees nothing at all, add +the wire itself: `usb-sniffer` skill (hardware tap, PID-level). + +## Manuals + +- J-Link (UM08001): — flash breakpoints, RTT, SWO, monitor mode, Commander. +- OpenOCD: — `rtt`, `bp`/`wp`, `cortex_m vector_catch`/`maskisr`, `itm`/`tpiu`. +- "Debugging with GDB" (§5.1 = break/watch/dprintf): Tenth Edition (GDB 18) + via calibre/`read-doc`, or + `curl -sL -o /tmp/gdb.pdf https://sourceware.org/gdb/current/onlinedocs/gdb.pdf` + (the HTML mirror blocks fetchers). Installed `arm-none-eabi-gdb` + `help ` is authoritative here. + +## Warnings + +- **Halting/resetting via the probe does NOT disconnect the device**: a DWC2 + soft-connect pullup stays up through core halt *and* reset, so the host's + stuck URBs stay stuck and a wedged DUT stays wedged — recover the Linux + host side with the `usb-kernel-recover` skill. +- **A bug that vanishes under LOG=2 is a timing bug**, not fixed: switch to + the ring buffer; if it vanishes under GDB too, PC-sampling only. +- **UART TU_LOG blocks in the write path** (worst perturbation, including + inside the ISR); RTT is much cheaper but not free; `LOG=3` multiplies both. +- Flash/GDB only with the board lock held; a `hold` refused with reason + `hil_test.py` means CI is mid-test on that board — wait, don't force. +- **Instrumentation is temporary**: before `release`, reflash pristine + firmware (the next CI run must not inherit a debug build) and revert the + instrumentation diff — or hand it over explicitly with the diagnosis. +- **A register snapshot without a validity anchor lies**: J-Link tool sessions + can reset or briefly halt the DUT as a side effect, and a snapshot of a + freshly-reset chip (e.g. NVIC ISER = 0) reads like a smoking gun. Read DHCSR + (0xE000EDF0: bit 17 S_HALT, bit 25 S_RESET_ST) with every snapshot, and + cross-check against something the device demonstrably still does. +- **"Flash OK" can lie** (silent no-op — old firmware keeps running). When + behavior contradicts the flashed code: `objcopy -O binary fw.elf + /tmp/fw.bin`, then `verifybin /tmp/fw.bin,` (J-Link, verified) + or `verify_image` (OpenOCD); on mismatch reflash before debugging further. +- **A marginal link can fake a deterministic firmware bug** — down to failing + the same test at the same iteration twice. "USB disconnect" in dmesg on a + freshly re-cabled port (high devnum = churn) means the plug, not the code: + first sustained bulk traffic is when a bad contact drops. Before declaring a + regression, re-run the OLD build on the SAME link state — and if a bisect + exonerates every hunk, believe it: re-test the exact failing binary. +- **Release your manual lock before `hil_test.py`** — it self-locks each board + and fails immediately on your own hold (`hil` skill). diff --git a/.claude/skills/usb-debug/SKILL.md b/.claude/skills/usb-debug/SKILL.md deleted file mode 100644 index 20ab7d764..000000000 --- a/.claude/skills/usb-debug/SKILL.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -name: usb-debug -description: Use when USB enumeration fails or misbehaves and usbmon alone can't explain WHY the host acted — port reset storms, repeated re-enumeration, address errors, xHCI ring/command errors, "device descriptor read error", babble, or when you need the host driver's own reasoning from dmesg on the ci HIL rig. ---- - -# usb-debug — host-side kernel dynamic debug for USB - -usbmon shows the URBs; kernel **dynamic debug** shows the host driver's -*reasoning* usbmon can't: port resets and their causes, enumeration retries, -address (re)assignment, EP halts, xHCI ring/command errors. - -Run this skill's `scripts/usb_dyndbg.sh` with `sudo` (abbreviated to -`usb_dyndbg.sh` in the examples below). It flips the dynamic-debug print flag -for an allowlisted set of USB host modules only: - -```bash -sudo usb_dyndbg.sh on usbcore xhci_hcd # enable +p; pick modules from `lsusb -t` Driver= -sudo usb_dyndbg.sh status [module] # list enabled print sites -sudo usb_dyndbg.sh off usbcore xhci_hcd # ALWAYS turn off when done — very noisy -``` - -Allowlisted modules: `usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd -ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 cdc_acm usb_storage uas`. - -## Workflow - -1. `sudo usb_dyndbg.sh on usbcore ` — `usbcore` for enumeration/hub - logic, plus the controller module (`lsusb -t` shows the driver per bus). -2. Reproduce (replug / re-enumerate / rerun the failing test) while following - `sudo dmesg -w` (or grab `sudo dmesg | tail` afterwards). -3. `sudo usb_dyndbg.sh off ...` — leaving it on floods the log and skews timing. - -Pair with the `usbmon` skill: usbmon for what crossed the bus, dynamic debug for -why the host reacted. For a wedged device/bus use the `usb-recover` skill. - -Requires `CONFIG_DYNAMIC_DEBUG` and mounted debugfs (standard on distro kernels). diff --git a/.claude/skills/usb-kernel-debug/SKILL.md b/.claude/skills/usb-kernel-debug/SKILL.md new file mode 100644 index 000000000..e4169b049 --- /dev/null +++ b/.claude/skills/usb-kernel-debug/SKILL.md @@ -0,0 +1,47 @@ +--- +name: usb-kernel-debug +description: Use when USB enumeration fails or misbehaves and packet/URB capture can't explain WHY the Linux kernel acted — port reset storms, repeated re-enumeration, address errors, xHCI ring/command errors, "device descriptor read error", babble — on whichever end of the link runs Linux: the PC host when testing a TinyUSB device, or a Linux gadget peer (e.g. Raspberry Pi) when testing the TinyUSB host stack. +--- + +# usb-kernel-debug — Linux kernel dynamic debug for USB + +Kernel **dynamic debug** shows the Linux side's *reasoning* that packet +capture can't: port resets and their causes, enumeration retries, address +(re)assignment, EP halts, xHCI ring/command errors. It applies wherever Linux +sits in the link — the rig PC when it is the host, or a Linux gadget peer +(dwc2/UDC + gadget modules) when TinyUSB is the host. It cannot see inside +the TinyUSB MCU — that is the `target-debug` skill. + +Run this skill's `scripts/usb_dyndbg.sh` with `sudo` (abbreviated to +`usb_dyndbg.sh` in the examples below). It flips the dynamic-debug print flag +for an allowlisted set of USB modules only: + +```bash +sudo usb_dyndbg.sh on usbcore xhci_hcd # enable +p; pick modules from `lsusb -t` Driver= +sudo usb_dyndbg.sh status [module] # list enabled print sites +sudo usb_dyndbg.sh off usbcore xhci_hcd # ALWAYS turn off when done — very noisy +``` + +Allowlisted modules: `usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd +ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 dwc3 cdc_acm usb_storage uas +libcomposite udc_core` (`dwc2`/`dwc3` + the last two cover a Linux gadget +peer's device side). + +## Workflow + +1. `sudo usb_dyndbg.sh on usbcore ` — `usbcore` for enumeration/hub + logic, plus the controller module (`lsusb -t` shows the driver per bus). + On a gadget peer: `dwc2` (or `dwc3`) + `udc_core` + `libcomposite` instead — + run on the peer itself (its SSH/serial console); the script is self-contained, + copy it over or use the raw `dynamic_debug/control` writes from the `usbmon` + skill. +2. Reproduce (replug / re-enumerate / rerun the failing test) while following + `sudo dmesg -w` (or grab `sudo dmesg | tail` afterwards). +3. `sudo usb_dyndbg.sh off ...` — leaving it on floods the log and skews timing. + +On a Linux-PC-host link, pair with the `usbmon` skill: usbmon for what crossed +the bus, dynamic debug for why the kernel reacted. A gadget peer's UDC has no +usbmon — pair with `usb-sniffer` on the wire instead. For a wedged device/bus +on the rig PC use the `usb-kernel-recover` skill. + +Requires `CONFIG_DYNAMIC_DEBUG` and mounted debugfs (standard on distro kernels). diff --git a/.claude/skills/usb-debug/scripts/usb_dyndbg.sh b/.claude/skills/usb-kernel-debug/scripts/usb_dyndbg.sh similarity index 74% rename from .claude/skills/usb-debug/scripts/usb_dyndbg.sh rename to .claude/skills/usb-kernel-debug/scripts/usb_dyndbg.sh index 0dc880469..3923cdc6d 100755 --- a/.claude/skills/usb-debug/scripts/usb_dyndbg.sh +++ b/.claude/skills/usb-kernel-debug/scripts/usb_dyndbg.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash -# usb_dyndbg.sh — toggle kernel dynamic-debug on USB host drivers; run with sudo. -# Flips +p/-p only on an allowlisted set of USB modules, so it can't reach -# arbitrary kernel debug or unrelated subsystems. +# usb_dyndbg.sh — toggle kernel dynamic-debug on USB drivers (host or gadget +# side); run with sudo. Flips +p/-p only on an allowlisted set of USB modules, +# so it can't reach arbitrary kernel debug or unrelated subsystems. # # Usage: # sudo usb_dyndbg.sh on ... # enable +p (e.g. usbcore xhci_hcd) @@ -10,8 +10,9 @@ set -euo pipefail CTL=/sys/kernel/debug/dynamic_debug/control -# Allowlist: USB host-controller + core + common host class drivers. -ALLOW='usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 cdc_acm usb_storage uas' +# Allowlist: USB core + host-controller + common class drivers, plus the +# gadget/UDC side of a Linux peer (dwc2/dwc3, udc_core, libcomposite). +ALLOW='usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 dwc3 cdc_acm usb_storage uas libcomposite udc_core' die() { echo "usb_dyndbg: $*" >&2; exit 1; } usage() { diff --git a/.claude/skills/usb-recover/SKILL.md b/.claude/skills/usb-kernel-recover/SKILL.md similarity index 91% rename from .claude/skills/usb-recover/SKILL.md rename to .claude/skills/usb-kernel-recover/SKILL.md index beb6fd862..3f03722fe 100644 --- a/.claude/skills/usb-recover/SKILL.md +++ b/.claude/skills/usb-kernel-recover/SKILL.md @@ -1,9 +1,9 @@ --- -name: usb-recover -description: Use when a USB device or fixture on the ci HIL rig is stuck, hung, not enumerating, or wedged after a failed flash or test, or when processes touching USB (testusb, JLinkExe, uhubctl, libusb tools) start hanging in D state. +name: usb-kernel-recover +description: Use when a USB device or fixture attached to the ci HIL rig's Linux host is stuck, hung, not enumerating, or wedged after a failed flash or test, or when processes touching USB (testusb, JLinkExe, uhubctl, libusb tools) start hanging in D state. Linux-kernel-side only — a bus owned by a TinyUSB host is out of reach (reset the target / cycle its VBUS instead); the rig's probes and serial fixtures always remain in scope. --- -# USB Recovery on the HIL Rig +# USB Recovery on the HIL Rig (Linux kernel side) Run this skill's `scripts/usb_recover.sh` with `sudo` (abbreviated to `usb_recover.sh` in the examples below). It wraps the sysfs reset actions, a diff --git a/.claude/skills/usb-recover/scripts/usb_recover.sh b/.claude/skills/usb-kernel-recover/scripts/usb_recover.sh similarity index 100% rename from .claude/skills/usb-recover/scripts/usb_recover.sh rename to .claude/skills/usb-kernel-recover/scripts/usb_recover.sh diff --git a/.claude/skills/usb-sniffer/SKILL.md b/.claude/skills/usb-sniffer/SKILL.md index 7c2cd2644..8d070709f 100644 --- a/.claude/skills/usb-sniffer/SKILL.md +++ b/.claude/skills/usb-sniffer/SKILL.md @@ -1,22 +1,24 @@ --- name: usb-sniffer -description: Use when you need wire-level USB evidence that host-side capture can't provide — a device that never enumerates (usbmon shows nothing or only Submits), suspected NAK storms/STALL/babble/bad handshakes, bus-reset or enumeration timing, split-transaction issues, or a usbmon-vs-device-log disagreement the wire must arbitrate. Captures LS/FS/HS packets (PIDs, tokens, handshakes, SE0/line states) with the ataradov usb-sniffer hardware into Wireshark pcapng. +description: Use when you need wire-level USB evidence that host-side capture can't provide — a device that never enumerates (usbmon shows nothing or only Submits), suspected NAK storms/STALL/babble/bad handshakes, bus-reset or enumeration timing, split-transaction issues, a usbmon-vs-device-log disagreement the wire must arbitrate, or any link where TinyUSB is the host (no Linux PC host to run usbmon on). Captures LS/FS/HS packets (PIDs, tokens, handshakes, SE0/line states) with the ataradov usb-sniffer hardware into Wireshark pcapng. --- # usb-sniffer — wire-level capture with the ataradov hardware analyzer Extends the debugging trio with the layer below URBs: -| Skill | Answers | -|---|---| -| `usbmon` | what the host software exchanged (URBs) | -| `usb-debug` | why the host acted (dmesg / dynamic debug) | -| `usb-target-debug` | what the device firmware did | -| **`usb-sniffer`** | **what actually crossed D+/D-** (PIDs, handshakes, resets, timing) | +| Skill | Answers | +|--------------------|--------------------------------------------------------------------| +| `usbmon` | what a Linux PC host exchanged (URBs) | +| `usb-kernel-debug` | why the Linux kernel acted (dmesg / dynamic debug) | +| `target-debug` | what the TinyUSB target did (device or host role) | +| **`usb-sniffer`** | **what actually crossed D+/D-** (PIDs, handshakes, resets, timing) | Reach for it when usbmon can't see (device never binds, pre-enumeration -failures) or can't be trusted (URB completed but did the wire really ACK?). -For everything visible in URBs, usbmon is cheaper — no hardware, no locks. +failures), can't be trusted (URB completed but did the wire really ACK?), or +doesn't exist — a link where TinyUSB is the host has no usbmon on either end +(an MCU host runs no kernel; a Linux gadget peer's UDC bypasses usbmon). +Where a Linux PC is the host, usbmon is cheaper — no hardware, no locks. ## Rig inventory — find the sniffer and what it taps diff --git a/.claude/skills/usb-target-debug/SKILL.md b/.claude/skills/usb-target-debug/SKILL.md deleted file mode 100644 index 71fac98f2..000000000 --- a/.claude/skills/usb-target-debug/SKILL.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -name: usb-target-debug -description: Use when a TinyUSB device misbehaves on real hardware and host-side capture can't explain it — a HIL test fails but usbmon shows only Submits with no Completes, the device silently NAKs, wedges, STALLs, babbles, or drops data, EP0 starves, an ISR or DCD/HCD state bug is suspected — and you need device-side evidence: TU_LOG/RTT logs, GDB state dumps, a RAM ring-buffer event trace, or PC-sampling of where the core spins. ---- - -# usb-target-debug — device-side capture & debugging on the HIL rig - -Completes the debugging trio (the `usb-sniffer` skill adds a fourth, -wire-level view when hardware tapping is available): - -| Skill | Answers | -|---|---| -| `usbmon` | what the host actually exchanged (URBs) | -| `usb-debug` | why the host acted (dmesg / dynamic debug) | -| **`usb-target-debug`** | **what the device did** (logs, driver state, PC) | -| `usb-sniffer` | what crossed the wire (PIDs, handshakes, resets — hardware tap) | - -For enumeration/transfer bugs the default posture is **dual-side capture** — -usbmon on the host *and* a target-side channel, simultaneously — not -host-first-then-escalate. - -## Rig discipline — lock first, always - -Hold the board lock for the WHOLE manual session; never stop the -actions-runner (see the `hil` skill for the full lock protocol): - -```bash -python3 test/hil/board_lock.py hold --reason "target debug: " -# ... instrument / build / flash / capture / GDB ... -python3 test/hil/board_lock.py release -``` - -Board → probe mapping: `test/hil/tinyusb.json` — `flasher.name` is the probe -family, `flasher.uid` the **probe serial** (many identical probes on the rig: -J-Link needs `-SelectEmuBySN ` / GDB server `-select usb=`; OpenOCD -`-c 'adapter serial '`). `JLINK_DEVICE` / `OPENOCD_OPTION` come from -`hw/bsp//boards//board.cmake` (or `board.mk`); find the family -with `ls -d hw/bsp/*/boards/`. Run on the host that owns the probe — -config is `test/hil/tinyusb.json` on ci, `local.json` on htpc (`hil` skill). - -## Pick the least intrusive technique that can answer the question - -Observation can mask the bug — the ch32v307 Heisenbug changed behavior under -logging *and* under the debugger. If the bug disappears when instrumented, -that IS a finding (timing-sensitive): move down in intrusiveness, not up. - -| Technique | Intrusiveness | Reach for it when | -|---|---|---| -| PC-sampling | none — no halt, no code change | core wedged/spinning somewhere unknown (rusb2 FRDY) | -| RAM ring-buffer | ~tens of cycles per event | ISR ordering/timing bugs (musb babble) | -| TU_LOG (RTT) | µs per line | logic bugs that survive logging | -| TU_LOG (UART) | ms per line — blocking write | same, when no J-Link on the board | -| GDB halt / breakpoints | stops USB service entirely | post-mortem state autopsy once wedged | - -## TU_LOG capture - -Build with `LOG=2` (`LOG=3` adds per-transfer noise and much more timing skew). -`LOGGER=rtt` routes it over the debug probe (J-Link only) — no UART wiring: - -```bash -# RTT: JLinkGDBServer from CLAUDE.md "GDB Debugging" + -RTTTelnetPort, then: -timeout 20s JLinkRTTClient > /tmp/rtt.log # non-interactive capture -# UART (board's debug serial, if wired): -stty -F /dev/ttyACM 115200 raw && timeout 20s cat /dev/ttyACM | tee /tmp/uart.log -``` - -An RTT-built firmware that has since wedged still holds a log tail in RAM — -but ONLY what fits the drain model: the default SEGGER mode (NO_BLOCK_SKIP) -**drops** writes once the ring fills with no reader, so an undrained target -holds the first KB after boot, not the wedge tail. There is no overwrite mode -in stock SEGGER RTT (only SKIP/TRIM/BLOCK): post-mortem RTT is evidence only -if a live drain was running — otherwise instrument with the RAM ring below. -Use `JLinkGDBServer -RTTTelnetPort 19021` + `JLinkRTTClient` for the drain -(proven; note the server briefly halts the core on connect). `JLinkRTTLogger` -fails to find the control block on some parts (LPC4088) even when it exists -and even given `-RTTAddress`; don't fight it — `nm` the ELF for `_SEGGER_RTT`, -read the aUp[0] descriptor (`mem32`), `savebin` the buffer — debug-AP RAM -reads don't halt the target. - -## GDB — state autopsy and watchpoints - -Connect/load recipes per probe family (J-Link, OpenOCD for ST-Link / -CMSIS-DAP / WCH-Link) are in CLAUDE.md "GDB Debugging". Release builds keep -DWARF (`MinSizeRel`), so `p`/struct access works on HIL firmware. - -**Autopsy of a wedged board: attach and halt ONLY** — skip CLAUDE.md's -`monitor reset halt` + `load` (those are for fresh starts; a reset destroys -the evidence). Symbolize with the ELF that is actually flashed — -`/cmake-build-//.elf` from the run that -wedged; do not rebuild while the wedge is still on the board. The debug-loop -specifics: - -```gdb -p/x _usbd_dev.ep_status # usbd core [epnum][dir] (1=IN): busy/stalled/claimed -p/x # per-port names — read the board's dcd_*.c first -x/32wx # raw EP/FIFO regs; base = the macro the dcd uses -watch xfer_status[2][1].total_len # HW watchpoint (Cortex-M: ~4); dwc2 names shown -break dcd_int_handler # works, but see warning below -``` - -While halted the device answers **nothing**: host control transfers time out -in ~5 s and the OS may reset/re-enumerate — after `continue`, the bus traffic -shows recovery, not the original bug. Prefer one halt for a post-mortem dump -over stepping through live USB traffic. - -## RAM ring-buffer trace - -The zero-print instrument (cracked the musb babble): a small event ring in the -dcd/hcd, dumped over GDB after the failure. Single-writer (ISR) — no locking: - -```c -typedef struct { uint16_t ev; uint16_t a; uint32_t b; } dbg_ev_t; -#define DBG_N 512 // power of two -static volatile dbg_ev_t dbg_ring[DBG_N]; // volatile REQUIRED: -Os dead-store- -static volatile uint32_t dbg_wr; // eliminates a write-only static array -static inline void DBG_EV(uint16_t ev, uint16_t a, uint32_t b) { - uint32_t i = dbg_wr++; - dbg_ring[i & (DBG_N - 1)] = (dbg_ev_t){ ev, a, b }; -} -// call sites: DBG_EV(__LINE__, ep_addr, count); — __LINE__ as event id -``` - -After building, `nm` the ELF for `dbg_ring`/`dbg_wr` — if they're missing the -compiler deleted your instrument and the run will "reproduce" with an empty ring. - -Order is the index; if durations matter add a `uint32_t t = DWT->CYCCNT` field -(enable once: `CoreDebug->DEMCR |= CoreDebug_DEMCR_TRCENA_Msk; DWT->CTRL |= 1;` -RISC-V: read `mcycle`). Let the failure happen, halt, then: - -```gdb -p dbg_wr # total events; oldest slot = dbg_wr & (DBG_N-1) once wrapped -p dbg_ring -dump binary memory /tmp/ring.bin &dbg_ring[0] &dbg_ring[512] -``` - -## PC-sampling (J-Link) — find where the core spins, without halting - -`DWT_PCSR` (0xE000101C) returns the current PC on every read, target running -(Cortex-M3+; optional on M0+, reads 0 if absent; 0xFFFFFFFF = core halted or -WFI-asleep — `mem32 E000EDF0, 1`, DHCSR bit 17 S_HALT, tells which). One -probe serves one client: quit JLinkExe before starting JLinkGDBServer on the -same probe. Nailed the rusb2 FRDY wedge: - -```bash -for i in $(seq 300); do echo 'mem32 E000101C, 1'; done \ - | JLinkExe -device $JLINK_DEVICE -SelectEmuBySN -if swd -speed 4000 -autoconnect 1 -nogui 1 \ - | awk '/E000101C = /{print $3}' | sort | uniq -c | sort -rn | head -arm-none-eabi-addr2line -e -f -a 0x ... # PCs → functions -``` - -OpenOCD variant: repeat `mdw 0xE000101C` over telnet :4444. The histogram's -top entries are the spin site; a flat histogram = core is servicing normally. - -## Dual-side capture — the default for enumeration/transfer bugs - -Start both channels, then trigger the failing test: - -```bash -.claude/skills/usbmon/scripts/usbcap.sh cafe: 30 /tmp/host.pcapng & # host URBs (usbmon skill) -timeout 30s JLinkRTTClient > /tmp/target.rtt & # target (or ring dump after) -wait -``` - -RTT lines and ring events carry no wall-clock: correlate on unambiguous -anchors — bus reset, SET_ADDRESS, the first transfer on the failing EP — then -lay device events between anchors in host-URB order. Logging the SOF/frame -number on the target gives a shared clock when you need finer alignment. -When host and target evidence disagree, or the host sees nothing at all, add -the wire itself: `usb-sniffer` skill (hardware tap, PID-level). - -## Warnings - -- **Halting/resetting via the probe does NOT disconnect the device**: a DWC2 - soft-connect pullup stays up through core halt *and* reset, so the host's - stuck URBs stay stuck and a wedged DUT stays wedged — recover the host side - with the `usb-recover` skill. -- **A bug that vanishes under LOG=2 is a timing bug**, not fixed: switch to - the ring buffer; if it vanishes under GDB too, PC-sampling only. -- **UART TU_LOG blocks in the write path** (worst perturbation, including - inside the ISR); RTT is much cheaper but not free; `LOG=3` multiplies both. -- Flash/GDB only with the board lock held; a `hold` refused with reason - `hil_test.py` means CI is mid-test on that board — wait, don't force. -- **Instrumentation is temporary**: before `release`, reflash pristine - firmware (the next CI run must not inherit a debug build) and revert the - instrumentation diff — or hand it over explicitly with the diagnosis. -- **A register snapshot without a validity anchor lies**: J-Link tool sessions - can reset or briefly halt the DUT as a side effect, and a snapshot of a - freshly-reset chip (e.g. NVIC ISER = 0) reads like a smoking gun. Read DHCSR - (0xE000EDF0: bit 17 S_HALT, bit 25 S_RESET_ST) with every snapshot, and - cross-check against something the device demonstrably still does. -- **A marginal link can fake a deterministic firmware bug** — down to failing - the same test at the same iteration twice. "USB disconnect" in dmesg on a - freshly re-cabled port (high devnum = churn) means the plug, not the code: - first sustained bulk traffic is when a bad contact drops. Before declaring a - regression, re-run the OLD build on the SAME link state — and if a bisect - exonerates every hunk, believe it: re-test the exact failing binary. -- **Release your manual lock before `hil_test.py`** — it self-locks each board - and fails immediately on your own hold (`hil` skill). diff --git a/.claude/skills/usbmon/SKILL.md b/.claude/skills/usbmon/SKILL.md index 85ec33248..490c79e7f 100644 --- a/.claude/skills/usbmon/SKILL.md +++ b/.claude/skills/usbmon/SKILL.md @@ -1,11 +1,11 @@ --- name: usbmon -description: Use when capturing, analyzing, or debugging USB bus traffic for TinyUSB device development on Linux — enumeration failures, STALLed control transfers, missing/short bulk or interrupt transfers, isochronous/audio dropouts, or descriptor problems. Captures host-side URBs with usbmon + tshark into a Wireshark pcapng and decodes them. Use whenever you need to see what the host actually exchanged with a device on real hardware, even if the user just says "sniff USB", "capture the enumeration", or "why won't my device enumerate". +description: Use when capturing, analyzing, or debugging USB bus traffic on a link where a Linux PC is the host (TinyUSB in device role) — enumeration failures, STALLed control transfers, missing/short bulk or interrupt transfers, isochronous/audio dropouts, or descriptor problems. Captures host-side URBs with usbmon + tshark into a Wireshark pcapng and decodes them. Not applicable when TinyUSB is the host — no URBs traverse the PC (use usb-sniffer / target-debug). Use whenever you need to see what the Linux host actually exchanged with a device on real hardware, even if the user just says "sniff USB", "capture the enumeration", or "why won't my device enumerate". --- # usbmon — capture & debug USB traffic -`usbmon` records host-side **URBs** — control / bulk / interrupt / isochronous transfers, descriptors, class requests, STALLs, short packets — i.e. exactly what the host exchanged with a device. Use it to debug a TinyUSB device on real hardware. (It's host/URB-level, not wire-level; for SOF/ACK/electrical use a hardware analyzer.) +`usbmon` records host-side **URBs** — control / bulk / interrupt / isochronous transfers, descriptors, class requests, STALLs, short packets — i.e. exactly what the host exchanged with a device. Use it to debug a TinyUSB device on real hardware. (It's host/URB-level, not wire-level; for SOF/ACK/electrical use a hardware analyzer.) It exists only on the Linux host side of a link: when TinyUSB runs the *host* stack (peer = another TinyUSB board or a Linux gadget, e.g. a Raspberry Pi), neither end has usbmon — capture the wire (`usb-sniffer` skill) or instrument the target (`target-debug` skill). **Setup (assumed in place):** `usbmon` loaded and a udev rule `SUBSYSTEM=="usbmon", GROUP="wireshark", MODE="0640"` with your user in the `wireshark` group — so `tshark` captures with no `sudo`. Freshly added to the group? The running shell doesn't have it yet (group adds need a new login) — wrap captures in `sg wireshark -c 'tshark -i usbmon3 -s 128 -a duration:30 -w /tmp/cap.pcapng'`; reading a finished `.pcapng` (`tshark -r`) needs no group. `-s 128` (snaplen) keeps only URB headers/status, not payloads — use it for long/high-throughput captures. @@ -29,18 +29,18 @@ tshark -r cap.pcapng -Y 'usb.device_address==26' # filter to one device ## Filter (`-Y ''`) -| Goal | Expression | -|---|---| -| One device / endpoint | `usb.device_address==26` / `usb.endpoint_address==0x81` | -| IN (to host) / OUT (from host) | `usb.endpoint_address.direction==1` / `==0` | -| Submit / Complete event | `usb.urb_type=='S'` / `=='C'` (char literal: single quotes) | -| Control / bulk / interrupt / iso | `usb.transfer_type==2` / `3` / `1` / `0` | -| Only transfers carrying data | `usb.data_len>0` | -| GET_DESCRIPTOR / SET_ADDRESS / SET_CONFIGURATION | `usb.setup.bRequest==6` / `5` / `9` | -| SET_INTERFACE / CLEAR_FEATURE (clear-halt) | `usb.setup.bRequest==11` / `1` | -| Descriptor type DEVICE/CONFIG/STRING/HID-report | `usb.bDescriptorType==1` / `2` / `3` / `0x22` | -| Class / vendor requests | `usb.bmRequestType.type!=0` | -| STALLs / errors | `usb.urb_status!=0 && usb.urb_status!=-115` | +| Goal | Expression | +|--------------------------------------------------|-------------------------------------------------------------| +| One device / endpoint | `usb.device_address==26` / `usb.endpoint_address==0x81` | +| IN (to host) / OUT (from host) | `usb.endpoint_address.direction==1` / `==0` | +| Submit / Complete event | `usb.urb_type=='S'` / `=='C'` (char literal: single quotes) | +| Control / bulk / interrupt / iso | `usb.transfer_type==2` / `3` / `1` / `0` | +| Only transfers carrying data | `usb.data_len>0` | +| GET_DESCRIPTOR / SET_ADDRESS / SET_CONFIGURATION | `usb.setup.bRequest==6` / `5` / `9` | +| SET_INTERFACE / CLEAR_FEATURE (clear-halt) | `usb.setup.bRequest==11` / `1` | +| Descriptor type DEVICE/CONFIG/STRING/HID-report | `usb.bDescriptorType==1` / `2` / `3` / `0x22` | +| Class / vendor requests | `usb.bmRequestType.type!=0` | +| STALLs / errors | `usb.urb_status!=0 && usb.urb_status!=-115` | Combine with `&&` — e.g. one endpoint's data: `usb.endpoint_address==0x02 && usb.data_len>0`. diff --git a/.claude/skills/usbtest/SKILL.md b/.claude/skills/usbtest/SKILL.md index 8146d94e4..76a01839c 100644 --- a/.claude/skills/usbtest/SKILL.md +++ b/.claude/skills/usbtest/SKILL.md @@ -1,6 +1,6 @@ --- name: usbtest -description: Use when running, debugging, or porting the Linux usbtest/testusb battery (examples/device/usbtest, cafe:4010) — device "did not bind", SET_CONFIGURATION fails, a case fails with errno 110/32/5/71, toggle-clear/halt/unlink/iso failures, iso packets dropped, or a new MCU/DCD needs the full 30/30 sign-off. +description: Use when running, debugging, or porting the Linux usbtest/testusb battery (examples/device/usbtest, cafe:4010) — device "did not bind", SET_CONFIGURATION fails, a case fails with errno 110/32/5/71, toggle-clear/halt/unlink/iso failures, iso packets dropped, or a new MCU/DCD needs the full 30/30 sign-off. Needs a Linux PC as the link's host driving TinyUSB in device role — it exercises the DCD, not the TinyUSB host stack. --- # usbtest — porting & debugging the Linux kernel USB battery @@ -80,12 +80,12 @@ python3 test/hil/usbtest.py --serial --keep-binding --tests 29 # one case ## Debug ladder (escalate in order) -| errno | Meaning | -|---|---| -| 110 | timeout — endpoint NAKing forever / device wedged | -| 32 | EPIPE — unexpected STALL | -| 5 | EIO — iso packet errors (check `dmesg`: "N errors out of M") | -| 71 | EPROTO — device answered wrong / too slow (after HC retries) | +| errno | Meaning | +|-------|--------------------------------------------------------------| +| 110 | timeout — endpoint NAKing forever / device wedged | +| 32 | EPIPE — unexpected STALL | +| 5 | EIO — iso packet errors (check `dmesg`: "N errors out of M") | +| 71 | EPROTO — device answered wrong / too slow (after HC retries) | 1. `usbtest.py` per-case output + its captured `dmesg` (`TEST n` markers bracket each case). 2. **usbmon** (`usbmon` skill): URB-level ground truth. **It cannot show data toggles or NAKs** — diff --git a/.claude/workflows/hil-validate.js b/.claude/workflows/hil-validate.js index aa0556abc..50559135f 100644 --- a/.claude/workflows/hil-validate.js +++ b/.claude/workflows/hil-validate.js @@ -52,7 +52,7 @@ if (!args.force) { } const wedged = results.filter(r => r.wedged).map(r => r.board) -if (wedged.length) log(`WEDGED boards needing usb-recover: ${wedged.join(', ')}`) +if (wedged.length) log(`WEDGED boards needing usb-kernel-recover: ${wedged.join(', ')}`) // Workers cannot prompt the user — surface still-locked boards for the main // session to ask: force (re-invoke with force: true), wait, or accept. const locked = args.force ? [] : results.filter(r => !r.pass && r.detail.startsWith('board locked')).map(r => r.board) diff --git a/CLAUDE.md b/CLAUDE.md index 2acdc3a63..77dab4565 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -116,7 +116,7 @@ Cutting a release — version bump, regenerated files, the per-release changelog ## References -- MCU reference manuals, datasheets, schematics: before answering register/bitfield/pinout/errata/timing questions from memory or the web, use the `read-doc` skill (`.claude/skills/read-doc/SKILL.md`) to search and read them from `$HOME/Documents/calibre-library` (skill no-ops if the library is absent). +- MCU reference manuals, datasheets, schematics: before answering register/bitfield/pinout/errata/timing questions from memory or the web — or changing a specific dcd/hcd driver — use the `read-doc` skill (`.claude/skills/read-doc/SKILL.md`) to cross-check against docs in `$HOME/Documents/calibre-library`; tell the user if the needed document is missing (skill no-ops if the library is absent). - Supported MCUs/boards: `hw/bsp/` and `docs/reference/boards.rst`. - USB classes: `src/class/{cdc,hid,msc,audio,…}/` — each has `*_device.c` and `*_host.c`. - Key files: `src/tusb.h`, `src/tusb_config.h`, `tools/get_deps.py`, `tools/build.py`, `test/unit-test/project.yml`. diff --git a/docs/superpowers/plans/2026-07-23-esp-target-debug-skill.md b/docs/superpowers/plans/2026-07-23-esp-target-debug-skill.md new file mode 100644 index 000000000..d08902111 --- /dev/null +++ b/docs/superpowers/plans/2026-07-23-esp-target-debug-skill.md @@ -0,0 +1,74 @@ +# esp-target-debug Skill Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Create `.claude/skills/esp-target-debug/SKILL.md` (Espressif built-in USB-Serial-JTAG debug backend) with every recipe verified on the rig's P4, the S3 PHY boundary verified both ways, plus pointer edits in `target-debug` and the `target-debugger` agent. + +**Architecture:** Per spec `docs/superpowers/specs/2026-07-23-esp-target-debug-design.md`. Verification-first: hardware gates 1–6 run before the skill text lands, so only proven content ships unmarked. One lock session per board. + +**Tech Stack:** ESP-IDF at `$HOME/code/esp-idf` (`export.sh` → `openocd-esp32`, `riscv32-esp-elf-gdb`, `xtensa-esp32s3-elf-gdb`, `esptool.py`), rig boards `espressif_p4_function_ev` (uid 6055F9F98715), `espressif_s3_devkitm` (uid 84F703C084E4). + +## Global Constraints + +- Worktree `/home/hathach/code/tinyusb/.claude/worktrees/improve-debug-skill-agent`, branch `claude/improve-debug-skill-agent`. +- Board-lock discipline per `hil` skill; reflash pristine firmware before release; evidence (command + output snippet) in commit message bodies. +- Formatting: aligned table columns, skill-name-only cross-references. +- Unverified content ships tagged `(untested)` or not at all. +- Espressif anything requires `. $HOME/code/esp-idf/export.sh` in that shell first. + +--- + +### Task 1: P4 recon + coexistence gate (spec gates 1) + +- [x] **Step 1: Environment + firmware recon** + +```bash +ls $HOME/code/esp-idf/export.sh && source $HOME/code/esp-idf/export.sh && which openocd riscv32-esp-elf-gdb +ls /home/hathach/code/tinyusb/examples/cmake-build-espressif_p4_function_ev 2>/dev/null || echo "no prebuilt" +lsusb -d 303a:1001 # USB-SJ devices present +``` +If no prebuilt firmware: build `device/cdc_msc_freertos` for the P4 (`idf.py -DBOARD=espressif_p4_function_ev build` in that example, per CLAUDE.md), else use the prebuilt binary. Identify the ELF path for gdb symbolization. + +- [x] **Step 2: Lock P4, ensure known firmware, confirm DUT traffic** + +```bash +python3 test/hil/board_lock.py hold espressif_p4_function_ev --reason "esp-target-debug verify: coexistence" +# flash known build (esptool/idf.py flash -p ), settle, then confirm enumeration: +lsusb | grep -i cafe # TinyUSB VID on the DUT port +# generate traffic: echo > /dev/ttyACM of the cdc, or timeout 5s cat +``` + +- [x] **Step 3: Attach openocd over USB-SJ while the device runs** + +```bash +openocd -f board/esp32p4-builtin.cfg -c 'adapter serial 60:55:F9:F9:87:15' & # gdb :3333 — USB-SJ iSerial = MAC with colons +riscv32-esp-elf-gdb -batch -ex 'target extended-remote :3333' -ex 'monitor halt' \ + -ex bt -ex 'monitor resume' +``` +Expected: backtrace with symbols; after resume the CDC device still answers (re-run the traffic check). Record: does the DUT drop off the bus during halt (host URB timeouts — expected per target-debug) and does it recover on resume without re-enumeration? + +- [x] **Step 4: Release-or-continue checkpoint** — keep the lock for Task 2 (same session). No commit yet; evidence to `/tmp/esp_evidence.txt`. + +### Task 2: P4 budget, watchpoint, threads, console (spec gates 2–4) + +- [x] **Step 1: Breakpoint/watchpoint budget** — RISC-V trigger count: in gdb `monitor riscv info` or set watchpoints until rejection; verify a hardware watchpoint on a TinyUSB variable (e.g. `watch -l` on a usbd counter) reports and hits. +- [x] **Step 2: FreeRTOS threads** — `info threads` after halt; expect ESP-IDF tasks incl. the USB task; note whether it works at attach or needs run→stop (mirror the ARM finding). +- [x] **Step 3: Console during traffic** — OUTCOME: stock builds route the console to UART0 (the CP2102 flasher tty — boot log captured there); the USB-SJ CDC carries no log without sdkconfig `ESP_CONSOLE_USB_SERIAL_JTAG`, which stays (untested) in the skill. +- [x] **Step 4: Reflash pristine, release P4 lock.** Evidence appended to `/tmp/esp_evidence.txt`. + +### Task 3: P4 apptrace spike — GATED (spec gate 5) + +Budget 30 min. `openocd -c 'esp apptrace start ...'` against a firmware built with apptrace enabled? Stock HIL firmware has no apptrace init — if a code change would be required, that's the gate answer: land apptrace as `(untested — needs CONFIG_APPTRACE + firmware init)` with the recipe sketch. Only a working capture lands unmarked. + +### Task 4: S3 boundary (spec gate 6) + +- [x] **Step 1: Lock S3, flash `board_test`** (no TinyUSB → PHY free). Attach `openocd -f board/esp32s3-builtin.cfg -c 'adapter serial 84F703C084E4'` + `xtensa-esp32s3-elf-gdb`: halt + bt works. +- [x] **Step 2: Flash a USB device example** — record the exact failure: does 303a:1001 vanish from lsusb (PHY switched), does openocd fail to attach or die mid-session? Capture verbatim error. +- [x] **Step 3: Reflash pristine (a USB example — that is the CI-expected state), release.** + +### Task 5: Write the skill + integration edits + commit + +- [x] **Step 1: Write `.claude/skills/esp-target-debug/SKILL.md`** per spec section order (role/defer, PHY map with verified boundary symptoms, toolchain+attach with the real commands from Tasks 1–4, technique mapping table with verified annotations, rig deltas, external-JTAG TODO). Aligned tables. +- [x] **Step 2: `target-debug` pointer** (2 lines, after probe-mapping bullets) + `target-debugger` agent table row. +- [x] **Step 3: pre-commit, single commit** with evidence summary from `/tmp/esp_evidence.txt`. +- [x] **Step 4: Retrieval sanity** — one fresh-subagent scenario: "debug a TinyUSB hang on the rig's P4" routes to esp-target-debug (not JLink recipes); "same on S3 while cdc_msc runs" routes to the PHY boundary + external-JTAG TODO. diff --git a/docs/superpowers/plans/2026-07-23-target-debug-skill-enhancement.md b/docs/superpowers/plans/2026-07-23-target-debug-skill-enhancement.md new file mode 100644 index 000000000..36a3144c2 --- /dev/null +++ b/docs/superpowers/plans/2026-07-23-target-debug-skill-enhancement.md @@ -0,0 +1,570 @@ +# target-debug Skill & target-debugger Agent Enhancement Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Extend `.claude/skills/target-debug/SKILL.md` (and its agent) with the full debugger facility arsenal from the J-Link, OpenOCD, and GDB manuals — breakpoint/watchpoint depth, OpenOCD RTT, vector catch + fault autopsy, SWO/ITM trace, flash verification — each recipe hardware-verified on the ci rig before it lands unmarked. + +**Architecture:** The skill's organizing spine is its intrusiveness table ("pick the least intrusive technique that can answer the question"); every new facility slots into that model with an honest cost row. Recipes keep the existing dense, copy-paste style. The skill's value is that its recipes are *proven on this rig* — so each task pairs drafting with a bounded hardware verification, and anything unverifiable lands tagged `(untested)` or is dropped. + +**Tech Stack:** arm-none-eabi-gdb 15.2, OpenOCD 0.12.0+dev, SEGGER J-Link V7.94b (`JLinkExe`, `JLinkGDBServer`, `JLinkSWOViewerCLExe`), ci rig boards from `test/hil/tinyusb.json` (10 jlink / 6 openocd / 1 stlink probes). + +**Reference manual:** "Debugging with GDB", **Tenth Edition** (for GDB 18.0.50) — prefer the calibre-library copy via the `read-doc` skill, but **verify the edition on the title page first**: the library also holds an outdated Ninth Edition (2002, GDB 5.1.1, txt) that predates `dprintf`/`watch -l` — do not use it. Fallback fetch: `curl -sL -o /tmp/gdb.pdf https://sourceware.org/gdb/current/onlinedocs/gdb.pdf` (HTML pages block fetchers; the PDF does not). Sections used by this plan: §5.1.2 Setting Watchpoints, §5.1.6 Break Conditions, §5.1.7 Breakpoint Command Lists, §5.1.8 Dynamic Printf (PDF page = book page + 18). NOTE: the manual documents GDB 18; the rig runs 15.2 — the installed `arm-none-eabi-gdb`'s `help ` is authoritative for feature availability. + +## Global Constraints + +- Worktree: `/home/hathach/code/tinyusb/.claude/worktrees/improve-debug-skill-agent`, branch `claude/improve-debug-skill-agent`. All paths below are relative to it. +- J-Link User Guide link must be exactly `https://kb.segger.com/UM08001_J-Link_/_J-Trace_User_Guide` (user-specified, verified live 2026-07-23). +- **Hardware-verify before landing**: a recipe is committed unmarked only with captured evidence from a rig board; otherwise tag it `(untested)` inline or drop it. Record evidence (command + output snippet) in the task's commit message body. +- Rig discipline (from `hil` + `target-debug` skills): `python3 test/hil/board_lock.py hold --reason "skill-enhance verify: "` before touching hardware, `release` after; reflash pristine firmware before release; NEVER stop the actions-runner; one J-Link client per probe at a time; we are ON host `ci` (config `test/hil/tinyusb.json`). +- Hardware tasks are strictly serial (one board session at a time). Bash timeouts ≥ 10 min for flash+debug cycles. +- Style: match the skill's existing voice — dense, recipe-first, caveats inline. Skill word budget after all tasks: ≤ 2 700 words (`wc -w`, currently 1 763). +- Run `pre-commit run --files ` before every commit. No Co-Authored-By trailers. +- Board selection is runtime data (boards come/go, locks): resolve with the exact python snippet in Task 2 Step 2 and reuse `$JB` (jlink board) / `$OB` (openocd board) thereafter. + +--- + +### Task 1: Manuals reference block + +**Files:** +- Modify: `.claude/skills/target-debug/SKILL.md` (insert new `## Manuals` section immediately before `## Warnings`) + +**Interfaces:** +- Produces: `## Manuals` section that later tasks' text may reference as "see Manuals". + +- [x] **Step 1: Insert the Manuals section** + +In `.claude/skills/target-debug/SKILL.md`, find the line `## Warnings` and insert immediately before it: + +```markdown +## Manuals + +- J-Link / J-Trace User Guide (UM08001): — flash breakpoints, RTT, SWO, monitor mode, Commander commands. +- OpenOCD User's Guide: — `rtt`, `bp`/`wp`, `cortex_m vector_catch` / `maskisr`, `itm`/`tpiu`. +- "Debugging with GDB" (the official manual; §5.1 covers break/watch/dprintf): + calibre library first (`read-doc` skill) — use the **Tenth Edition (GDB 18)** + copy, not the 2002 Ninth-Edition txt also present; fallback + `curl -sL -o /tmp/gdb.pdf https://sourceware.org/gdb/current/onlinedocs/gdb.pdf` + (the HTML mirror blocks fetchers; the PDF works). The installed + `arm-none-eabi-gdb`'s `help ` is authoritative for what this rig runs. + +``` + +- [x] **Step 2: Verify formatting and word count** + +Run: `cd /home/hathach/code/tinyusb/.claude/worktrees/improve-debug-skill-agent && grep -A5 '^## Manuals' .claude/skills/target-debug/SKILL.md && wc -w .claude/skills/target-debug/SKILL.md` +Expected: section present before `## Warnings`; word count ≤ 1 830. + +- [x] **Step 3: Commit** + +```bash +cd /home/hathach/code/tinyusb/.claude/worktrees/improve-debug-skill-agent +pre-commit run --files .claude/skills/target-debug/SKILL.md +git add .claude/skills/target-debug/SKILL.md +git commit -m "docs(target-debug): link J-Link UM08001, OpenOCD and GDB manuals" +``` + +--- + +### Task 2: Breakpoint & watchpoint arsenal (GDB + OpenOCD) + +**Files:** +- Modify: `.claude/skills/target-debug/SKILL.md` — extend the `## GDB — state autopsy and watchpoints` section +- Read-only reference: `test/hil/tinyusb.json` (board resolution) + +**Interfaces:** +- Consumes: nothing from other tasks. +- Produces: board env vars `$JB`, `$OB` resolution snippet (reused by Tasks 3-6); the "halt-per-hit cost model" wording that Task 7's table row cites. + +- [x] **Step 1: Draft the section extension** + +In `.claude/skills/target-debug/SKILL.md`, the GDB section currently ends with the paragraph beginning `While halted the device answers **nothing**`. Insert immediately BEFORE that paragraph: + +```markdown +**Hardware budget — read it off the chip, not from memory** (counts differ +per core: M0+ typically 4 bp/2 wp, M3/M4 6/4, M7 8/4): + +```gdb +p ((*(unsigned*)0xE0002000)>>4) & 0xF # FPB NUM_CODE = hw breakpoints (M7 adds bits[14:12]) +p (*(unsigned*)0xE0001000)>>28 # DWT_CTRL NUMCOMP = watchpoint comparators +``` + +- `hbreak`/`thbreak` force a hardware breakpoint (code in flash can't take a + software break unless the probe does flash breakpoints — J-Link does, + OpenOCD needs `bp 2 hw`); `tbreak` = one-shot. +- `watch -l ` watches the *address* the expression evaluates to once — + cheap and what you almost always want; `rwatch`/`awatch` trap reads/any + access (hardware-only — they error rather than fall back). OpenOCD (telnet + :4444) adds a data-VALUE match GDB cannot express: `wp 4 w + [mask]` — fires only when the written value matches (e.g. catch who writes + 0 into a busy flag, ignoring writes of 1). +- **Demand the word "Hardware" in the confirmation.** `watch` silently falls + back to a SOFTWARE watchpoint when no DWT comparator fits (expression too + wide/complex, budget exhausted): GDB then single-steps the whole program — + hundreds of times slower, certain USB death. `Watchpoint 2:` without + "Hardware" = delete it; `set can-use-hw-watchpoints 1` is the default but + narrowing the expression (`watch -l`, cast to a 4-byte int) is the real fix. +- Conditional breaks/watches (`break dcd_edpt_xfer if ep_addr==0x81`) are + evaluated by GDB on the HOST with our stubs — neither JLinkGDBServer nor + OpenOCD supports target-side agent expressions on Cortex-M — so every hit + is a halt+resume (~ms) whether the condition matches or not: fine + post-wedge or on cold paths, wrong under live USB traffic. +- `commands ... end` auto-runs GDB commands at each hit (start with + `silent`, end with `continue` for hands-free evidence collection) — same + halt-per-hit cost. +- `dprintf ,"fmt",args` = printf without recompiling. Stay on the + default `dprintf-style gdb` (host prints): the `call` style runs the + target's own printf mid-halt and `agent` needs stub support — neither is + viable on these probes. Same cost model as conditional breaks; for + ISR-rate events use the RAM ring buffer instead. +- Stepping while the USB ISR fires between every step is chaos: OpenOCD + `cortex_m maskisr steponly` masks interrupts during single-steps only. + The bus keeps running either way — the host may still reset a device that + stops responding mid-step. +- While halted you can poke state to test a hypothesis (`set var + _usbd_dev.ep_status[2][1].busy = 0`) — but that invalidates the snapshot + as post-mortem evidence; dump first, poke after. +``` + +- [x] **Step 2: Resolve verification boards (runtime data)** + +```bash +cd /home/hathach/code/tinyusb +python3 - <<'EOF' +import json +cfg = json.load(open('test/hil/tinyusb.json')) +jl = [b['name'] for b in cfg['boards'] if b['flasher']['name']=='jlink'] +oo = [b['name'] for b in cfg['boards'] if b['flasher']['name']=='openocd'] +print('JLINK candidates:', jl) +print('OPENOCD candidates:', oo) +EOF +``` +Pick the first candidate of each that `python3 test/hil/board_lock.py status` shows unlocked; export as `JB=` `OB=`. Look up `flasher.uid` for each in `test/hil/tinyusb.json` (`JB_UID`, `OB_UID`) and `JLINK_DEVICE`/`OPENOCD_OPTION` from `hw/bsp/*/boards/$JB/board.cmake` (family via `ls -d hw/bsp/*/boards/$JB`). + +- [x] **Step 3: Hardware-verify the budget reads on both probe families** + +```bash +python3 test/hil/board_lock.py hold $JB --reason "skill-enhance verify: bp/wp budget" +printf 'mem32 E0002000, 1\nmem32 E0001000, 1\nqc\n' | \ + JLinkExe -device $JLINK_DEVICE -SelectEmuBySN $JB_UID -if swd -speed 4000 -autoconnect 1 -nogui 1 +python3 test/hil/board_lock.py release $JB +``` +Expected: two register values; decode NUM_CODE and NUMCOMP by hand and check they are plausible (2-8 range). Repeat for `$OB` via `openocd $OPENOCD_OPTION -c init -c 'mdw 0xE0002000' -c 'mdw 0xE0001000' -c shutdown` under its own lock. +If a register reads 0 on one board, note which core and adjust the skill text's example counts if contradicted. + +- [x] **Step 4: Hardware-verify dprintf + commands round-trip on $JB** + +With the board lock held and an already-flashed example (any; do not reflash), a JLinkGDBServer on :2331 (per CLAUDE.md GDB Debugging), run bounded — `commands` blocks cannot be passed via `-ex`, so use a command file: + +```bash +cat > /tmp/bpcmd.gdb <<'EOF' +target remote :2331 +set var $count=0 +watch -l *(unsigned*)&_usbd_dev +delete +dprintf tud_task_ext,"tick\n" +break tud_task_ext +commands 3 +silent +set var $count=$count+1 +continue +end +continue& +shell sleep 3 +interrupt +print $count +EOF +timeout 120 arm-none-eabi-gdb -batch -x /tmp/bpcmd.gdb \ + $(find examples/cmake-build-$JB -name 'cdc_msc.elf' | head -1) +``` +Expected: the `watch` line answers `Hardware watchpoint 1:` (the word +"Hardware" present — this is the skill's software-fallback check, then +deleted), "tick" lines printed, and `$count > 0`. (`tud_task_ext` is the real +symbol — `tud_task` is an inline wrapper; the breakpoint is number 3 after +the watchpoint and dprintf.) Kill the GDB server, reflash pristine +(`ninja`-flash target or `hil_test.py` flash path), release the lock. + +- [x] **Step 5: Apply the Step-1 text, run pre-commit, commit** + +```bash +cd /home/hathach/code/tinyusb/.claude/worktrees/improve-debug-skill-agent +pre-commit run --files .claude/skills/target-debug/SKILL.md +git add .claude/skills/target-debug/SKILL.md +git commit -m "docs(target-debug): breakpoint/watchpoint arsenal with halt-per-hit cost model + +Verified on (J-Link) + (OpenOCD): FPB/DWT budget reads, dprintf, +breakpoint command lists. " +``` + +--- + +### Task 3: OpenOCD RTT — RTT is not J-Link-only + +**Files:** +- Modify: `.claude/skills/target-debug/SKILL.md` — `## TU_LOG capture` section + +**Interfaces:** +- Consumes: `$OB`, `$OB_UID`, `$OPENOCD_OPTION` from Task 2 Step 2. +- Produces: the corrected claim "RTT works on any OpenOCD-driven probe" that Task 7's agent text repeats. + +- [x] **Step 1: Replace the J-Link-only claim** + +In the `## TU_LOG capture` section, replace: + +```markdown +Build with `LOG=2` (`LOG=3` adds per-transfer noise and much more timing skew). +`LOGGER=rtt` routes it over the debug probe (J-Link only) — no UART wiring: +``` + +with: + +```markdown +Build with `LOG=2` (`LOG=3` adds per-transfer noise and much more timing skew). +`LOGGER=rtt` routes it over the debug probe — no UART wiring. SEGGER's host +tools need a J-Link, but OpenOCD serves the same RTT buffer on ST-Link / +CMSIS-DAP / WCH-Link boards: +``` + +- [x] **Step 2: Add the OpenOCD RTT recipe** + +Immediately after the existing J-Link/UART capture code block (ends with `... | tee /tmp/uart.log`), add: + +```markdown +```bash +# OpenOCD RTT (any probe OpenOCD drives) — in telnet :4444 (or -c equivalents): +rtt setup 0x20000000 0x8000 "SEGGER RTT" # search range = RAM ORIGIN + LENGTH (from the .ld / map file) +rtt start # after firmware booted; rerun after each reflash +rtt server start 19021 0 +# then: timeout 20s nc localhost 19021 > /tmp/rtt.log +``` + +OpenOCD polls the buffer (default 10 ms): bursty logs can drop lines a J-Link +would keep — prefer J-Link where both exist; the drain-model warning below +applies unchanged. +``` + +- [x] **Step 3: Hardware-verify on $OB** + +```bash +python3 test/hil/board_lock.py hold $OB --reason "skill-enhance verify: openocd rtt" +cd examples/device/cdc_msc && cmake -B build-rtt -DBOARD=$OB -G Ninja -DCMAKE_BUILD_TYPE=MinSizeRel -DLOG=2 -DLOGGER=rtt && cmake --build build-rtt +# flash it (ninja -C build-rtt cdc_msc-openocd), then: +openocd $OPENOCD_OPTION & # gdb :3333, telnet :4444 +{ echo 'rtt setup 0x20000000 0x8000 "SEGGER RTT"'; echo 'rtt start'; echo 'rtt server start 19021 0'; sleep 1; } | nc -q1 localhost 4444 +timeout 10s nc localhost 19021 > /tmp/ob_rtt.log; head /tmp/ob_rtt.log +``` +Expected: TinyUSB boot banner / log lines in `/tmp/ob_rtt.log`. Adjust the search range from the board's linker script if the control block isn't found ("rtt: No control block found") and mirror any correction into the Step-2 text. Kill openocd, reflash pristine cdc_msc (no LOG), release lock, delete `build-rtt`. + +- [x] **Step 4: Commit** + +```bash +pre-commit run --files .claude/skills/target-debug/SKILL.md +git add .claude/skills/target-debug/SKILL.md +git commit -m "docs(target-debug): RTT via OpenOCD on non-J-Link probes + +Verified on : rtt setup/start/server + nc capture of boot log. +" +``` + +--- + +### Task 4: Vector catch + fault autopsy + +**Files:** +- Modify: `.claude/skills/target-debug/SKILL.md` — new section after `## GDB — state autopsy and watchpoints` + +**Interfaces:** +- Consumes: `$JB` from Task 2. (Corrected during execution: $OB/rp2040 is ARMv6-M — no CFSR/BFAR and only VC_HARDERR, so the full autopsy verify needs the ARMv7-M $JB; the payload is a bad LOAD because stores fault imprecisely with BFAR invalid.) +- Produces: section title `## Vector catch + fault autopsy` cited by Task 7's table row. + +- [x] **Step 1: Insert the new section** + +After the GDB section (i.e. before `## RAM ring-buffer trace`), insert: + +```markdown +## Vector catch + fault autopsy — catch the crash, not the wedge + +A "wedge" that is really a fault (HardFault loop, lockup) autopsies best AT +the faulting instruction, not minutes later. Arm before reproducing: + +```gdb +# tool-agnostic (any probe, incl. J-Link): DEMCR trap bits — halt on fault +set *(unsigned*)0xE000EDFC |= (1<<10)|(1<<9)|(1<<8)|(1<<7)|(1<<6)|(1<<5)|(1<<4) +# = VC_HARDERR|INTERR|BUSERR|STATERR|CHKERR|NOCPERR|MMERR; bit0 VC_CORERESET halts at reset +``` + +OpenOCD native form: `cortex_m vector_catch hard_err bus_err state_err chk_err mm_err`. +When it fires the core halts at the fault; decode: + +```gdb +p/x *(unsigned*)0xE000ED28 # CFSR — low byte MemManage, byte1 BusFault, top half UsageFault +p/x *(unsigned*)0xE000ED2C # HFSR — bit30 FORCED = an escalated lower-priority fault +p/x *(unsigned*)0xE000ED38 # BFAR — faulting address (valid if CFSR bit15 BFARVALID) +x/8wx $msp # stacked frame: r0 r1 r2 r3 r12 lr pc xpsr — pc = culprit +``` + +`arm-none-eabi-addr2line -e ` names the line. Caveats: a +vector-catch halt is still a halt (host-side URB timeouts apply); the bits +persist until power-cycle — clear them (`... &= ~0x7F1`) before handing the +board back; RISC-V ports have no DEMCR — use a breakpoint on the trap handler. +``` + +- [x] **Step 2: Hardware-verify with a deliberate fault on $JB (ARMv7-M)** + +Create the fault build (NOT committed): + +```bash +python3 test/hil/board_lock.py hold $JB --reason "skill-enhance verify: vector catch" +cd examples/device/cdc_msc # executed on $JB (stm32f407disco, ARMv7-M) via JLinkExe — see commit evidence +# temporary patch — revert after: fault 5 s after boot +python3 - <<'EOF' +import pathlib +p = pathlib.Path('src/main.c'); s = p.read_text() +import re +s = re.sub(r'\\nint main\\(void\\)', + '\\nstatic void _fault_after_5s(void){ static uint32_t t0=0; if(!t0) t0=tusb_time_millis_api();' + ' if(tusb_time_millis_api()-t0>5000) (void)*(volatile uint32_t*)0xCF000000u; }\\n\\nint main(void)', s, count=1) # board_millis is gone; helper must sit after the includes +s = s.replace('led_blinking_task();', 'led_blinking_task(); _fault_after_5s();', 1) +p.write_text(s) +EOF +grep -n '_fault_after_5s' src/main.c # expect 3 hits: definition + call + (none in decl block) +cmake -B build-fault -DBOARD=$JB -G Ninja -DCMAKE_BUILD_TYPE=MinSizeRel && cmake --build build-fault +``` +(If `app_led_task`/`board_millis` anchors differ in the current `main.c`, place the same 3-line helper on whatever per-loop task function exists — the fault line `*(volatile uint32_t*)0xCF000000u = 0;` is the payload.) +Flash `build-fault`, then: + +```bash +# executed variant: DEMCR armed + autopsy via JLinkExe command file on $JB (see commit c1d2d305f evidence); OpenOCD-native form: +openocd $OPENOCD_OPTION -c init -c 'cortex_m vector_catch hard_err bus_err' & +timeout 60 arm-none-eabi-gdb -batch -ex 'target remote :3333' -ex 'monitor reset run' \ + -ex 'shell sleep 8' -ex 'interrupt' \ + -ex 'p/x *(unsigned*)0xE000ED28' -ex 'p/x *(unsigned*)0xE000ED38' -ex 'x/8wx $msp' \ + build-fault/cdc_msc.elf +``` +Expected: halted in the fault path, CFSR BusFault bits set, **BFAR = 0xCF000000**, stacked pc addr2lines to `_fault_after_5s`. If the write is silently ignored on this core (some buses RAZ/WI), switch payload to a NULL-function call `((void(*)(void))0x1)();` and note UsageFault/INVSTATE instead. + +- [x] **Step 3: Clean up hardware state** + +`git checkout -- src/main.c`, delete `build-fault/`, clear DEMCR bits (`set *(unsigned*)0xE000EDFC &= ~0x7F1` via a final gdb attach or power-cycle note), reflash pristine cdc_msc, `board_lock.py release $JB`. + +- [x] **Step 4: Commit** + +```bash +pre-commit run --files .claude/skills/target-debug/SKILL.md +git add .claude/skills/target-debug/SKILL.md +git commit -m "docs(target-debug): vector catch + Cortex-M fault autopsy recipe + +Verified on : deliberate bad-address write halted via vector_catch, +CFSR= BFAR=0xCF000000, stacked pc resolved by addr2line." +``` + +--- + +### Task 5: SWO/ITM experiment — exception trace & hardware PC sampling + +This is an EXPERIMENT task with an explicit gate: the section lands **unmarked only if packets are actually captured** on a rig board; otherwise it lands tagged `(untested — SWO wiring unconfirmed on this rig)`. Budget: 30 min of hardware time, then decide. + +**Files:** +- Modify: `.claude/skills/target-debug/SKILL.md` — new subsection inside the PC-sampling section (after the OpenOCD variant paragraph) + +**Interfaces:** +- Consumes: `$JB`, `$JB_UID`, `$JLINK_DEVICE` from Task 2. +- Produces: verified-or-tagged status consumed by Task 7's table row for SWO. + +- [x] **Step 1: Probe for SWO output (gate experiment)** + +```bash +python3 test/hil/board_lock.py hold $JB --reason "skill-enhance verify: SWO" +# arm DWT sources while the fw runs (background mem write, no halt): +printf 'w4 E0001000, 0x00011401\nqc\n' | JLinkExe -device $JLINK_DEVICE -SelectEmuBySN $JB_UID -if swd -speed 4000 -autoconnect 1 -nogui 1 +# EXCTRCENA(16)|PCSAMPLENA(12)|SYNCTAP(10)|CYCCNTENA(0); tune POSTPRESET[4:1] if PC samples flood — then hand the probe to the viewer: +timeout 20s JLinkSWOViewerCLExe -device $JLINK_DEVICE -usb $JB_UID -swofreq 4000000 -itmmask 0xFFFFFFFF | head -40 +``` +Gate: ANY decoded output (stimulus, PC samples, exception packets) = SWO wired on `$JB` → land unmarked with the observed invocation. No output → try one more J-Link board, then land tagged. Either way `release $JB` after reflashing nothing (this experiment flashes nothing). + +- [x] **Step 2: Insert the section (wording per gate outcome)** + +Append to the `## PC-sampling` section: + +```markdown +### SWO/ITM — hardware-timed trace on one pin (J-Link) + +If the board routes SWO (TRACESWO), DWT emits packets with ZERO code change: +**exception trace** (`DWT_CTRL` bit16 EXCTRCENA) — every IRQ enter/exit, +timestamped, the ISR-ordering evidence the ring buffer needs code for — and +**hardware PC sampling** (bit12 PCSAMPLENA), better histograms than DWT_PCSR +polling. Arm the bits, then give the probe to the viewer (one client rule): + +```bash +printf 'w4 E0001000, 0x00011401\nqc\n' | JLinkExe -device $JLINK_DEVICE -SelectEmuBySN ... +timeout 20s JLinkSWOViewerCLExe -device $JLINK_DEVICE -usb -swofreq 4000000 -itmmask 0xFFFFFFFF +``` + +SWO needs the pin physically wired to the probe — many rig boards route only +SWDIO/SWCLK. If the viewer shows nothing, that is the wiring, not the recipe. +``` + +If the gate FAILED on both boards, append ` (untested — SWO wiring unconfirmed on this rig)` to the subsection heading and keep the text. + +- [x] **Step 3: Commit** + +```bash +pre-commit run --files .claude/skills/target-debug/SKILL.md +git add .claude/skills/target-debug/SKILL.md +git commit -m "docs(target-debug): SWO exception-trace / hw PC-sampling recipe + +Gate result on : ." +``` + +--- + +### Task 6: Flash verification, FreeRTOS thread awareness, semihosting & monitor-mode notes + +**Files:** +- Modify: `.claude/skills/target-debug/SKILL.md` — `## Warnings` section + GDB section tail + +**Interfaces:** +- Consumes: `$JB`, `$JB_UID`, `$JLINK_DEVICE` from Task 2. +- Produces: warning-list entries cited in Task 7's retrieval test scenarios. + +- [x] **Step 1: Add flash-content verification to Warnings** + +In `## Warnings`, after the "A marginal link can fake a deterministic firmware bug" bullet, add: + +```markdown +- **"Flash OK" can lie** (silent no-op: old firmware keeps running after a + green flash). When behavior contradicts the code you think is flashed, + verify flash against the build: + `arm-none-eabi-objcopy -O binary fw.elf /tmp/fw.bin`, then J-Link + `verifybin /tmp/fw.bin,` (Commander) or OpenOCD + `verify_image /tmp/fw.bin ` — a mismatch means reflash with + verification before debugging another minute. +``` + +- [x] **Step 2: Add FreeRTOS + semihosting + monitor-mode notes to the GDB section** + +Append to the end of the `## GDB — state autopsy and watchpoints` section (after the Task-2 additions): + +```markdown +FreeRTOS examples (`*_freertos`): add `-rtos GDBServer/RTOSPlugin_FreeRTOS` +to JLinkGDBServer (OpenOCD: `-rtos FreeRTOS` on the target) and `info +threads` / `thread ` shows every task's stack — a USB task blocked on a +queue vs. spinning is one `bt` away. Semihosting is never the answer here: +each call traps and halts the core — RTT does the same job without stopping. +**Monitor-mode debugging** (J-Link, M3+) can keep the USB ISR serviced while +you sit at a breakpoint — needs SEGGER's `JLINK_MONITOR.c`/ISR files compiled +in + `SetMonModeDebug=1`; not set up in this repo, reach for it when a bug +truly needs live breakpoints without killing the bus: + (untested). +``` + +- [x] **Step 3: Hardware-verify verifybin + FreeRTOS awareness on $JB** + +```bash +python3 test/hil/board_lock.py hold $JB --reason "skill-enhance verify: verifybin+rtos" +# (a) verifybin positive path against whatever is flashed — first reflash a known build: +# flash examples/cmake-build-$JB/device/cdc_msc, then: +arm-none-eabi-objcopy -O binary examples/cmake-build-$JB/device/cdc_msc/cdc_msc.elf /tmp/fw.bin +printf 'verifybin /tmp/fw.bin,\nqc\n' | \ + JLinkExe -device $JLINK_DEVICE -SelectEmuBySN $JB_UID -if swd -speed 4000 -autoconnect 1 -nogui 1 +# (b) rtos plugin: flash cdc_msc_freertos for $JB (build if missing), start +JLinkGDBServer -device $JLINK_DEVICE -select usb=$JB_UID -if swd -speed 4000 -port 2331 -nogui -rtos GDBServer/RTOSPlugin_FreeRTOS & +timeout 60 arm-none-eabi-gdb -batch -ex 'target remote :2331' -ex 'monitor halt' -ex 'info threads' \ + +python3 test/hil/board_lock.py release $JB # after pristine reflash +``` +Expected: (a) `Verify successful.` (b) `info threads` lists FreeRTOS tasks (`usbd`, `IDLE`, ...). If the plugin errors ("Could not load RTOS plugin"), drop the JLinkGDBServer variant from the Step-2 text and keep only the OpenOCD `-rtos FreeRTOS` form tagged `(untested)`. + +- [x] **Step 4: Commit** + +```bash +pre-commit run --files .claude/skills/target-debug/SKILL.md +git add .claude/skills/target-debug/SKILL.md +git commit -m "docs(target-debug): flash verifybin, FreeRTOS thread awareness, monitor-mode pointer + +Verified on : verifybin 'Verify successful.'; info threads listed tasks." +``` + +--- + +### Task 7: Intrusiveness table integration, agent update, retrieval test + +**Files:** +- Modify: `.claude/skills/target-debug/SKILL.md` — the technique/intrusiveness table +- Modify: `.claude/agents/target-debugger.md` — primary-playbook bullet + +**Interfaces:** +- Consumes: verified/untested status of every technique from Tasks 2-6. + +- [x] **Step 1: Extend the intrusiveness table** + +The table under `## Pick the least intrusive technique that can answer the question` currently has 5 rows (PC-sampling → GDB halt). Replace it with (keep the header row and any wording the earlier tasks did not contradict): + +```markdown +| Technique | Intrusiveness | Reach for it when | +|---|---|---| +| PC-sampling | none — no halt, no code change | core wedged/spinning somewhere unknown (rusb2 FRDY) | +| SWO exception trace / hw PC-sample | none — needs SWO pin wired | ISR ordering/timing with zero code change | +| Vector catch | none until a fault fires | crash-shaped wedges — autopsy AT the faulting pc | +| RAM ring-buffer | ~tens of cycles per event | ISR ordering/timing bugs (musb babble) | +| TU_LOG (RTT) | µs per line | logic bugs that survive logging (J-Link or OpenOCD rtt) | +| TU_LOG (UART) | ms per line — blocking write | same, when no debug-probe RTT path | +| dprintf / conditional breakpoint | halt+resume per hit (~ms) | low-rate probes post-wedge; never ISR-rate events | +| GDB halt / breakpoints | stops USB service entirely | post-mortem state autopsy once wedged | +``` + +If Task 5's gate failed, keep the SWO row but append ` (untested)` in its "Reach for it" cell. + +- [x] **Step 2: Update the agent's playbook bullet** + +In `.claude/agents/target-debugger.md`, replace: + +```markdown +- `.claude/skills/target-debug/SKILL.md` — your primary playbook: technique + choice by intrusiveness, channel choice by link topology, capture recipes, + GDB autopsy, all rig warnings. +``` + +with: + +```markdown +- `.claude/skills/target-debug/SKILL.md` — your primary playbook: technique + choice by intrusiveness, channel choice by link topology, capture recipes, + breakpoint/watchpoint budget and cost model, vector catch + fault autopsy, + GDB autopsy, all rig warnings. +``` + +- [x] **Step 3: Word-count and stale-reference check** + +Run: `wc -w .claude/skills/target-debug/SKILL.md` — expected ≤ 2 700. If over, trim prose (not recipes) until under. +Run: `grep -n 'J-Link only' .claude/skills/target-debug/SKILL.md` — expected: no output (Task 3 removed the claim). + +- [x] **Step 4: Retrieval test (skill-TDD GREEN gate)** + +Dispatch a fresh read-only subagent (Explore) that reads ONLY the updated `.claude/skills/target-debug/SKILL.md` and answers: + +1. "A CH32 board's firmware wedges; you suspect a HardFault loop. Least-intrusive next step?" — expected: vector catch (with the RISC-V caveat noted: CH32 is RISC-V → breakpoint on trap handler). +2. "You need RTT logs on an ST-Link-only board." — expected: OpenOCD `rtt setup/start/server`, NOT "impossible/J-Link only". +3. "Who is writing 0 into a busy flag, under live traffic?" — expected: OpenOCD value-match watchpoint `wp 4 w 0`, NOT a GDB conditional watch (halt-per-hit cost). +4. "Flash reported OK but behavior matches last week's build." — expected: verifybin/verify_image. +5. "You set `watch xfer_status[2][1]` and GDB answered `Watchpoint 2:` (no 'Hardware'). Proceed?" — expected: NO — software-watchpoint fallback single-steps the program; delete and narrow the expression. + +All five must route correctly; a miss = fix the text (usually the table row or a heading), re-test. + +- [x] **Step 5: Final commit** + +```bash +pre-commit run --files .claude/skills/target-debug/SKILL.md .claude/agents/target-debugger.md +git add .claude/skills/target-debug/SKILL.md .claude/agents/target-debugger.md +git commit -m "docs(target-debug): integrate new techniques into intrusiveness table; agent playbook bullet + +Retrieval test: 4/4 scenarios routed correctly." +``` + +--- + +## Deferred / out of scope (deliberate) + +- **ETM / J-Trace instruction trace** — no J-Trace hardware on the rig; UM08001 "Trace" chapter is linked for the day one arrives. +- **Monitor-mode debugging as a working recipe** — needs SEGGER monitor files compiled into firmware (a firmware feature, not a doc change); landed as a pointer + `(untested)` in Task 6. +- **ITM stimulus-port logging backend for TU_LOG** — would be a `lib/` + `LOGGER=itm` firmware feature; out of scope for a skill-doc plan. +- **GDB tracepoints (`trace`/`tfind`)** — need a tracing-capable stub; neither JLinkGDBServer nor OpenOCD implements them for Cortex-M. diff --git a/docs/superpowers/specs/2026-07-23-esp-target-debug-design.md b/docs/superpowers/specs/2026-07-23-esp-target-debug-design.md new file mode 100644 index 000000000..491466992 --- /dev/null +++ b/docs/superpowers/specs/2026-07-23-esp-target-debug-design.md @@ -0,0 +1,102 @@ +# esp-target-debug Skill Design + +Backend skill for debugging TinyUSB firmware on Espressif targets (rig: +`espressif_p4_function_ev`, `espressif_s3_devkitm`) via the chips' **built-in +USB-Serial-JTAG**, with external JTAG documented as a TODO until the rig has +an adapter. Companion to `target-debug`, which keeps the architecture-neutral +methodology (intrusiveness ladder, board locks, dual-side capture, diagnosis +standards) — this skill is the Espressif toolchain/probe backend, the same +boundary that makes `usb-kernel-debug` its own skill. + +## Goals + +- An agent can attach, halt, backtrace, set breakpoints/watchpoints, list + FreeRTOS threads, and capture logs on the rig's P4 **while TinyUSB device + traffic is live** — every recipe hardware-verified before landing unmarked + (the `target-debug` ethos). +- The S3's USB-SJ/OTG PHY conflict is mapped precisely, not hand-waved: + verified working via `board_test` (TinyUSB off — PHY free), verified failure + mode with a USB device example, external-JTAG escape hatch documented as + TODO. + +## Non-goals (deferred) + +- External JTAG bring-up (no adapter on the rig) — TODO section with S3 JTAG + pin notes (GPIO39-42) and openocd-esp32 adapter support pointers. +- Xtensa/S3 full parity under live USB traffic (needs external JTAG). +- ETM-class instruction trace; SystemView tooling beyond an apptrace spike. + +## Architecture + +New skill `.claude/skills/esp-target-debug/SKILL.md`; two integration edits: + +- `target-debug` gains a 2-line pointer under the probe-mapping bullets: + Espressif boards use a different toolchain, probe model, and trace story — + read `esp-target-debug`. +- `target-debugger` agent table gains an `esp-target-debug` row (name-only, + aligned columns, per the established conventions). + +Skill content (order): + +1. **Role + defer line** — methodology lives in `target-debug`; this file is + the Espressif backend. Built-in USB-SJ now; external JTAG TODO. +2. **PHY-conflict map** — + - S3: USB-SJ and OTG share one PHY (GPIO19/20). TinyUSB claiming the PHY + drops JTAG-over-USB mid-session: JTAG works for non-USB examples + (`board_test`), dies for USB device examples (verified boundary, exact + symptom recorded). External JTAG = the future escape hatch (TODO). + - P4: OTG-HS has a dedicated HS PHY; USB-SJ is separate — JTAG and the + TinyUSB DUT port coexist (verified). USB-SJ doubles as a live log + console during device traffic — the TU_LOG-equivalent channel. +3. **Toolchain & attach** — `. $HOME/code/esp-idf/export.sh` provides + `openocd-esp32` + `riscv32-esp-elf-gdb` (P4) / `xtensa-esp32s3-elf-gdb` + (S3). Rig path is raw openocd (HIL firmware isn't an idf project on disk): + `openocd -f board/esp32p4-builtin.cfg` with `adapter serial ` (USB-SJ + is VID 303A:1001; uid = the `flasher.uid` already in `tinyusb.json`), gdb + on :3333. `idf.py openocd` / `idf.py gdb` noted for idf-project work. +4. **Technique mapping table** (aligned) — ARM technique → Espressif + equivalent: + + | target-debug technique | Espressif backend | + |---|---| + | GDB autopsy, bp/wp | same flow; RISC-V trigger module (P4) / Xtensa 2 bp + 2 wp (S3); budget read verified on P4 | + | Vector catch | none — breakpoint the panic handler; decode `mcause`/`mepc`/`mtval` (P4) | + | SWO / DWT data trace | none — apptrace over JTAG is the analog (gated spike; lands `(untested)` if it fails) | + | RTT / TU_LOG | USB-SJ console — on P4 it coexists with DUT traffic | + | FreeRTOS threads | native in openocd-esp32 — `info threads` out of the box | + | verifybin | `esptool.py verify_flash` | + +5. **Rig discipline deltas** — same `board_lock.py` protocol; flasher is + esptool (serial-port-by-uid); reflash pristine before release; one client + per USB-SJ device. +6. **External JTAG — TODO** — S3 JTAG pins, adapter classes openocd-esp32 + supports, and the efuse caveat (JTAG pin selection), unverified. + +## Verification gates (execution order) + +All under board locks, serial, evidence in commit messages: + +1. **P4 coexistence (headline)**: flash a device example, confirm enumeration + + traffic on the DUT port, then attach openocd+gdb over USB-SJ → + halt, `bt`, resume — device stays functional after resume. +2. **P4 budget**: read trigger/watchpoint counts via openocd/gdb; set a + hardware watchpoint on a TinyUSB variable, confirm hit. +3. **P4 threads**: `info threads` lists ESP-IDF tasks (usbd task visible). +4. **P4 console**: capture USB-SJ console log output during device traffic. +5. **P4 apptrace spike (gated)**: bounded attempt; verified recipe or + `(untested)` tag. +6. **S3 boundary**: `board_test` flashed → attach works (halt+bt); then a USB + device example → record the exact JTAG failure symptom when the PHY + switches. No further S3 work (external JTAG TODO). + +## Constraints + +- Worktree `claude/improve-debug-skill-agent`; commit per gate; pre-commit + before each; no Co-Authored-By trailers. +- Formatting conventions already established: aligned table columns, + skill-name-only cross references, bullets over run-on paragraphs. +- Espressif builds need `export.sh` first (CLAUDE.md); P4/S3 examples build + via idf.py — reuse existing HIL-built firmware where possible instead of + rebuilding. +- Hardware-verify-before-landing: unverified content ships tagged + `(untested)` or not at all. diff --git a/hw/bsp/rp2040/family.c b/hw/bsp/rp2040/family.c index 55feec159..15f179656 100644 --- a/hw/bsp/rp2040/family.c +++ b/hw/bsp/rp2040/family.c @@ -153,7 +153,7 @@ static stdio_driver_t stdio_rtt = { .in_chars = stdio_rtt_read }; -void stdio_rtt_init(void) { +static void stdio_rtt_init(void) { stdio_set_driver_enabled(&stdio_rtt, true); } #endif diff --git a/test/hil/usbtest.py b/test/hil/usbtest.py index a2841f0b6..e17705a48 100755 --- a/test/hil/usbtest.py +++ b/test/hil/usbtest.py @@ -38,7 +38,7 @@ PID = '4010' GZ_REF = '0525 a4a0' # copy Gadget Zero's capability profile (ctrl_out+iso+intr) SYS_USB = Path('/sys/bus/usb/devices') DRIVER = Path('/sys/bus/usb/drivers/usbtest') -USB_RECOVER = Path(__file__).resolve().parents[2] / '.claude/skills/usb-recover/scripts/usb_recover.sh' +USB_RECOVER = Path(__file__).resolve().parents[2] / '.claude/skills/usb-kernel-recover/scripts/usb_recover.sh' PATTERN_PARAM = Path('/sys/module/usbtest/parameters/pattern') # Battery per tier, in run order: control sanity first, then simple bulk, @@ -391,7 +391,7 @@ def main(): if pci: print(f'aborting battery: kernel-side hang, device wedged mid-transfer.\n' f'auto-recovering: sudo {USB_RECOVER} pci-reset {pci} ' - f'(see .claude/skills/usb-recover)', file=sys.stderr) + f'(see .claude/skills/usb-kernel-recover)', file=sys.stderr) # FLR frees the D-state ioctl without the device lock; must run BEFORE # any unbind/remove_id, which would deadlock the bus otherwise if sudo([str(USB_RECOVER), 'pci-reset', pci]).returncode != 0: @@ -418,7 +418,7 @@ def main(): try: if unrecovered_hang: # testusb is still stuck in a usbfs ioctl holding the device lock; remove_id/unbind - # would join the convoy and deadlock the bus (see usb-recover skill) — leave it be + # would join the convoy and deadlock the bus (see usb-kernel-recover skill) — leave it be print('skipping cleanup after unrecovered hang: reboot required to release the bus', file=sys.stderr) elif not args.keep_binding: