A wedged USB device used to take the whole HIL run with it. Every worker that
touched the poisoned node blocked uninterruptibly, the pool could not be joined,
map_async discarded every board's result, and the job ran to the GitHub ceiling
with no report at all -- while the self-hosted runner's single job slot stayed
occupied and every queued job waited behind it.
Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all
answer within a wall clock; read_sysfs distinguishes "absent" from "unknown",
because a blocked read is not evidence of absence, and caps stranded readers at
four (each costs a thread and an fd for the life of the process) after which the
worker declares itself blind. mtype, the gio unmount, the libmtp session and the
arecord/iperf reaps go through those bounds; the MTP session runs in a disposable
subprocess, since libmtp's ctypes calls block unkillably in D state.
Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job
ceiling and still writes a report. When the pool will not shut down, the sweep
kills what the workers spawned -- descendants, not just direct children, since
flashers run in their own session -- confirms each kill actually landed, and
exits early so the runner is freed. Whatever survived is named in the report.
Deliberately shallow past that point. We do not re-scan process groups, prove
pid ownership, or escalate through sudo: a root-owned survivor is reported, not
force-killed, because signalling a pid we cannot prove is ours is the worse
failure, and the job ceiling backstops whatever this misses. A D-state holder
was never killable anyway.
Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT
through its roster flasher, but only where the flasher can reach its probe past
a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it
cannot, the run says so rather than reserving budget for a path that cannot fire.
Raise the CI ceilings above the pool guard so the guard fires first and still
writes its report, and pin --retry 1 on every HIL leg: the guard is a flat
constant and does not scale with max_retry, so argparse's default of 3 would
triple the serialized usbtest tail against an unchanged guard.
Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in
helper/ (locking, health, selection, shared bounded IO), and the two matrix
generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where
GitHub treats every file as a workflow definition. 193 tests cover the bounded
paths, the kill ladder, the guard and the selector against synthetic /proc trees
and PATH-injected fakes; a real wedge cannot be manufactured on demand.
ceedling 1.1.0 (released 2026-07-17) fails this project's mock
preprocessing ('Failed to read _build/test/preprocess/.../raw/*.h for
comment stripping'); reproduced locally with an isolated 1.1.0 install
while 1.0.1 passes all 61 tests on the same tree. Unpin once fixed
upstream.
- enable --one-per-family to build 1 board per family, also skip family if board specified in -b also present
- minimize ci run for push event
- only build one board per family
- skip hil test on both pi4 and hfp
- full build will be runn for PR event
- IAR always build 1 board per family regardless of event
- update build.py to optimize make
- remove all setup python since we don't really need it
* change concurrency group to ${{ github.workflow }}-${{ github.ref }}
* use argparse for build.py hil_test.py, remove the need to install click
* move ci win/mac to build_cmake.yml
* rename build_family.yml to build_util.yml
* build_util.yml support esp32
* integrate build-espressif into build.yml
* build.py support make with --board option
* add get_deps action
* update hil test to reuse action
* add name field to usbd_class_driver_t
* ci: use set matrix py script
* add riscv32 and cmake support for ch32v307, fomu, gd32vf103
* update build_cmake.py to take --family --board --toolchain
* separate hil test to its own workflow
* move esp32 board into separated hil json
* add make build to ci
* remov build_make.py
* build.py support esp32 board
* setup toolchain support esp-idf
* fix missing click
* merge family in matrix build to reduce jobs
* skip cifuzz since it still has issue with get_deps and click
Currently OSS fuzz expects to have complete control over the
sanitizer flags. As we currently have these set it's causing
problems with the OSS fuzz build. Instead we should use the
provided variables from the OSS fuzz build environment. For
local testing we'll create a set a well defined defaults.
