mirror/tinyusb
3
0
Fork 0
mirror of https://github.com/hathach/tinyusb.git synced 2026-08-18 02:53:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

hil, ci: scope HIL builds and tests to the boards a PR affects (#3797)

Browse Source 
hil, ci: scope HIL builds and tests to the boards a PR affects

Add test/hil/hil_select.py, a stdlib-only selector that maps a PR diff to the
rig boards, tests and BSP families a change can affect, and wire it into CI so
pull requests build and run only those. A port change picks its families' boards,
a class change picks the examples enabling that class, and device/host changes
prune the other role. Anything unclassified — infra, an unmapped port, a selector
error — falls back to the full matrix, and push/schedule runs are untouched.
Move the shared example lists to hil_examples.py; 54 hardware-free tests cover
the rules.
This commit is contained in:
Ha Thach
2026-07-30 02:29:32 +07:00
committed by GitHub
parent 5d8afbb232
commit eef5af86aa
14 changed files with 2479 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
.claude/skills/hil/SKILL.md
View File

@ -41,6 +41,27 @@ python3 test/hil/hil_lock.py release BOARD [BOARD...]
Board/probe health scanning (`test/hil/hil_pool_check.py`) has its own skill: **hil-pool-check**.
Use it before a HIL campaign, after rig maintenance/reboot, or when boards fail to flash.
## PR-scoped selection
`test/hil/hil_select.py` maps a diff to affected boards/tests (used by CI on PRs; fail-open
to the full matrix). Manual use:
```bash
SEL=$(python3 test/hil/hil_select.py --base master test/hil/tinyusb.json)
FULL=$(printf '%s' "$SEL" | python3 -c "import json,sys; print(json.load(sys.stdin)['full'])")
ARGS=$(printf '%s' "$SEL" | python3 -c "import json,sys; print(json.load(sys.stdin)['args']['tinyusb.json'])")
if [ "$FULL" = "True" ] || [ -n "$ARGS" ]; then
python3 test/hil/hil_test.py -B examples $ARGS test/hil/tinyusb.json # $ARGS empty when full: run everything
else
echo "diff affects nothing on this rig - skip HIL"
fi
```
Read `full`, never `args` alone: `args` is empty for BOTH `full: true` (run the whole matrix — a broad or
unclassified change) and "nothing selected" (skip). Skip only when `full` is false AND `args` is empty.
Unit suite: `python3 test/hil/test_hil_select.py` (no hardware).
## Prerequisites
Examples must be built for the target board(s) — see CLAUDE.md "Build" → "All examples for a board" (produces `examples/cmake-build-<board>/`). `-B examples` points `hil_test.py` at that parent folder. (This applies to `hil_test.py`; `hil_pool_check.py` builds its own missing firmware.)

27
.claude/skills/pre-pr/SKILL.md
View File

@ -15,12 +15,27 @@ Run the software + hardware gate for the current branch. The user invoking this
## 2. Map changes to boards
- For each changed `src/portable/<vendor>/<ip>/` (or `src/portable/<name>/` for single-level ports): families = the `hw/bsp/<family>` directories whose build files reference it — `grep -rl "<vendor>/<ip>" hw/bsp/*/family.cmake hw/bsp/*/family.mk`, then take each matching file's directory name.
- For `src/class/*`, `src/common/*`, `src/device/*`, `src/host/*`, or `src/tusb.c`: broad change — use `stm32f407disco` + `raspberry_pi_pico` PLUS any families from portable changes.
- For `hw/bsp/<family>/...` changes: that family directly.
- Catch-all: any other C/CMake source change (`examples/*`, `test/*`, anything unmatched above) → the representative set `stm32f407disco` + `raspberry_pi_pico`. The boards list must NEVER end up empty — final fallback is `[stm32f407disco]` (full-check throws on an empty list).
- Rig roster: `python3 -c "import json;print([b['name'] for b in json.load(open('test/hil/tinyusb.json'))['boards']])"`
- Pick ONE board per affected family, preferring boards on the rig roster; otherwise the first entry in `hw/bsp/<family>/boards/`. Cap at 4 boards and tell the user which families the cap dropped.
- `python3 test/hil/hil_select.py --base $BASE test/hil/tinyusb.json` → JSON with the affected
bsp `families`, the affected rig `boards`, and per-file `reasons`. `full: true` means a
broad/infra change.
- Affected families = `families` ∪ the family of every name in `boards`. Neither half is
enough alone: only the port and bsp rules fill `families` (a class/core/example change
reports boards but no families), and `boards` only ever names rig boards (an off-rig driver
change — `dcd_samx7x.c` → `same7x`, `boards: {}` — would never be compiled).
- A board's family is the `hw/bsp/<family>/boards/<board>/` directory holding it.
- When `full: true`, `boards` names every rig board and carries no signal — use `families`
alone there, plus the representative set below.
- Sample ONE board per affected family: prefer a rig-roster board of that family, else the
first entry in `hw/bsp/<family>/boards/`.
- Rig roster: `python3 -c "import json;print([b['name'] for b in json.load(open('test/hil/tinyusb.json'))['boards']])"`
- Add the representative set `stm32f407disco` + `raspberry_pi_pico` when `full: true` (broad
change — class/core/common/infra).
- Cap at 4 boards and tell the user which families the cap dropped. A broad change affects ~20
families, so the order matters: keep `stm32f407disco` and `raspberry_pi_pico` first whenever
their families are affected, then fill from the remaining families (spread across vendors —
don't let one vendor's family names take every slot). The list must NEVER end up empty —
final fallback is `[stm32f407disco]` (full-check throws on an empty list).
- Docs-only (`full: false`, no families, no boards) keeps §1's minimal software-only gate.
## 3. HIL boards

208
.github/workflows/build.yml vendored
View File

@ -48,20 +48,87 @@ jobs:
outputs:
json: ${{ steps.set-matrix-json.outputs.matrix }}
hil_json: ${{ steps.set-matrix-json.outputs.hil_matrix }}
# one pair per rig job: hil-tinyusb (tinyusb.json minus esptool boards),
# hil-tinyusb-esp (esptool boards only), hil-tinyusb (hfp.json)
hil_args_tinyusb: ${{ steps.hil-select.outputs.args_tinyusb }}
hil_run_tinyusb: ${{ steps.hil-select.outputs.run_tinyusb }}
hil_args_tinyusb_esp: ${{ steps.hil-select.outputs.args_tinyusb_esp }}
hil_run_tinyusb_esp: ${{ steps.hil-select.outputs.run_tinyusb_esp }}
hil_args_hfp: ${{ steps.hil-select.outputs.args_hfp }}
hil_run_hfp: ${{ steps.hil-select.outputs.run_hfp }}
steps:
- name: Checkout TinyUSB
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: HIL selection (PR only)
id: hil-select
if: github.event_name == 'pull_request'
env:
BASE_REF: ${{ github.base_ref }}
run: |
# Best-effort by design: set-matrix gates cmake, hil-build and every rig job,
# so a missing origin/<base>, a shallow-clone hiccup or a selector traceback
# must fall back to the FULL matrix (no --select, run=true, no args) instead
# of failing the job. Same fail-open shape as pr_comment.yml's `|| true`.
SELECT_JSON=''
if ! python3 test/hil/test_hil_select.py; then
echo "::error::hil_select unit tests failed - falling back to the full HIL matrix"
elif ! SELECT_JSON=$(python3 test/hil/hil_select.py --base "origin/$BASE_REF" test/hil/tinyusb.json test/hil/hfp.json); then
echo "::warning::hil_select failed - falling back to the full HIL matrix"
SELECT_JSON=''
fi
# One args/run pair per rig job, split by flasher: a job whose own subset is
# empty skips explicitly instead of running a board filter that matches zero
# boards ("No tests were run." exits 0 and would read as a green HIL run).
OUT=''
if [ -n "$SELECT_JSON" ]; then
OUT=$(SELECT_JSON="$SELECT_JSON" python3 -c '
import json, os
s = json.loads(os.environ["SELECT_JSON"])
tin = s.get("args_flasher", {}).get("tinyusb.json", {})
legs = (("tinyusb", " ".join(a for f, a in sorted(tin.items()) if f != "esptool" and a)),
("tinyusb_esp", tin.get("esptool", "")),
("hfp", s.get("args", {}).get("hfp.json", "")))
for key, a in legs:
print("args_" + key + "=" + a)
print("run_" + key + "=" + ("true" if (s.get("full") or a) else "false"))
') || OUT=''
if [ -z "$OUT" ]; then
echo "::warning::hil_select output unusable - falling back to the full HIL matrix"
SELECT_JSON=''
fi
fi
if [ -z "$OUT" ]; then
OUT=$(for k in tinyusb tinyusb_esp hfp; do printf 'args_%s=\nrun_%s=true\n' "$k" "$k"; done)
fi
echo "$OUT"
{ echo "select=$SELECT_JSON"; echo "$OUT"; } >> $GITHUB_OUTPUT
- name: Generate matrix json
id: set-matrix-json
env:
SELECT: ${{ steps.hil-select.outputs.select }}
run: |
# build matrix
MATRIX_JSON=$(python .github/workflows/ci_set_matrix.py)
echo "matrix=$MATRIX_JSON"
echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT
# HIL matrix (merged from tinyusb + hifiphile configs)
HIL_MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py test/hil/tinyusb.json test/hil/hfp.json)
# HIL matrix (merged from tinyusb + hifiphile configs), scoped on PRs.
# Scoping is best-effort too: fall back to the unscoped (full) matrix.
HIL_MATRIX_JSON=''
if [ -n "$SELECT" ]; then
HIL_MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py --select "$SELECT" test/hil/tinyusb.json test/hil/hfp.json) || HIL_MATRIX_JSON=''
if [ -z "$HIL_MATRIX_JSON" ]; then
echo "::warning::scoped HIL matrix failed - falling back to the full HIL matrix"
fi
fi
if [ -z "$HIL_MATRIX_JSON" ]; then
HIL_MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py test/hil/tinyusb.json test/hil/hfp.json)
fi
echo "hil_matrix=$HIL_MATRIX_JSON"
echo "hil_matrix=$HIL_MATRIX_JSON" >> $GITHUB_OUTPUT
@ -271,6 +338,18 @@ jobs:
strategy:
fail-fast: false
matrix:
# These names are the bucket keys of test/hil/hil_ci_set_matrix.py: every
# non-esptool roster board must land in one of them (esptool boards go to
# 'esp-idf', built by hil-build-esp below). hil_ci_set_matrix.py rejects a
# board whose "toolchain" is not a bucket, so a new bucket must be added in
# both places.
#
# INVARIANT the PR-scoped skip cascade rests on: hil_run_tinyusb / hil_run_hfp
# true => hil-build has at least one non-empty leg. It holds because those
# flags count only non-esptool boards and every such board builds here. It
# would break if an esptool board were added to hfp.json, because
# hil_args_hfp is NOT flasher-split: the hfp leg of hil-tinyusb would want to
# run while hil-build (and therefore that leg) skipped.
toolchain:
- 'arm-gcc'
- 'riscv-gcc'
@ -298,7 +377,7 @@ jobs:
# self-hosted on local VM, for attached hardware checkout HIL_JSON
# ---------------------------------------
hil-tinyusb:
needs: hil-build
needs: [ hil-build, set-matrix ]
name: hil-tinyusb (${{ matrix.display }})
strategy:
fail-fast: false
@ -357,7 +436,31 @@ jobs:
- name: Test on actual hardware
# Single attempt per test (--retry 1), no in-run second pass: a broken fixture
# fails fast instead of holding the runner (and other PRs' HIL jobs) for hours.
run: python3 test/hil/hil_test.py --retry 1 ${{ matrix.test_args }} ${{ env.HIL_JSON }} $RERUN_ARGS
env:
# tinyusb.json minus the esptool boards (they run in hil-tinyusb-esp): each
# job gates on its own flasher subset, never on a rig-wide flag
SEL_ARGS_TINYUSB: ${{ needs.set-matrix.outputs.hil_args_tinyusb }}
SEL_RUN_TINYUSB: ${{ needs.set-matrix.outputs.hil_run_tinyusb }}
SEL_ARGS_HFP: ${{ needs.set-matrix.outputs.hil_args_hfp }}
SEL_RUN_HFP: ${{ needs.set-matrix.outputs.hil_run_hfp }}
run: |
case "$HIL_JSON" in
*tinyusb.json) SEL_ARGS="$SEL_ARGS_TINYUSB"; SEL_RUN="$SEL_RUN_TINYUSB" ;;
*hfp.json) SEL_ARGS="$SEL_ARGS_HFP"; SEL_RUN="$SEL_RUN_HFP" ;;
esac
if [ "$SEL_RUN" = "false" ]; then
echo "HIL skipped by PR selection (no affected boards on this rig)"
# leave a marker so the combined PR comment says so instead of dropping the
# section (and leaving a stale table from an earlier push in its place)
mkdir -p "$HIL_REPORT_DIR"
echo "_Skipped by PR selection: no affected boards on this rig._" > "$HIL_REPORT_DIR/hil_report.md"
exit 0
fi
# a re-run spec is already a subset of the selection (only the boards/tests
# that failed); -b/-bt accumulate, so keeping SEL_ARGS here would re-run the
# entire original selection instead of just what failed
if [ -n "$RERUN_ARGS" ]; then SEL_ARGS=''; fi
python3 test/hil/hil_test.py --retry 1 ${{ matrix.test_args }} $SEL_ARGS ${{ env.HIL_JSON }} $RERUN_ARGS
- name: Upload HIL report
if: always() && github.event_name == 'pull_request'
@ -376,7 +479,7 @@ jobs:
# second slot would double the per-controller flash/usbtest budgets.
# ---------------------------------------
hil-tinyusb-esp:
needs: hil-build-esp
needs: [ hil-build-esp, set-matrix ]
name: hil-tinyusb (tinyusb-esp.json)
runs-on: [ self-hosted, X64, hathach, hardware-in-the-loop ]
env:
@ -420,7 +523,25 @@ jobs:
merge-multiple: true
- name: Test on actual hardware
run: python3 test/hil/hil_test.py --retry 1 $TEST_ARGS ${{ env.HIL_JSON }} $RERUN_ARGS
env:
# esptool subset of tinyusb.json: this job must gate on its own boards, not
# on the rig-wide flag (which would run a filter matching zero boards)
SEL_ARGS: ${{ needs.set-matrix.outputs.hil_args_tinyusb_esp }}
SEL_RUN: ${{ needs.set-matrix.outputs.hil_run_tinyusb_esp }}
run: |
if [ "$SEL_RUN" = "false" ]; then
echo "HIL skipped by PR selection (no affected esptool boards)"
# leave a marker so the combined PR comment says so instead of dropping the
# section (and leaving a stale table from an earlier push in its place)
mkdir -p "$HIL_REPORT_DIR"
echo "_Skipped by PR selection: no affected esptool boards._" > "$HIL_REPORT_DIR/hil_report.md"
exit 0
fi
# a re-run spec is already a subset of the selection (only the boards/tests
# that failed); -b/-bt accumulate, so keeping SEL_ARGS here would re-run the
# entire original selection instead of just what failed
if [ -n "$RERUN_ARGS" ]; then SEL_ARGS=''; fi
python3 test/hil/hil_test.py --retry 1 $TEST_ARGS $SEL_ARGS ${{ env.HIL_JSON }} $RERUN_ARGS
- name: Upload HIL report
if: always() && github.event_name == 'pull_request'
@ -460,23 +581,75 @@ jobs:
- name: Checkout TinyUSB
uses: actions/checkout@v6
with:
# full history: the "HIL selection" step below needs
# merge-base(HEAD, origin/<base_ref>) for PR-scoped selection
fetch-depth: 0
# Computed BEFORE the build: the IAR build is four boards and up to 30 minutes on
# a runner that hil-tinyusb (hfp.json) also needs, so an unaffected PR must release
# it immediately instead of building everything and then skipping. The selection
# also narrows what gets built.
# This job has no needs: on set-matrix (it must run even if that unrelated job
# fails), so it computes its own selection instead of reading set-matrix's outputs.
- name: HIL selection (PR only)
if: github.event_name == 'pull_request'
env:
BASE_REF: ${{ github.base_ref }}
run: |
# Best-effort: this job is deliberately decoupled from set-matrix so unrelated
# failures cannot kill hfp coverage - a selector failure here must likewise
# fall back to the full hfp matrix (no hil_select.json, no SEL_* vars), never
# fail the job.
if ! python3 test/hil/hil_select.py --base "origin/$BASE_REF" test/hil/hfp.json > hil_select.json; then
echo "::warning::hil_select failed - running the full hfp matrix"
rm -f hil_select.json
exit 0
fi
# hil_select.json is passed to hil_ci_set_matrix.py --select below to scope the
# build; it already honours full=true by ignoring the board list.
# The hil_test.py args go to a file, never to $GITHUB_ENV: they are derived
# from roster board names, which a PR can edit. Only SEL_RUN (a literal
# true/false computed here, needed by the step-level `if:`) goes to the env.
if ! SEL_RUN=$(python3 -c '
import json
s = json.load(open("hil_select.json"))
a = s["args"]["hfp.json"]
open("hil_sel_args.txt", "w").write(a)
print("true" if (s["full"] or a) else "false")
'); then
echo "::warning::hil_select output unusable - running the full hfp matrix"
rm -f hil_select.json hil_sel_args.txt
exit 0
fi
echo "SEL_RUN=$SEL_RUN"
echo "SEL_RUN=$SEL_RUN" >> $GITHUB_ENV
- name: Get build boards
if: env.SEL_RUN != 'false'
run: |
MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py test/hil/hfp.json)
BUILD_ARGS=$(echo $MATRIX_JSON | jq -r '.["arm-gcc"] | join(" ")')
if [ -f hil_select.json ]; then
MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py --select "$(cat hil_select.json)" test/hil/hfp.json)
else
MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py test/hil/hfp.json)
fi
# Each variant carries its own --build-name/--cflag, which are global to a
# single build.py invocation — so keep one matrix entry per line and build
# them one at a time (joining would leak a variant's flags onto every board).
echo "$MATRIX_JSON" | jq -r '.["arm-gcc"][]' > hil_build_entries.txt
cat hil_build_entries.txt
BUILD_ARGS=$(echo "$MATRIX_JSON" | jq -r '.["arm-gcc"] | join(" ")')
echo "BUILD_ARGS=$BUILD_ARGS"
echo "BUILD_ARGS=$BUILD_ARGS" >> $GITHUB_ENV
- name: Get Dependencies
if: env.SEL_RUN != 'false'
run: python3 tools/get_deps.py $BUILD_ARGS
- name: Build
if: env.SEL_RUN != 'false'
run: |
# Each variant carries its own --build-name/--cflag, which are global to a
# single build.py invocation — so build one matrix entry at a time rather
# than joining them (joining would leak a variant's flags onto every board).
readarray -t ENTRIES < <(python test/hil/hil_ci_set_matrix.py test/hil/hfp.json | jq -r '.["arm-gcc"][]')
readarray -t ENTRIES < hil_build_entries.txt
for entry in "${ENTRIES[@]}"; do
echo "+ tools/build.py --toolchain iar $entry"
python3 tools/build.py --toolchain iar $entry
@ -484,7 +657,16 @@ jobs:
- name: Test on actual hardware (hardware in the loop)
run: |
python3 test/hil/hil_test.py hfp.json
if [ "$SEL_RUN" = "false" ]; then
echo "HIL skipped by PR selection (no affected boards on this rig)"
# leave a marker so the combined PR comment says so instead of dropping
# the section (and leaving a stale table from an earlier push)
echo "_Skipped by PR selection: no affected boards on this rig._" > hil_report.md
exit 0
fi
# empty/absent on a non-PR event or a selector fallback -> full hfp matrix
SEL_ARGS=$(cat hil_sel_args.txt 2>/dev/null || true)
python3 test/hil/hil_test.py $SEL_ARGS hfp.json
- name: Upload HIL report
if: always() && github.event_name == 'pull_request'

15
.github/workflows/build_util.yml vendored
View File

@ -39,6 +39,21 @@ on:
jobs:
family:
# PR-scoped HIL selection can produce an empty build-args list for a toolchain
# (e.g. a dwc2-only change with no riscv boards affected); an empty matrix
# vector fails the job outright ("Matrix vector 'arg' does not contain any
# values"), so skip cleanly instead.
#
# Why that is safe for callers: GitHub SKIPS the dependents of a skipped
# `needs:` job, so this only works because the caller (hil-build) is itself a
# *matrix* job - a matrix with one skipped leg and one successful leg
# aggregates to success, and its dependents run.
#
# NOT covered: if every leg is empty the whole caller job skips, and so does
# everything that needs it. That is fine only because an all-empty selection
# means no board was selected for those rigs, so the rig jobs would have had
# nothing to run anyway (see the invariant on hil-build in build.yml).
if: inputs.build-args != '[]'
runs-on: ${{ inputs.os }}
strategy:
fail-fast: false

9
.github/workflows/pr_comment.yml vendored
View File

@ -101,7 +101,16 @@ jobs:
shopt -s nullglob
dirs=(hil-reports/hil-report-*)
if [ ${#dirs[@]} -eq 0 ]; then
# No rig produced a report: PR selection matched no board anywhere, or the
# HIL jobs did not run at all. Post it rather than exiting, so a table from
# an earlier push is replaced instead of being left to look current.
echo "No HIL reports found"
{
echo "## Hardware-in-the-loop (HIL) Test Report"
echo
echo "_No HIL run for this push (no affected boards, or hardware testing did not run)._"
} > hil_combined.md
echo "found=true" >> "$GITHUB_OUTPUT"
exit 0
fi
{

856
docs/superpowers/plans/2026-07-29-hil-select.md Normal file
View File

@ -0,0 +1,856 @@
# PR-Scoped HIL Selection Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** A diff→(boards, tests) selector (`test/hil/hil_select.py`) that scopes CI's HIL build+test jobs on pull requests and is reusable locally, per `docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md`.
**Architecture:** Pure-stdlib classification engine (changed files → per-board test selection, fail-open to full) + thin CLI emitting JSON with per-rig `hil_test.py` arg strings; consumed by `hil_ci_set_matrix.py --select` (prunes hil-build) and shell steps in the three HIL jobs (prunes rig runs). Test lists shared via new `hil_examples.py`.
**Tech Stack:** Python 3.11 stdlib only (`re`, `json`, `glob`, `subprocess` for git), `unittest` for tests, GitHub Actions YAML.
## Global Constraints
- Work in worktree `.claude/worktrees/hil-select` (branch `claude/hil-select`); never touch the primary checkout.
- `hil_select.py`, `hil_examples.py`, `test_hil_select.py` import NOTHING outside the stdlib and each other — in particular never `hil_test`/`hil_flash`/`hil_lock` (GitHub's bare runner has no pyserial/pymtp).
- Fail-open: any changed file matching no classification rule ⇒ `full: true`. Scoping applies to `pull_request` events only; push/scheduled runs stay full.
- Behavior-preserving for existing tools: `hil_test.py` runtime behavior unchanged (only its test-list constants move to `hil_examples.py`); `hil_ci_set_matrix.py` without `--select` emits byte-identical output to today.
- The selector only ever emits board names present in the given roster (`config['boards']`); `boards-skip` is invisible to it.
- Commit messages: imperative, scoped, NO Co-Authored-By/Claude-Session trailers.
- Every commit: `python3 -m py_compile` clean on touched python files, `python3 test/hil/test_hil_select.py` green (once it exists), `pre-commit run --files <touched>` clean.
---
### Task 1: hil_examples.py + selection engine with unit tests
**Files:**
- Create: `test/hil/hil_examples.py`
- Create: `test/hil/hil_select.py` (engine only; CLI comes in Task 2)
- Create: `test/hil/test_hil_select.py`
- Modify: `test/hil/hil_test.py` (import test lists from hil_examples)
- Modify: `test/hil/hil_ci.sh` (scp list gains `hil_examples.py`)
**Interfaces:**
- Produces `hil_examples.py`: `device_tests: list[str]`, `dual_tests: list[str]`, `host_test: list[str]` — the three lists moved VERBATIM (incl. comments) from `hil_test.py`.
- Produces `hil_select.py` engine API used by Task 2:
- `classify(changed_files: list[str], repo_root: str, rosters: list[tuple[str, list[dict]]]) -> dict`
returning `{'full': bool, 'boards': {board_name: 'all' | sorted list[str]}, 'reasons': list[str]}`
where `rosters` = `[(config_path, config['boards']), ...]`.
- `board_roles(board: dict) -> set[str]` — subset of `{'device', 'host'}` from the roster
entry's `tests` flags (`device`/`host`/`dual` booleans; an `only` list contributes the
roles of its entries' path prefixes; `dual` implies both roles).
- `board_family(board_name: str, repo_root: str) -> str | None` — the `<family>` for which
`hw/bsp/<family>/boards/<board_name>` exists.
- `port_families(port_dir: str, repo_root: str) -> set[str]` — directories of
`hw/bsp/*/family.cmake` and `hw/bsp/*/family.mk` whose text contains `port_dir`
(e.g. `raspberrypi/rp2040`).
- `class_examples(class_dir: str, role: str, repo_root: str) -> set[str]` — tests from
`hil_examples` lists whose example `tusb_config.h` enables the class for that role (regex
`#define\s+CFG_TUD_<C>\s+\(?\s*0*[1-9]` / `CFG_TUH_<C>`; exceptions per spec:
`dfu_rt_device.*`→`CFG_TUD_DFU_RUNTIME`, `dfu_device.*`→`CFG_TUD_DFU`, class dir `net`
→ `CFG_TUD_ECM_RNDIS|CFG_TUD_NCM`). Test path `device/x` ⇒ config at
`examples/device/x/src/tusb_config.h`; same pattern for `host/` and `dual/`.
- [ ] **Step 1: Move the test lists into `hil_examples.py`**
Create `test/hil/hil_examples.py`:
```python
#!/usr/bin/env python3
# SPDX-License-Identifier: MIT
# HIL example test lists, shared by hil_test.py (runner) and hil_select.py
# (PR-diff selector). Stdlib-only: hil_select runs on bare CI runners.
```
then MOVE the `device_tests`, `dual_tests`, `host_test` list definitions (and their preceding
comment block "The per-board run order is shuffled...") VERBATIM from `hil_test.py` into it.
In `hil_test.py`, add `from hil_examples import device_tests, dual_tests, host_test` where the
lists were (a `from`-import of data constants is fine here — they are read-only lists used by
name throughout `test_board`). Add `"$ROOT_DIR/test/hil/hil_examples.py" \` to the
`hil_ci.sh` scp list after the `hil_lock.py` line.
- [ ] **Step 2: Verify the move broke nothing**
Run: `cd /home/hathach/code/tinyusb/.claude/worktrees/hil-select && python3 -m py_compile test/hil/hil_examples.py test/hil/hil_test.py && python3 test/hil/hil_test.py --help >/dev/null && echo ok`
Expected: `ok`
- [ ] **Step 3: Write the failing unit tests (spec acceptance cases)**
Create `test/hil/test_hil_select.py`. ROSTER is a trimmed but real-shaped fixture; tests call
the engine API directly (no git, no CLI):
```python
#!/usr/bin/env python3
# SPDX-License-Identifier: MIT
# Unit tests for hil_select.py — pure logic, no hardware, no git. Run directly:
# python3 test/hil/test_hil_select.py
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import hil_select
from hil_examples import device_tests, dual_tests, host_test
REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
ROSTER = [
# device-only, rp2040 family
{'name': 'raspberry_pi_pico', 'uid': 'u1',
'tests': {'device': True, 'host': True, 'dual': True}},
# device-only, stm32f4 family
{'name': 'stm32f407disco', 'uid': 'u2',
'tests': {'device': True, 'host': False, 'dual': False}},
# host-only board
{'name': 'raspberry_pi_pico2', 'uid': 'u3',
'tests': {'device': False, 'host': True, 'dual': False}},
# only-list board (espressif-style)
{'name': 'espressif_s3_devkitm', 'uid': 'u4',
'tests': {'only': ['device/cdc_msc_freertos', 'host/device_info']}},
]
ROSTERS = [('test/hil/tinyusb.json', ROSTER)]
def sel(files):
return hil_select.classify(files, REPO, ROSTERS)
class TestPortRule(unittest.TestCase):
def test_dcd_rp2040_selects_pico_family_only(self):
s = sel(['src/portable/raspberrypi/rp2040/dcd_rp2040.c'])
self.assertFalse(s['full'])
self.assertIn('raspberry_pi_pico', s['boards'])
self.assertNotIn('stm32f407disco', s['boards'])
self.assertNotIn('espressif_s3_devkitm', s['boards'])
# device role: no host tests in pico's list
self.assertTrue(all(not t.startswith('host/') for t in s['boards']['raspberry_pi_pico']))
# host-only boards drop out entirely on a device-role change
self.assertNotIn('raspberry_pi_pico2', s['boards'])
def test_shared_port_file_is_both_roles(self):
s = sel(['src/portable/synopsys/dwc2/dwc2_common.c'])
self.assertFalse(s['full'])
self.assertNotIn('raspberry_pi_pico', s['boards']) # rp2040 is not a dwc2 family
self.assertIn('stm32f407disco', s['boards']) # stm32f4 is
class TestCoreRoleRule(unittest.TestCase):
def test_usbd_selects_all_device_tests_everywhere(self):
s = sel(['src/device/usbd.c'])
self.assertFalse(s['full'])
self.assertNotIn('raspberry_pi_pico2', s['boards']) # host-only board dropped
pico = s['boards']['raspberry_pi_pico']
self.assertTrue(set(device_tests).issubset(set(pico)))
self.assertTrue(set(dual_tests).issubset(set(pico))) # dual survives device role
self.assertTrue(all(not t.startswith('host/') for t in pico))
# only-list board: selection intersects its only-list
esp = s['boards']['espressif_s3_devkitm']
self.assertEqual(esp, ['device/cdc_msc_freertos'])
def test_host_change_drops_device(self):
s = sel(['src/host/usbh.c'])
self.assertFalse(s['full'])
self.assertIn('raspberry_pi_pico2', s['boards'])
self.assertNotIn('stm32f407disco', s['boards']) # device-only board dropped
class TestClassRule(unittest.TestCase):
def test_cdc_device_selects_cdc_examples_only(self):
s = sel(['src/class/cdc/cdc_device.c'])
self.assertFalse(s['full'])
pico = s['boards']['raspberry_pi_pico']
self.assertIn('device/cdc_msc', pico)
self.assertIn('device/cdc_dual_ports', pico)
self.assertNotIn('device/msc_dual_lun', pico) # CFG_TUD_CDC 0 there
self.assertNotIn('device/usbtest', pico) # CFG_TUD_CDC 0 there
self.assertTrue(all(not t.startswith('host/') for t in pico))
def test_msc_host_selects_host_side(self):
s = sel(['src/class/msc/msc_host.c'])
self.assertFalse(s['full'])
self.assertNotIn('stm32f407disco', s['boards']) # device-only board
pico2 = s['boards']['raspberry_pi_pico2']
self.assertIn('host/msc_file_explorer', pico2)
self.assertTrue(all(not t.startswith('device/') for t in pico2))
class TestFallbackRules(unittest.TestCase):
def test_unknown_tool_is_full(self):
s = sel(['tools/random_new_script.py'])
self.assertTrue(s['full'])
def test_docs_only_is_empty_not_full(self):
s = sel(['docs/info/contributing.rst', 'README.rst'])
self.assertFalse(s['full'])
self.assertEqual(s['boards'], {})
def test_bsp_family_selects_family_boards(self):
s = sel(['hw/bsp/rp2040/family.cmake'])
self.assertFalse(s['full'])
self.assertIn('raspberry_pi_pico', s['boards'])
self.assertEqual(s['boards']['raspberry_pi_pico'], 'all')
self.assertNotIn('stm32f407disco', s['boards'])
def test_bsp_board_narrows_to_board(self):
s = sel(['hw/bsp/rp2040/boards/raspberry_pi_pico/board.h'])
self.assertFalse(s['full'])
self.assertEqual(list(s['boards'].keys()), ['raspberry_pi_pico'])
def test_example_change_selects_that_example(self):
s = sel(['examples/device/cdc_msc/src/main.c'])
self.assertFalse(s['full'])
self.assertEqual(s['boards']['raspberry_pi_pico'], ['device/cdc_msc'])
def test_core_common_is_full(self):
for f in ['src/tusb.c', 'src/common/tusb_fifo.c', 'src/osal/osal_freertos.h']:
self.assertTrue(sel([f])['full'], f)
def test_harness_is_full(self):
for f in ['test/hil/hil_test.py', '.github/workflows/build.yml', 'hw/mcu/nxp/x.c', 'lib/foo/x.c']:
self.assertTrue(sel([f])['full'], f)
def test_mixed_roles_no_pruning(self):
s = sel(['src/device/usbd.c', 'src/host/usbh.c'])
self.assertFalse(s['full'])
self.assertIn('raspberry_pi_pico2', s['boards'])
self.assertIn('stm32f407disco', s['boards'])
if __name__ == '__main__':
unittest.main(verbosity=1)
```
- [ ] **Step 4: Run tests to verify they fail**
Run: `python3 test/hil/test_hil_select.py 2>&1 | tail -2`
Expected: `ModuleNotFoundError: No module named 'hil_select'` (or import error).
- [ ] **Step 5: Implement the engine**
Create `test/hil/hil_select.py`:
```python
#!/usr/bin/env python3
# SPDX-License-Identifier: MIT
"""PR-diff -> HIL selection: which rig boards and which tests a change can affect.
Stdlib-only (runs on bare CI runners; never imports hil_test/hil_flash/hil_lock).
Fail-open: any file no rule classifies forces the full matrix. See
docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md.
"""
import argparse
import glob
import json
import os
import re
import subprocess
import sys
from hil_examples import device_tests, dual_tests, host_test
ALL_TESTS = {'device': device_tests, 'dual': dual_tests, 'host': host_test}
# class dir -> config macro suffix exceptions (rule 3); dfu is per-file, handled inline
NET_MACROS = ('ECM_RNDIS', 'NCM')
_NONCODE_RE = re.compile(
r'^(docs/|\.claude/|.*\.(md|rst|txt)$|LICENSE)')
_FULL_RE = re.compile(
r'^(src/common/|src/osal/|src/tusb\.c$|src/tusb\.h$|src/tusb_option\.h$|'
r'test/hil/|\.github/workflows/build.*\.yml$|\.github/actions/|'
r'tools/build\.py$|tools/get_deps\.py$|tools/cmake/|hw/mcu/|lib/|'
r'hw/bsp/(family_support\.cmake|board_api\.h|board\.c|ansi_escape\.h)$|'
r'examples/build_system/|examples/CMakeLists\.txt$)')
def test_role(test: str) -> str:
return test.split('/', 1)[0] # 'device' | 'dual' | 'host'
def board_roles(board: dict) -> set:
t = board.get('tests', {})
roles = set()
if t.get('device'):
roles.add('device')
if t.get('host'):
roles.add('host')
if t.get('dual'):
roles.update(('device', 'host'))
for only in t.get('only', []):
r = test_role(only)
roles.update(('device', 'host') if r == 'dual' else (r,))
return roles
def board_tests(board: dict) -> list:
"""Every test this board would run today (mirrors hil_test.test_board's default)."""
t = board.get('tests', {})
if 'only' in t:
run = list(t['only'])
else:
run = []
if t.get('device'):
run += device_tests
if t.get('dual'):
run += dual_tests
if t.get('host'):
run += host_test
return [x for x in run if x not in t.get('skip', [])]
def board_family(board_name: str, repo_root: str):
hits = glob.glob(os.path.join(repo_root, 'hw/bsp/*/boards', board_name))
return os.path.basename(os.path.dirname(os.path.dirname(hits[0]))) if hits else None
def port_families(port_dir: str, repo_root: str) -> set:
fams = set()
for f in glob.glob(os.path.join(repo_root, 'hw/bsp/*/family.cmake')) + \
glob.glob(os.path.join(repo_root, 'hw/bsp/*/family.mk')):
try:
if port_dir in open(f).read():
fams.add(os.path.basename(os.path.dirname(f)))
except OSError:
pass
return fams
def _config_enables(cfg_path: str, macros) -> bool:
try:
text = open(cfg_path).read()
except OSError:
return False
return any(re.search(rf'#define\s+{m}\s+\(?\s*0*[1-9]', text) for m in macros)
def class_examples(macros, role: str, repo_root: str) -> set:
"""Tests (from role's + dual lists) whose example config enables any macro."""
pools = {'device': device_tests + dual_tests, 'host': host_test + dual_tests}
out = set()
for test in pools[role]:
cfg = os.path.join(repo_root, 'examples', test, 'src', 'tusb_config.h')
if _config_enables(cfg, macros):
out.add(test)
return out
class _Sel:
"""Accumulates contributions. board->set(tests) plus 'all-board' markers."""
def __init__(self):
self.full = False
self.by_board = {} # name -> set of tests, or 'all'
self.roles = set() # roles touched by any contribution
self.reasons = []
def add(self, boards, tests, reason):
"""tests: 'all' or iterable of test paths."""
self.reasons.append(reason)
for b in boards:
cur = self.by_board.get(b)
if tests == 'all' or cur == 'all':
self.by_board[b] = 'all'
else:
self.by_board[b] = (cur or set()) | set(tests)
def force_full(self, reason):
self.full = True
self.reasons.append(reason)
def _classify_one(path, repo_root, roster_boards, s: _Sel):
base = os.path.basename(path)
if _NONCODE_RE.match(path):
s.reasons.append(f'{path}: non-code, no contribution')
return
if _FULL_RE.match(path):
s.force_full(f'{path}: core/infra -> full matrix')
return
m = re.match(r'src/portable/((?:[^/]+/)?[^/]+)/', path)
if m:
port = m.group(1)
if re.match(r'(dcd_|.*_device)', base):
roles = {'device'}
elif re.match(r'(hcd_|.*_host)', base):
roles = {'host'}
else:
roles = {'device', 'host'}
fams = port_families(port, repo_root)
boards = [b['name'] for b in roster_boards
if board_family(b['name'], repo_root) in fams and (board_roles(b) & roles)]
tests = [t for r in roles for t in ALL_TESTS[r]] + dual_tests
s.roles.update(roles)
s.add(boards, tests, f'{path}: port {port} -> families {sorted(fams)} -> boards {boards} ({"/".join(sorted(roles))})')
return
m = re.match(r'src/class/([^/]+)/', path)
if m:
cls = m.group(1)
if re.search(r'_device\.[ch]$', base):
roles = {'device'}
elif re.search(r'_host\.[ch]$', base):
roles = {'host'}
else:
roles = {'device', 'host'}
# macro names per role
def macros(prefix):
if cls == 'net':
return [f'CFG_{prefix}_{m2}' for m2 in NET_MACROS]
if cls == 'dfu':
if base.startswith('dfu_rt'):
return [f'CFG_{prefix}_DFU_RUNTIME']
if base.startswith('dfu_device') or base.startswith('dfu_host'):
return [f'CFG_{prefix}_DFU']
return [f'CFG_{prefix}_DFU', f'CFG_{prefix}_DFU_RUNTIME']
return [f'CFG_{prefix}_{cls.upper()}']
tests = set()
if 'device' in roles:
tests |= class_examples(macros('TUD'), 'device', repo_root)
if 'host' in roles:
tests |= class_examples(macros('TUH'), 'host', repo_root)
boards = [b['name'] for b in roster_boards if board_roles(b) & roles]
s.roles.update(roles)
s.add(boards, tests, f'{path}: class {cls} -> {sorted(tests)} ({"/".join(sorted(roles))})')
return
m = re.match(r'src/(device|host)/', path)
if m:
role = m.group(1)
boards = [b['name'] for b in roster_boards if role in board_roles(b)]
s.roles.add(role)
s.add(boards, ALL_TESTS[role] + dual_tests, f'{path}: core {role} stack -> all {role} tests')
return
m = re.match(r'hw/bsp/([^/]+)/(?:boards/([^/]+)/)?', path)
if m:
fam, brd = m.group(1), m.group(2)
if brd:
boards = [b['name'] for b in roster_boards if b['name'] == brd]
why = f'{path}: bsp board {brd}'
else:
boards = [b['name'] for b in roster_boards
if board_family(b['name'], repo_root) == fam]
why = f'{path}: bsp family {fam}'
s.roles.update(('device', 'host'))
s.add(boards, 'all', f'{why} -> boards {boards}')
return
m = re.match(r'examples/(device|host|dual)/([^/]+)/', path)
if m:
test = f'{m.group(1)}/{m.group(2)}'
known = any(test in pool for pool in ALL_TESTS.values())
if known:
boards = [b['name'] for b in roster_boards]
role = test_role(test)
s.roles.update(('device', 'host') if role == 'dual' else (role,))
s.add(boards, [test], f'{path}: example -> {test} on all boards')
else:
s.reasons.append(f'{path}: example not in HIL lists, no contribution')
return
s.force_full(f'{path}: unclassified -> full matrix')
def classify(changed_files, repo_root, rosters):
all_boards = []
seen = set()
for _, boards in rosters:
for b in boards:
if b['name'] not in seen:
seen.add(b['name'])
all_boards.append(b)
s = _Sel()
for path in changed_files:
_classify_one(path, repo_root, all_boards, s)
if s.full:
break
if s.full:
return {'full': True, 'boards': {b['name']: 'all' for b in all_boards},
'reasons': s.reasons}
# role pruning: single-role selections drop the other role's tests and boards
by_name = {b['name']: b for b in all_boards}
out = {}
for name, tests in s.by_board.items():
allowed = board_tests(by_name[name])
if tests == 'all':
kept = list(allowed)
else:
kept = [t for t in allowed if t in tests]
if s.roles and s.roles != {'device', 'host'}:
role = next(iter(s.roles))
kept = [t for t in kept if test_role(t) in (role, 'dual')]
if kept:
out[name] = 'all' if set(kept) == set(allowed) else sorted(kept)
return {'full': False, 'boards': out, 'reasons': s.reasons}
```
- [ ] **Step 6: Run tests to verify they pass**
Run: `python3 test/hil/test_hil_select.py`
Expected: all tests PASS (OK line). Iterate on the engine (not the tests) until green; if a
test premise contradicts the repo (e.g. a family name), verify against the tree and fix the
test only with evidence noted in your report.
- [ ] **Step 7: Commit**
```bash
git add test/hil/hil_examples.py test/hil/hil_select.py test/hil/test_hil_select.py test/hil/hil_test.py test/hil/hil_ci.sh
git commit -m "hil: add PR-diff selection engine (hil_select) with shared example lists"
```
---
### Task 2: CLI + args emission
**Files:**
- Modify: `test/hil/hil_select.py` (add `selection_args`, `main`)
- Modify: `test/hil/test_hil_select.py` (add CLI/args tests)
**Interfaces:**
- Consumes: Task 1's `classify` and roster shapes.
- Produces:
- `selection_args(sel: dict, rosters) -> dict` mapping each config path's basename to the
`hil_test.py` argument string for that rig: for each selected board ON that roster,
`-b <name>`, plus `-bt <name>:<t1>,<t2>` when the board's entry is a list (not 'all').
Empty string when no selected board is on that roster. When `sel['full']`, every roster
board gets bare `-b`? NO — full means "today's behavior": `selection_args` returns `''`
for every config (no filtering args at all).
- CLI: `python3 test/hil/hil_select.py [--base REF | --diff-file PATH] CONFIG...` printing
the JSON `{'full', 'boards', 'args', 'reasons'}` to stdout, reasons also to stderr
(one line each, prefixed `hil_select: `). Non-zero exit only on operational errors
(bad ref, unreadable config) — never on an empty selection.
- [ ] **Step 1: Add failing CLI/args tests to `test_hil_select.py`**
```python
class TestArgsEmission(unittest.TestCase):
def test_args_for_scoped_selection(self):
s = sel(['src/portable/raspberrypi/rp2040/dcd_rp2040.c'])
args = hil_select.selection_args(s, ROSTERS)
a = args['tinyusb.json']
self.assertIn('-b raspberry_pi_pico', a)
self.assertNotIn('stm32f407disco', a)
self.assertIn('-bt raspberry_pi_pico:', a) # device-only subset of a device+host board
def test_args_full_is_empty(self):
s = sel(['tools/random_new_script.py'])
self.assertEqual(hil_select.selection_args(s, ROSTERS), {'tinyusb.json': ''})
def test_args_all_board_gets_bare_b(self):
s = sel(['hw/bsp/rp2040/boards/raspberry_pi_pico/board.h'])
a = hil_select.selection_args(s, ROSTERS)['tinyusb.json']
self.assertIn('-b raspberry_pi_pico', a)
self.assertNotIn('-bt', a)
def test_cli_diff_file(self):
import subprocess, tempfile, json as j
with tempfile.NamedTemporaryFile('w', suffix='.txt', delete=False) as f:
f.write('src/class/cdc/cdc_device.c\n')
path = f.name
r = subprocess.run([sys.executable, os.path.join(REPO, 'test/hil/hil_select.py'),
'--diff-file', path, os.path.join(REPO, 'test/hil/tinyusb.json')],
capture_output=True, text=True)
self.assertEqual(r.returncode, 0, r.stderr)
out = j.loads(r.stdout)
self.assertFalse(out['full'])
self.assertIn('tinyusb.json', out['args'])
self.assertTrue(any('cdc_device' in line for line in out['reasons']))
os.unlink(path)
```
- [ ] **Step 2: Run to verify the new tests fail**
Run: `python3 test/hil/test_hil_select.py 2>&1 | tail -3`
Expected: failures/errors mentioning `selection_args`.
- [ ] **Step 3: Implement `selection_args` and `main`**
Append to `hil_select.py`:
```python
def selection_args(sel, rosters):
args = {}
for cfg_path, boards in rosters:
key = os.path.basename(cfg_path)
if sel['full']:
args[key] = ''
continue
parts = []
for b in boards:
chosen = sel['boards'].get(b['name'])
if chosen is None:
continue
parts.append(f'-b {b["name"]}')
if chosen != 'all':
parts.append(f'-bt {b["name"]}:{",".join(chosen)}')
args[key] = ' '.join(parts)
return args
def changed_files_from_git(base, repo_root):
mb = subprocess.run(['git', 'merge-base', 'HEAD', base], cwd=repo_root,
capture_output=True, text=True, check=True).stdout.strip()
diff = subprocess.run(['git', 'diff', '--name-only', f'{mb}..HEAD'], cwd=repo_root,
capture_output=True, text=True, check=True).stdout
return [l for l in diff.splitlines() if l.strip()]
def main():
ap = argparse.ArgumentParser(description=__doc__)
g = ap.add_mutually_exclusive_group(required=True)
g.add_argument('--base', help='git ref to diff against (merge-base..HEAD)')
g.add_argument('--diff-file', help='newline-separated changed-file list')
ap.add_argument('configs', nargs='+', help='rig roster JSON file(s)')
a = ap.parse_args()
repo_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
rosters = []
for c in a.configs:
with open(c) as f:
rosters.append((c, json.load(f)['boards']))
files = (open(a.diff_file).read().splitlines() if a.diff_file
else changed_files_from_git(a.base, repo_root))
files = [f for f in files if f.strip()]
s = classify(files, repo_root, rosters)
s['args'] = selection_args(s, rosters)
for r in s['reasons']:
print(f'hil_select: {r}', file=sys.stderr)
print(json.dumps(s))
if __name__ == '__main__':
main()
```
(The `parts.append f'...'` line above is pseudo-highlighted; write valid Python:
`parts.append(f'-bt {b["name"]}:{",".join(chosen)}')`.)
- [ ] **Step 4: Run the full suite**
Run: `python3 test/hil/test_hil_select.py && chmod +x test/hil/hil_select.py`
Expected: OK.
- [ ] **Step 5: Smoke against the real repo state**
Run: `python3 test/hil/hil_select.py --base HEAD test/hil/tinyusb.json test/hil/hfp.json`
Expected: empty diff ⇒ `{"full": false, "boards": {}, "args": {"tinyusb.json": "", "hfp.json": ""}, ...}` exit 0.
Then: `printf 'src/portable/wch/dcd_ch32_usbfs.c\n' > /tmp/d.txt && python3 test/hil/hil_select.py --diff-file /tmp/d.txt test/hil/tinyusb.json | python3 -m json.tool | head -20`
Expected: only WCH-family boards (nanoch32v203, ch32v103r_r1_1v0, ch32v307v_r1_1v0 — whichever reference that port) with device tests.
- [ ] **Step 6: Commit**
```bash
git add test/hil/hil_select.py test/hil/test_hil_select.py
git commit -m "hil: hil_select CLI with per-rig hil_test argument emission"
```
---
### Task 3: hil_ci_set_matrix --select + build.yml wiring
**Files:**
- Modify: `test/hil/hil_ci_set_matrix.py`
- Modify: `.github/workflows/build.yml` (set-matrix job; hil-build consumers unchanged; hil-tinyusb + hil-tinyusb-esp steps)
**Interfaces:**
- Consumes: Task 2's CLI JSON (`full`, `boards`, `args`).
- Produces:
- `hil_ci_set_matrix.py [--select JSON_STRING] CONFIG...`: with `--select` and
`full == false`, boards not in `select['boards']` are skipped when building the toolchain
buckets; otherwise identical behavior. Buckets stay present (possibly `[]`) so
`fromJSON(...)[toolchain]` keeps resolving.
- set-matrix outputs: `hil_select_json` (compact selection), `hil_args_tinyusb`,
`hil_args_hfp`, `hil_run_tinyusb`, `hil_run_hfp` (string 'true'/'false').
- [ ] **Step 1: Add `--select` to `hil_ci_set_matrix.py`**
In `main()` add:
```python
parser.add_argument('--select', help='hil_select.py JSON; scopes boards when full=false')
```
and after parsing:
```python
selected = None
sel = json.loads(args.select) if args.select else None
if sel and not sel.get('full'):
selected = set(sel.get('boards', {}))
```
then inside the per-board loop, first line:
```python
if selected is not None and board['name'] not in selected:
continue
```
- [ ] **Step 2: Verify byte-identical without --select and scoped with it**
Run: `python3 test/hil/hil_ci_set_matrix.py test/hil/tinyusb.json test/hil/hfp.json > /tmp/m1.json && git stash -q && python3 test/hil/hil_ci_set_matrix.py test/hil/tinyusb.json test/hil/hfp.json > /tmp/m0.json && git stash pop -q && diff /tmp/m0.json /tmp/m1.json && echo identical`
Expected: `identical`.
Then: `python3 test/hil/hil_ci_set_matrix.py --select '{"full": false, "boards": {"raspberry_pi_pico": "all"}}' test/hil/tinyusb.json test/hil/hfp.json`
Expected: JSON whose `arm-gcc` list contains only the raspberry_pi_pico entry, `riscv-gcc`/`esp-idf` = [].
- [ ] **Step 3: Wire set-matrix in `.github/workflows/build.yml`**
In the `set-matrix` job: give the checkout full history and add the selection step between
checkout and matrix generation; make the HIL matrix use it:
```yaml
- name: Checkout TinyUSB
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: HIL selection (PR only)
id: hil-select
if: github.event_name == 'pull_request'
run: |
python3 test/hil/test_hil_select.py
SELECT_JSON=$(python3 test/hil/hil_select.py --base "origin/${{ github.base_ref }}" test/hil/tinyusb.json test/hil/hfp.json)
echo "select=$SELECT_JSON" >> $GITHUB_OUTPUT
python3 - "$SELECT_JSON" >> $GITHUB_OUTPUT <<'EOF'
import json, sys
s = json.loads(sys.argv[1])
args = s.get('args', {})
for cfg, key in (('tinyusb.json', 'tinyusb'), ('hfp.json', 'hfp')):
a = args.get(cfg, '')
run = 'true' if (s['full'] or a) else 'false'
print(f'args_{key}={a}')
print(f'run_{key}={run}')
EOF
```
and in the existing "Generate matrix json" step, change the HIL line to:
```yaml
# HIL matrix (merged from tinyusb + hifiphile configs), scoped on PRs
SELECT='${{ steps.hil-select.outputs.select }}'
HIL_MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py ${SELECT:+--select "$SELECT"} test/hil/tinyusb.json test/hil/hfp.json)
```
Add to the job's `outputs:` block:
```yaml
hil_args_tinyusb: ${{ steps.hil-select.outputs.args_tinyusb }}
hil_args_hfp: ${{ steps.hil-select.outputs.args_hfp }}
hil_run_tinyusb: ${{ steps.hil-select.outputs.run_tinyusb }}
hil_run_hfp: ${{ steps.hil-select.outputs.run_hfp }}
```
(On non-PR events the step is skipped: outputs are empty strings — the consumers below treat
empty `run_*` as 'true' and empty args as no filtering, i.e. today's behavior.)
- [ ] **Step 4: Wire the rig jobs**
In the `hil-tinyusb` job (the matrixed one covering both rigs), find the step that runs
`hil_test.py --retry 1 ${{ matrix.test_args }} ${{ env.HIL_JSON }} $RERUN_ARGS` (~line 360)
and change the step's `run:` to select per-rig args and honor the skip flag:
```yaml
run: |
case "$HIL_JSON" in
*tinyusb.json) SEL_ARGS='${{ needs.set-matrix.outputs.hil_args_tinyusb }}'; SEL_RUN='${{ needs.set-matrix.outputs.hil_run_tinyusb }}' ;;
*hfp.json) SEL_ARGS='${{ needs.set-matrix.outputs.hil_args_hfp }}'; SEL_RUN='${{ needs.set-matrix.outputs.hil_run_hfp }}' ;;
esac
if [ "$SEL_RUN" = "false" ]; then echo "HIL skipped by PR selection (no affected boards on this rig)"; exit 0; fi
python3 test/hil/hil_test.py --retry 1 ${{ matrix.test_args }} $SEL_ARGS ${{ env.HIL_JSON }} $RERUN_ARGS
```
Apply the same pattern to the second `hil_test.py` invocation at ~line 423 (`hil-tinyusb-esp`,
which is tinyusb-rig only: use the `hil_args_tinyusb`/`hil_run_tinyusb` outputs directly, no
case needed) and to the hfp job's direct `python3 test/hil/hil_test.py hfp.json` call at
~line 487 (use `hil_args_hfp`/`hil_run_hfp`). Preserve each step's existing surrounding lines
(report-dir env, RERUN_ARGS logic) — only inject the SEL_ARGS/SEL_RUN mechanics.
- [ ] **Step 5: Validate the YAML and the exact shell locally**
Run: `pre-commit run check-yaml --files .github/workflows/build.yml && python3 -c "import yaml; yaml.safe_load(open('.github/workflows/build.yml')); print('yaml ok')"`
Expected: `yaml ok` (pyyaml is available; if not, `pip install --user pyyaml` first).
Also simulate the selection step's python inline script:
`SELECT_JSON=$(python3 test/hil/hil_select.py --diff-file /tmp/d.txt test/hil/tinyusb.json test/hil/hfp.json) && python3 -c "import json,sys; s=json.loads(sys.argv[1]); print(s['args'])" "$SELECT_JSON"`
Expected: the args dict prints.
- [ ] **Step 6: Commit**
```bash
git add test/hil/hil_ci_set_matrix.py .github/workflows/build.yml
git commit -m "ci: scope HIL build+test matrix by PR diff via hil_select"
```
---
### Task 4: pre-pr + hil skill docs, final validation
**Files:**
- Modify: `.claude/skills/pre-pr/SKILL.md` (mapping section delegates to the selector)
- Modify: `.claude/skills/hil/SKILL.md` (document the selector for manual runs)
**Interfaces:**
- Consumes: Task 2's CLI.
- [ ] **Step 1: Rewrite pre-pr's "2. Map changes to boards" section**
Replace the section's grep heuristics (keep its numbered-section structure and the roster/cap
policy) with:
```markdown
## 2. Map changes to boards
- `python3 test/hil/hil_select.py --base $BASE test/hil/tinyusb.json` → JSON with the affected
rig boards (`boards`) and per-file `reasons`. `full: true` means a broad/infra change.
- Build-board sampling: from the selection's boards (or, when `full`, the representative set
`stm32f407disco` + `raspberry_pi_pico`), pick ONE board per family, preferring rig-roster
boards; cap at 4 and tell the user which families the cap dropped. The boards list must
NEVER end up empty — final fallback is `[stm32f407disco]`.
- A `full: true` selection or an empty one (docs-only) keeps today's behavior: minimal
software-only gate for docs-only, representative set otherwise.
```
- [ ] **Step 2: Add a short "PR-scoped selection" note to the hil skill**
Append to `.claude/skills/hil/SKILL.md` after the pool-check section:
```markdown
## PR-scoped selection
`test/hil/hil_select.py` maps a diff to affected boards/tests (used by CI on PRs; fail-open
to the full matrix). Manual use:
```bash
ARGS=$(python3 test/hil/hil_select.py --base master test/hil/tinyusb.json | python3 -c "import json,sys; print(json.load(sys.stdin)['args']['tinyusb.json'])")
python3 test/hil/hil_test.py -B examples $ARGS test/hil/tinyusb.json
```
Unit suite: `python3 test/hil/test_hil_select.py` (no hardware).
```
- [ ] **Step 3: Full validation sweep**
Run: `python3 test/hil/test_hil_select.py && python3 -m py_compile test/hil/hil_select.py test/hil/hil_examples.py test/hil/hil_ci_set_matrix.py test/hil/hil_test.py && python3 test/hil/hil_test.py --help >/dev/null && pre-commit run --files $(git diff --name-only claude/hil-pool-check..HEAD) && echo ALL-GREEN`
Expected: `ALL-GREEN`.
- [ ] **Step 4: Real-diff spot checks (acceptance)**
Run each and eyeball the JSON (record outputs in your report):
```bash
for f in 'src/portable/raspberrypi/rp2040/dcd_rp2040.c' 'src/device/usbd.c' 'src/class/cdc/cdc_device.c' 'src/host/usbh.c'; do
printf '%s\n' "$f" > /tmp/d.txt
echo "=== $f"; python3 test/hil/hil_select.py --diff-file /tmp/d.txt test/hil/tinyusb.json test/hil/hfp.json 2>/dev/null | python3 -m json.tool | sed -n '1,25p'
done
```
Expected: matches the spec's acceptance examples (pico-family only / all-device / CDC examples
only / host side only, with hfp.json args populated only where hfp boards qualify).
- [ ] **Step 5: Commit**
```bash
git add .claude/skills/pre-pr/SKILL.md .claude/skills/hil/SKILL.md
git commit -m "docs: pre-pr and hil skill use hil_select for PR-scoped boards"
```

179
docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md Normal file
View File

@ -0,0 +1,179 @@
# PR-scoped HIL selection: hil_select.py
**Date:** 2026-07-29
**Branch:** `claude/hil-select` (based on `claude/hil-pool-check`, which carries the
hil_lock/hil_flash split and the current rig rosters)
## Motivation
Every PR currently builds and runs the full HIL matrix (both rigs, every roster board, every
test). Most PRs touch one port or one class: a `dcd_rp2040` change cannot affect an STM32 board,
a `cdc_device.c` change cannot affect an MSC-only example, and a device-stack change cannot
affect host tests. Scoping HIL to the affected boards/tests cuts CI wall time and rig wear
without losing relevant coverage.
## Goal / non-goals
**Goal:** a shared selector that maps a PR diff to (boards, per-board test lists), wired into
CI's `set-matrix` on `pull_request` events (pruning both `hil-build` and the rig jobs) and
callable locally (pre-pr, manual runs). Scoping may only shrink coverage when the mapping is
confident; every uncertainty widens to the full matrix.
**Non-goals:**
- Variant-level selection (all variants of a selected board run).
- Scoping the non-HIL build jobs (cmake/CircleCI one-per-family builds are independent build
coverage and stay untouched).
- Scoping push/master/scheduled runs (always full).
- Changing hil_test.py behavior (the selector only *composes* existing `-b`/`-bt` args).
## Component: `test/hil/hil_select.py`
Stdlib-only, importable and CLI. Lives beside the harness so `hil_ci.sh` copies are unaffected
(it runs on the GitHub runner / dev PC, not on the rig). It must NOT import `hil_test.py`
(which drags pyserial/pymtp onto the bare GitHub runner): the three test lists
(`device_tests`, `dual_tests`, `host_test`) move verbatim into a tiny stdlib-only
`test/hil/hil_examples.py` that both `hil_test.py` and `hil_select.py` import (behavior
preserving; `hil_ci.sh` scp list gains the new file).
```
python3 test/hil/hil_select.py --base <ref> [--diff-file <path>] CONFIG.json [CONFIG.json...]
```
- `--base REF`: changed files = `git diff --name-only $(git merge-base HEAD REF)..HEAD`
(mirrors pre-pr). `--diff-file`: newline-separated file list instead of git (unit tests, CI
reuse of a precomputed diff).
- Output (stdout, JSON):
```json
{
"full": false,
"boards": {"raspberry_pi_pico": "all", "stm32f407disco": ["device/cdc_msc", "device/cdc_dual_ports"]},
"args": {"tinyusb.json": "-b raspberry_pi_pico -b stm32f407disco -bt stm32f407disco:device/cdc_msc,device/cdc_dual_ports",
"hfp.json": ""},
"reasons": ["src/portable/raspberrypi/rp2040/dcd_rp2040.c: port rp2040 -> family rp2040 -> boards [raspberry_pi_pico, ...] (device role)"]
}
```
- `full: true` ⇒ `boards`/`args` cover the entire rosters (identical to today's behavior).
- `args` maps each input config file to the hil_test.py argument string for that rig: `-b` per
selected board on that roster, plus `-bt BOARD:t1,t2` for boards with a restricted test list
("all" boards get bare `-b`). An empty string means: nothing on this rig is affected — the
rig job is skipped for this PR.
- Per-file reasoning lines (`file → rule → contribution`) go in `reasons` and to stderr, so the
CI log answers "why did/didn't HIL run X" without archaeology.
## Classification rules
Each changed file yields a contribution; the selection is the union. Any file matching no rule
sets `full: true` (fail-open). Rules, first match wins:
1. **Non-code:** `docs/**`, `.claude/**` (except the workflows below via rule 8), `*.md`,
`*.rst`, `LICENSE*` → contributes nothing.
2. **Port:** `src/portable/<vendor>/<ip>/**` (or single-level `src/portable/<name>/**`).
Role from basename: `dcd_*`/`*_device*` → device; `hcd_*`/`*_host*` → host; anything else
(shared port files, e.g. `dwc2/dwc2_common.c`) → both. Families = directories of
`hw/bsp/*/family.cmake|family.mk` whose text references `<vendor>/<ip>` (pre-pr's grep),
boards = those families' entries on the input rosters. Tests = all tests of that role
(device_tests / host_test from hil_test.py's lists; dual_tests count as both roles).
3. **Class:** `src/class/<c>/*_device.*` → all device-capable roster boards; tests = the
device/dual examples in hil_test.py's lists whose `examples/<role>/<ex>/src/tusb_config.h`
defines `CFG_TUD_<C>` with a nonzero value (derived at runtime; `<C>` = upper-cased class
dir, with the map `musb→n/a`-style exceptions NOT needed — class dirs and config macros
share names: cdc, msc, hid, midi, audio, video, vendor, usbtmc, mtp, printer. Two
exceptions: in class dir `dfu`, `dfu_rt_device.*` maps to CFG_TUD_DFU_RUNTIME and
`dfu_device.*` to CFG_TUD_DFU; class dir `net` maps to CFG_TUD_ECM_RNDIS|CFG_TUD_NCM.) `*_host.*` analogously via `CFG_TUH_<C>`. Shared class files (e.g. `cdc.h`) →
both roles' matching examples. A class with zero matching examples contributes nothing
(known path, does not force full).
4. **Core role:** `src/device/**` → all device-capable boards, all device tests (+dual);
`src/host/**` → all host-capable boards, all host tests (+dual).
5. **Core common:** `src/common/**`, `src/osal/**`, `src/tusb.c`, `src/tusb.h`,
`src/tusb_option.h` → full.
6. **BSP:** `hw/bsp/<family>/**` → that family's roster boards, all their tests;
`hw/bsp/<family>/boards/<board>/**` narrows to that board if it is on a roster, and
contributes nothing when it is not (an off-rig board cannot be HIL-tested; known path,
does not force full).
Family-agnostic BSP files (`hw/bsp/board_api.h`, `hw/bsp/board.c`, ansi_escape.h) → full.
7. **Example:** `examples/<role>/<ex>/**` → all roster boards, tests = that example if present
in hil_test.py's lists, else contributes nothing. `examples/build_system/**`, top-level
`examples/CMakeLists.txt` → full. `examples/device/board_test/**` → full: it is the park
firmware hil_test.py flashes on every board (variant boundary + teardown), not a test.
8. **Harness/infra:** `test/hil/**`, `.github/workflows/build*.yml`,
`.github/actions/**`, `tools/build.py`, `tools/get_deps.py`, `tools/cmake/**`,
`hw/mcu/**`, `lib/**` → full.
9. **Everything else** (`test/unit-test/**`, `tools/**` not above, unknown paths) → full.
(Unit-test-only changes could safely skip HIL, but per the fail-open stance anything not
explicitly classified widens; narrowing rule 9 is a later refinement.)
**Role pruning:** after the union, if only device-role contributions exist, host-only boards
drop out and host tests are stripped from mixed boards (vice versa for host-only changes).
Dual tests survive either role. Board capability (device/host) comes from the roster entry's
`tests` flags/only-list, same logic hil_test.py uses.
**No-rig-coverage case:** a cleanly classified change whose boards intersect a roster to the
empty set yields an empty `args` string for that rig and a stderr line saying so — the rig job
is skipped, not widened (running unrelated boards would test nothing relevant).
**Roster source:** `config['boards']` only (boards-skip stays parked).
## CI wiring (`.github/workflows/build.yml`)
- `set-matrix` (PR events only): after generating today's matrices, run
`hil_select.py --base origin/${{ github.base_ref }} test/hil/tinyusb.json test/hil/hfp.json`
(checkout with enough history to reach the merge base: `fetch-depth: 0` on this one job, or
an explicit `git fetch origin $BASE_REF`). New job outputs: `hil_select_full`,
`hil_args_tinyusb`, `hil_args_hfp`, plus the selected-board list consumed by the matrix
generator. Non-PR events: skip the selector, outputs default to full/empty-args-means-all.
- `hil_ci_set_matrix.py` gains `--select '<json>'`: when given and `full` is false, it emits
build entries only for selected boards (per config). Untouched otherwise.
- `hil-tinyusb` job (one matrixed job covering both rigs, selected by `matrix.hil_json`): a
step picks the rig's selector args in shell (`case "$HIL_JSON" in ...`) from the set-matrix
outputs and either appends them to the `hil_test.py` invocation or exits the step early with
a "HIL skipped by selection" log line when that rig has nothing to run (`run` flag output
false). The separate `hil-tinyusb-esp` job (esptool split) gets the same treatment with the
tinyusb args. Non-PR events: outputs default to run=true with empty args (today's behavior).
- The `--flasher`/`--exclude-flasher` split in the existing matrix `test_args` composes fine
with `-b` (hil_test.py applies both filters).
## Local use
- pre-pr's "Map changes to boards" step delegates to
`python3 test/hil/hil_select.py --base $BASE test/hil/tinyusb.json` and derives its
one-board-per-family sample from the selector's board set (its capping/sampling policy is
unchanged — the selector provides the affected set, pre-pr samples it).
- Manual: `python3 test/hil/hil_test.py -B examples $(python3 test/hil/hil_select.py --base master test/hil/tinyusb.json | jq -r '.args["tinyusb.json"]') test/hil/tinyusb.json`
— documented in the hil skill.
## Testing
`test/hil/test_hil_select.py` — stdlib `unittest`, no hardware, injected diffs via
`--diff-file`/API. Cases (the acceptance examples):
1. `src/portable/raspberrypi/rp2040/dcd_rp2040.c` → only rp2040-family roster boards, device
tests only, host-only boards absent, `full` false.
2. `src/device/usbd.c` → every device-capable board on both rosters, all device tests + dual,
no host-only board, no host tests.
3. `src/class/cdc/cdc_device.c` → only examples with CFG_TUD_CDC enabled (must include
device/cdc_msc and device/cdc_dual_ports; must exclude device/msc_dual_lun and all
host tests).
4. `src/class/msc/msc_host.c` → host-capable boards only, host examples with CFG_TUH_MSC.
5. `tools/random_new_script.py` → `full: true`.
6. `docs/foo.rst` alone → contributes nothing ⇒ empty selection, `full` false, all `args`
empty (CI additionally has check-paths gating; the selector's answer is still honest).
7. `hw/bsp/rp2040/family.cmake` → rp2040-family boards, all their tests.
8. Mixed device+host diff → no pruning (both roles present).
The suite runs in `set-matrix` before the selector is used, and locally via
`python3 test/hil/test_hil_select.py`.
## Safety properties
- Fail-open: unknown/infra paths ⇒ full matrix; selector crash in CI ⇒ job fails visibly
(never silently skips HIL).
- Only `pull_request` events are scoped.
- The selection JSON + per-file reasons are printed in the job log for audit.
- hil_test.py errors on `-b` names not in the config — the selector only emits roster names,
and the unit suite locks that invariant.
## Sequencing
Lands on `claude/hil-select` on top of the pool-check/split stack. Follow-ups it does not
include: narrowing rule 9 for unit-test-only changes; variant-level selection; pre-pr skill
text update ships in the same change (its mapping section shrinks to a selector call).

11
hw/bsp/mcx/family.cmake
View File

@ -94,10 +94,19 @@ function(family_configure_example TARGET RTOS)
family_add_tinyusb(${TARGET} OPT_MCU_MCXA15)
endif()
# PORT is set per board (board.cmake), so pick the driver at configure time. Spelled out
# rather than $<IF:${PORT},...> so the port path stays greppable: test/hil/hil_select.py
# maps a portable-driver change to the families whose build file names that directory.
if (PORT)
set(PORT_SRC ${TOP}/src/portable/chipidea/ci_hs/dcd_ci_hs.c)
else ()
set(PORT_SRC ${TOP}/src/portable/chipidea/ci_fs/dcd_ci_fs.c)
endif ()
target_sources(${TARGET} PUBLIC
${CMAKE_CURRENT_FUNCTION_LIST_DIR}/family.c
${CMAKE_CURRENT_FUNCTION_LIST_DIR}/../board.c
${TOP}/src/portable/chipidea/$<IF:${PORT},ci_hs/dcd_ci_hs.c,ci_fs/dcd_ci_fs.c>
${PORT_SRC}
${STARTUP_FILE_${CMAKE_C_COMPILER_ID}}
)
target_include_directories(${TARGET} PUBLIC

1
test/hil/hil_ci.sh
View File

@ -55,6 +55,7 @@ scp -q "$ROOT_DIR/test/hil/hil_test.py" \
"$ROOT_DIR/test/hil/hil_flash.py" \
"$ROOT_DIR/test/hil/hil_lock.py" \
"$ROOT_DIR/test/hil/usbtest.py" \
"$ROOT_DIR/test/hil/hil_examples.py" \
"$ROOT_DIR/test/hil/pymtp.py" \
"$CONFIG" \
"$REMOTE:$REMOTE_DIR/test/hil/"

15
test/hil/hil_ci_set_matrix.py
View File

@ -17,8 +17,14 @@ def _resolve_config_path(config_file):
def main():
parser = argparse.ArgumentParser()
parser.add_argument('config_files', nargs='+', help='Configuration JSON file(s)')
parser.add_argument('--select', help='hil_select.py JSON; scopes boards when full=false')
args = parser.parse_args()
selected = None
sel = json.loads(args.select) if args.select else None
if sel and not sel.get('full'):
selected = set(sel.get('boards', {}))
# Toolchain buckets must match the toolchains instantiated by the hil-build
# job in .github/workflows/build.yml. Keep all keys present (even if empty)
# so `fromJSON(hil_json)[toolchain]` always resolves to a list.
@ -40,6 +46,8 @@ def main():
config = json.load(f)
for board in config['boards']:
if selected is not None and board['name'] not in selected:
continue
name = board['name']
flasher = board['flasher']
# esptool boards must build under esp-idf; others default to arm-gcc
@ -49,6 +57,13 @@ def main():
toolchain = 'esp-idf'
else:
toolchain = board.get('toolchain', 'arm-gcc')
if toolchain not in matrix:
# a board in no bucket would never be built, and the bare KeyError
# below would only say so as a traceback from the set-matrix job
raise SystemExit(
f'{name}: toolchain {toolchain!r} is not a build bucket '
f'({", ".join(matrix)}); add it here and to the hil-build / '
f'hil-build-esp jobs in .github/workflows/build.yml')
build_board = f'-b {name}'
if 'build' in board and 'args' in board['build']:

37
test/hil/hil_examples.py Normal file
View File

@ -0,0 +1,37 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: MIT
# HIL example test lists, shared by hil_test.py (runner) and hil_select.py
# (PR-diff selector). Stdlib-only: hil_select runs on bare CI runners.
# The per-board run order is shuffled (see test_board).
# Every example carries a unique hardcoded idProduct (see its usb_descriptors.c)
# device tests
device_tests = [
'device/cdc_dual_ports',
'device/cdc_msc',
'device/dfu',
'device/cdc_msc_throughput',
'device/audio_test_freertos',
'device/dfu_runtime',
'device/cdc_msc_freertos',
'device/hid_boot_interface',
'device/msc_dual_lun',
'device/hid_generic_inout',
'device/printer_to_cdc',
'device/midi_test',
'device/mtp',
'device/usbtest', # cafe:4010, unique PID; runs the Linux testusb tier-4 battery via usbtest.py
# 'device/net_lwip_webserver', # disabled for PR #3605: USB net iface enum is flaky on the CI HIL host
]
dual_tests = [
'dual/host_info_to_device_cdc',
]
host_test = [
'host/cdc_msc_hid',
'host/msc_file_explorer',
'host/msc_file_explorer_freertos',
'host/device_info',
]

520
test/hil/hil_select.py Executable file
View File

@ -0,0 +1,520 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: MIT
"""PR-diff -> HIL selection: which rig boards and which tests a change can affect.
Stdlib-only (runs on bare CI runners; never imports hil_test/hil_flash/hil_lock).
Fail-open: any file no rule classifies forces the full matrix. See
docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md.
JSON: full, boards (name -> 'all' | [tests]), families (bsp families the diff
touches, including ones with no rig board - build-only consumers such as /pre-pr
sample from these), args (hil_test.py args per config) and args_flasher (the same
args split by each board's flasher, for CI legs that split one rig by flasher).
"""
import argparse
import functools
import glob
import json
import os
import re
import subprocess
import sys
from hil_examples import device_tests, dual_tests, host_test
ALL_TESTS = {'device': device_tests, 'dual': dual_tests, 'host': host_test}
# class dir -> config macro suffix exceptions (rule 3); dfu is per-file, handled inline
NET_MACROS = ('ECM_RNDIS', 'NCM')
_NONCODE_RE = re.compile(
r'^(docs/|\.claude/|.*\.(md|rst)$|LICENSE)')
_FULL_RE = re.compile(
r'^(src/common/|src/osal/|src/tusb\.c$|src/tusb\.h$|src/tusb_option\.h$|'
r'test/hil/|\.github/workflows/build.*\.yml$|\.github/actions/|'
r'tools/build\.py$|tools/get_deps\.py$|tools/cmake/|hw/mcu/|lib/|'
r'hw/bsp/(family_support\.cmake|board_api\.h|board\.c|ansi_escape\.h)$|'
r'examples/build_system/|examples/CMakeLists\.txt$|'
# board_test is HIL infrastructure, not a test: hil_test.py flashes it to park
# every board (variant boundary + end-of-board teardown), so every board depends on it
r'examples/device/board_test/)')
# --no-renames: with rename detection git reports only a rename's destination, so code
# moved out of an HIL-relevant path would be classified by its new path alone
GIT_DIFF_ARGV = ['git', 'diff', '--no-renames', '--name-only']
def test_role(test: str) -> str:
return test.split('/', 1)[0] # 'device' | 'dual' | 'host'
def board_roles(board: dict) -> set:
t = board.get('tests', {})
roles = set()
if t.get('device'):
roles.add('device')
if t.get('host'):
roles.add('host')
if t.get('dual'):
roles.update(('device', 'host'))
for only in t.get('only', []):
r = test_role(only)
roles.update(('device', 'host') if r == 'dual' else (r,))
return roles
def board_tests(board: dict) -> list:
"""Every test this board would run today (mirrors hil_test.test_board's default)."""
t = board.get('tests', {})
if 'only' in t:
run = list(t['only'])
else:
run = []
if t.get('device'):
run += device_tests
if t.get('dual'):
run += dual_tests
if t.get('host'):
run += host_test
return [x for x in run if x not in t.get('skip', [])]
# cached: called per changed file x roster board, and the tree doesn't change mid-run
@functools.lru_cache(maxsize=None)
def board_family(board_name: str, repo_root: str):
hits = glob.glob(os.path.join(repo_root, 'hw/bsp/*/boards', board_name))
return os.path.basename(os.path.dirname(os.path.dirname(hits[0]))) if hits else None
# `if (OPTION STREQUAL "1")` guards in family_support.cmake, and the option tokens
# a roster entry passes to the build (NAME=VALUE / -DNAME=VALUE)
_CM_IF_RE = re.compile(r'if\s*\(')
_CM_ELSE_RE = re.compile(r'else(if)?\s*\(')
_CM_ENDIF_RE = re.compile(r'endif\s*\(')
_CM_OPT_RE = re.compile(r'if\s*\(\s*\$?\{?([A-Za-z_]\w*)\}?\s+STREQUAL\s+"?1"?\s*\)')
_CM_PORT_RE = re.compile(r'src/portable/((?:[^/\s]+/)?[^/\s]+)/')
_FALSY = ('', '0', 'off', 'false', 'no')
@functools.lru_cache(maxsize=None)
def port_option_gates(repo_root: str) -> dict:
"""port dir -> build options that compile it regardless of the board's family
file, e.g. {'analog/max3421': {'MAX3421_HOST'}} from family_support.cmake."""
gates = {}
try:
text = open(os.path.join(repo_root, 'hw/bsp/family_support.cmake')).read()
except OSError:
return gates
stack = [] # one entry per open if(): its option, or None
for line in text.splitlines():
line = line.strip()
if _CM_IF_RE.match(line):
m = _CM_OPT_RE.match(line)
stack.append(m.group(1) if m else None)
elif _CM_ELSE_RE.match(line):
if stack:
stack[-1] = None # the guard doesn't hold in this branch
elif _CM_ENDIF_RE.match(line):
if stack:
stack.pop()
opts = {o for o in stack if o}
m = _CM_PORT_RE.search(line)
if opts and m:
gates.setdefault(m.group(1), set()).update(opts)
return gates
_CM_SET_RE = re.compile(r'set\s*\(\s*([A-Za-z_]\w*)\s+([^)\s]+)\s*\)')
# cached: called per changed portable file x roster board
@functools.lru_cache(maxsize=None)
def bsp_board_options(board_name: str, repo_root: str) -> frozenset:
"""Build options a board turns on in its own BSP: `set(<OPT> <value>)` in
hw/bsp/<family>/boards/<board>/board.cmake, e.g. MAX3421_HOST on the espressif
and rp2040 max3421 boards. CMake only - HIL CI builds nothing with Make, so a
board.mk-only option (e.g. nrf5340dk's MAX3421_HOST) compiles no port here."""
fam = board_family(board_name, repo_root)
if not fam:
return frozenset()
path = os.path.join(repo_root, 'hw/bsp', fam, 'boards', board_name, 'board.cmake')
try:
text = open(path).read()
except OSError:
return frozenset()
out = set()
for line in text.splitlines():
line = line.strip()
if line.startswith('#'):
continue
m = _CM_SET_RE.match(line)
if m and m.group(2).strip('"').lower() not in _FALSY:
out.add(m.group(1))
return frozenset(out)
def board_options(board: dict, repo_root: str) -> set:
"""Build options a board has truthy: the roster entry's build.args plus each
variant's defines (NAME=VALUE) and raw CFLAGS (-DNAME=VALUE), plus whatever its
own board.cmake sets (a board can enable a gated port without the roster saying so)."""
toks = list(board.get('build', {}).get('args', []))
for v in board.get('variant', []):
toks += list(v.get('defines', []))
toks += v.get('flags', '').split()
out = set(bsp_board_options(board['name'], repo_root))
for t in toks:
name, _, val = (t[2:] if t.startswith('-D') else t).partition('=')
if name and val.strip().strip('"').lower() not in _FALSY:
out.add(name.strip())
return out
@functools.lru_cache(maxsize=None)
def port_families(port_dir: str, repo_root: str) -> set:
"""Board families that compile this src/portable dir. CMake only: HIL CI builds
every board with CMake, so a port wired up in family.mk alone is compiled for no
HIL board and must not select one. family.cmake lists portable sources directly
for most families; espressif instead references them from a nested component
CMakeLists.txt (hw/bsp/espressif/components/tinyusb_src/CMakeLists.txt)."""
fams = set()
bsp_root = os.path.join(repo_root, 'hw/bsp')
# trailing '/' so a port dir is not a prefix of a sibling: bare 'microchip/pic'
# would otherwise match '.../microchip/pic32mz/...' and inherit its families
needle = port_dir + '/'
for f in glob.glob(os.path.join(bsp_root, '*/family.cmake')) + \
glob.glob(os.path.join(bsp_root, '*/components/*/CMakeLists.txt')):
try:
if needle in open(f).read():
fam = os.path.relpath(f, bsp_root).split(os.sep, 1)[0]
fams.add(fam)
except OSError:
pass
return fams
_CLS_INC_RE = re.compile(r'#\s*include\s*[<"]class/([^/"<>]+)/([^"<>]+)[">]')
@functools.lru_cache(maxsize=None)
def class_include_edges(repo_root: str) -> dict:
"""'<class>/<header>' -> the other class dirs that include it. A class header
pulled in by a second class ships in every firmware enabling that second class:
src/class/midi/midi{,2}_{device,host}.h include class/audio/audio.h, and
net_device.h includes class/cdc/cdc.h. The class rule derives macros from the
directory name alone, so without this edge a change to the included header
selects only its own class's examples - and on a board that skips those (e.g.
metro_m4_express skips audio_test_freertos), nothing at all.
Derived from the actual #include lines rather than a hand-written table so it
cannot rot when a class picks up or drops a cross-class include."""
edges = {}
for f in sorted(glob.glob(os.path.join(repo_root, 'src/class/*/*.[ch]'))):
cls = os.path.basename(os.path.dirname(f))
try:
text = open(f).read()
except OSError:
continue
for inc_cls, inc_hdr in _CLS_INC_RE.findall(text):
if inc_cls != cls:
edges.setdefault(f'{inc_cls}/{inc_hdr}', set()).add(cls)
return edges
def class_macros(cls: str, base: str, prefix: str) -> list:
"""Config macros that compile a class dir's code, for role prefix TUD/TUH.
`base` refines dfu only (it splits DFU from DFU_RUNTIME per file); pass '' for
a class reached through an include edge, where the widest set is correct."""
if cls == 'net':
return [f'CFG_{prefix}_{m}' for m in NET_MACROS]
if cls == 'dfu':
if base.startswith('dfu_rt'):
return [f'CFG_{prefix}_DFU_RUNTIME']
if base.startswith('dfu_device') or base.startswith('dfu_host'):
return [f'CFG_{prefix}_DFU']
return [f'CFG_{prefix}_DFU', f'CFG_{prefix}_DFU_RUNTIME']
return [f'CFG_{prefix}_{cls.upper()}']
def _config_enables(cfg_path: str, macros) -> bool:
try:
text = open(cfg_path).read()
except OSError:
return False
return any(re.search(rf'#define\s+{m}\s+\(?\s*0*[1-9]', text) for m in macros)
def roster_only_tests(all_boards) -> set:
"""Test paths that only appear in a roster board's tests.only list (e.g.
espressif boards), not in the shared device/dual/host_test lists."""
out = set()
for b in all_boards:
out.update(b.get('tests', {}).get('only', []))
return out
def class_examples(macros, role: str, repo_root: str, extra_tests: set) -> set:
"""Tests (from role's + dual lists, plus roster-only-list tests of that role)
whose example config enables any macro."""
pool = role_tests({role}, extra_tests)
out = set()
for test in pool:
cfg = os.path.join(repo_root, 'examples', test, 'src', 'tusb_config.h')
if _config_enables(cfg, macros):
out.add(test)
return out
def role_tests(roles: set, extras: set) -> set:
"""Every test for the given role(s): each role's own list + dual tests,
plus roster-only-list tests (extras) matching those roles or 'dual'."""
pool = set(dual_tests)
for r in roles:
pool |= set(ALL_TESTS[r])
pool |= {t for t in extras if test_role(t) in roles or test_role(t) == 'dual'}
return pool
class _Sel:
"""Accumulates contributions. board->set(tests) plus 'all-board' markers."""
def __init__(self):
self.full = False
self.by_board = {} # name -> set of tests, or 'all'
self.roles = set() # roles touched by any contribution
self.families = set() # bsp families touched (incl. off-rig ones: build-only consumers)
self.reasons = []
def add(self, boards, tests, reason):
"""tests: 'all' or iterable of test paths."""
self.reasons.append(reason)
for b in boards:
cur = self.by_board.get(b)
if tests == 'all' or cur == 'all':
self.by_board[b] = 'all'
else:
self.by_board[b] = (cur or set()) | set(tests)
def force_full(self, reason):
self.full = True
self.reasons.append(reason)
def _classify_one(path, repo_root, roster_boards, extras: set, s: _Sel):
base = os.path.basename(path)
if _NONCODE_RE.match(path):
s.reasons.append(f'{path}: non-code, no contribution')
return
if _FULL_RE.match(path):
s.force_full(f'{path}: core/infra -> full matrix')
return
m = re.match(r'src/portable/((?:[^/]+/)?[^/]+)/', path)
if m:
port = m.group(1)
if re.match(r'(dcd_|.*_device)', base):
roles = {'device'}
elif re.match(r'(hcd_|.*_host)', base):
roles = {'host'}
else:
roles = {'device', 'host'}
fams = port_families(port, repo_root)
if not fams:
# no family references this port: either a new/renamed port dir or a
# family.cmake layout the scan misses - widen instead of contributing nothing
s.force_full(f'{path}: port {port} maps to no board family -> full matrix')
return
s.families.update(fams)
# a board can also pull the port in through a build option (e.g. MAX3421_HOST=1
# from the roster on metro_m4_express, or from its own board.cmake), which its
# family file never names
gates = port_option_gates(repo_root).get(port, set())
boards = [b['name'] for b in roster_boards
if (board_family(b['name'], repo_root) in fams or
(gates and board_options(b, repo_root) & gates)) and (board_roles(b) & roles)]
tests = role_tests(roles, extras)
s.roles.update(roles)
why = f'{path}: port {port} -> families {sorted(fams)}'
if gates:
why += f' + option {sorted(gates)}'
s.add(boards, tests, f'{why} -> boards {boards} ({"/".join(sorted(roles))})')
return
m = re.match(r'src/class/([^/]+)/', path)
if m:
cls = m.group(1)
if re.search(r'_device\.[ch]$', base):
roles = {'device'}
elif re.search(r'_host\.[ch]$', base):
roles = {'host'}
else:
roles = {'device', 'host'}
# this file's own class, plus any class whose headers include it
via = sorted(class_include_edges(repo_root).get(f'{cls}/{base}', ()))
def macros(prefix):
return (class_macros(cls, base, prefix) +
[m2 for c in via for m2 in class_macros(c, '', prefix)])
tests = set()
if 'device' in roles:
tests |= class_examples(macros('TUD'), 'device', repo_root, extras)
if 'host' in roles:
tests |= class_examples(macros('TUH'), 'host', repo_root, extras)
boards = [b['name'] for b in roster_boards if board_roles(b) & roles]
s.roles.update(roles)
why = f'{path}: class {cls}' + (f' (+ included by {via})' if via else '')
s.add(boards, tests, f'{why} -> {sorted(tests)} ({"/".join(sorted(roles))})')
return
m = re.match(r'src/(device|host)/', path)
if m:
role = m.group(1)
boards = [b['name'] for b in roster_boards if role in board_roles(b)]
s.roles.add(role)
s.add(boards, role_tests({role}, extras), f'{path}: core {role} stack -> all {role} tests')
return
m = re.match(r'hw/bsp/([^/]+)/(?:boards/([^/]+)/)?', path)
if m:
fam, brd = m.group(1), m.group(2)
s.families.add(fam)
if brd:
boards = [b['name'] for b in roster_boards if b['name'] == brd]
why = f'{path}: bsp board {brd}'
else:
boards = [b['name'] for b in roster_boards
if board_family(b['name'], repo_root) == fam]
why = f'{path}: bsp family {fam}'
s.roles.update(('device', 'host'))
s.add(boards, 'all', f'{why} -> boards {boards}')
return
m = re.match(r'examples/(device|host|dual)/([^/]+)/', path)
if m:
test = f'{m.group(1)}/{m.group(2)}'
known = any(test in pool for pool in ALL_TESTS.values()) or test in extras
if known:
boards = [b['name'] for b in roster_boards]
role = test_role(test)
s.roles.update(('device', 'host') if role == 'dual' else (role,))
s.add(boards, [test], f'{path}: example -> {test} on all boards')
else:
s.reasons.append(f'{path}: example not in HIL lists, no contribution')
return
s.force_full(f'{path}: unclassified -> full matrix')
def classify(changed_files, repo_root, rosters):
all_boards = []
seen = set()
for _, boards in rosters:
for b in boards:
if b['name'] not in seen:
seen.add(b['name'])
all_boards.append(b)
extras = roster_only_tests(all_boards)
s = _Sel()
# no early exit once full: keep classifying so `families` still reports every
# family the diff touches (build-only consumers need it). Nothing after the first
# force_full can change full/boards/args - the full branch below ignores by_board.
for path in changed_files:
_classify_one(path, repo_root, all_boards, extras, s)
if s.full:
return {'full': True, 'boards': {b['name']: 'all' for b in all_boards},
'families': sorted(s.families), 'reasons': s.reasons}
# role pruning: single-role selections drop the other role's tests and boards
by_name = {b['name']: b for b in all_boards}
out = {}
for name, tests in s.by_board.items():
allowed = board_tests(by_name[name])
if tests == 'all':
kept = list(allowed)
else:
kept = [t for t in allowed if t in tests]
if s.roles and s.roles != {'device', 'host'}:
role = next(iter(s.roles))
kept = [t for t in kept if test_role(t) in (role, 'dual')]
if kept:
out[name] = 'all' if set(kept) == set(allowed) else sorted(kept)
return {'full': False, 'boards': out, 'families': sorted(s.families),
'reasons': s.reasons}
def _board_args(name, chosen) -> list:
parts = [f'-b {name}']
if chosen != 'all':
parts.append(f'-bt {name}:{",".join(chosen)}')
return parts
def selection_args(sel, rosters):
"""hil_test.py args per config. Empty means either 'full matrix' or 'nothing
selected' - callers must read sel['full'] to tell them apart."""
args = {}
for cfg_path, boards in rosters:
parts = []
if not sel['full']:
for b in boards:
chosen = sel['boards'].get(b['name'])
if chosen is not None:
parts += _board_args(b['name'], chosen)
args[os.path.basename(cfg_path)] = ' '.join(parts)
return args
def selection_args_by_flasher(sel, rosters):
"""{config: {flasher name: args}}. CI runs one rig as several jobs split by
flasher (esptool vs the rest); each must gate on its own subset, otherwise the
other leg runs a filter matching zero boards and reports a vacuous green."""
out = {}
for cfg_path, boards in rosters:
per = {}
if not sel['full']:
for b in boards:
chosen = sel['boards'].get(b['name'])
if chosen is None:
continue
per.setdefault(b.get('flasher', {}).get('name', ''), []).extend(
_board_args(b['name'], chosen))
out[os.path.basename(cfg_path)] = {f: ' '.join(p) for f, p in per.items()}
return out
def changed_files_from_git(base, repo_root):
mb = subprocess.run(['git', 'merge-base', 'HEAD', base], cwd=repo_root,
capture_output=True, text=True, check=True).stdout.strip()
diff = subprocess.run(GIT_DIFF_ARGV + [f'{mb}..HEAD'], cwd=repo_root,
capture_output=True, text=True, check=True).stdout
return [l for l in diff.splitlines() if l.strip()]
def main():
ap = argparse.ArgumentParser(description=__doc__)
g = ap.add_mutually_exclusive_group(required=True)
g.add_argument('--base', help='git ref to diff against (merge-base..HEAD)')
g.add_argument('--diff-file', help='newline-separated changed-file list')
ap.add_argument('configs', nargs='+', help='rig roster JSON file(s)')
a = ap.parse_args()
repo_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
rosters = []
for c in a.configs:
with open(c) as f:
rosters.append((c, json.load(f)['boards']))
files = (open(a.diff_file).read().splitlines() if a.diff_file
else changed_files_from_git(a.base, repo_root))
files = [f for f in files if f.strip()]
s = classify(files, repo_root, rosters)
s['args'] = selection_args(s, rosters)
s['args_flasher'] = selection_args_by_flasher(s, rosters)
for r in s['reasons']:
print(f'hil_select: {r}', file=sys.stderr)
print(json.dumps(s))
if __name__ == '__main__':
main()

102
test/hil/hil_test.py
View File

@ -59,6 +59,7 @@ from multiprocessing import TimeoutError as MpTimeoutError
import hil_flash
import hil_lock
from hil_examples import device_tests, dual_tests, host_test
# Raw Lock/Semaphore objects passed via Pool initargs are inheritable only under the fork
# start method (spawn/forkserver pickle them and fail at Pool creation) — pin it so a
@ -1351,39 +1352,6 @@ def test_device_usbtest(board):
# Main
# -------------------------------------------------------------
# The per-board run order is shuffled (see test_board).
# Every example carries a unique hardcoded idProduct (see its usb_descriptors.c)
# device tests
device_tests = [
'device/cdc_dual_ports',
'device/cdc_msc',
'device/dfu',
'device/cdc_msc_throughput',
'device/audio_test_freertos',
'device/dfu_runtime',
'device/cdc_msc_freertos',
'device/hid_boot_interface',
'device/msc_dual_lun',
'device/hid_generic_inout',
'device/printer_to_cdc',
'device/midi_test',
'device/mtp',
'device/usbtest', # cafe:4010, unique PID; runs the Linux testusb tier-4 battery via usbtest.py
# 'device/net_lwip_webserver', # disabled for PR #3605: USB net iface enum is flaky on the CI HIL host
]
dual_tests = [
'dual/host_info_to_device_cdc',
]
host_test = [
'host/cdc_msc_hid',
'host/msc_file_explorer',
'host/msc_file_explorer_freertos',
'host/device_info',
]
def test_example(board: Board, variant: str, example: str) -> tuple[int, str, str | None]:
"""
@ -1517,6 +1485,10 @@ def build_board(board: Board) -> tuple[str, int]:
return name, failed
# pseudo-test column for a variant boundary the park-flash could not clear (see below)
BOUNDARY_CELL = 'same-PID boundary'
def test_board(board: Board) -> tuple[str, int, list[str], list, float]:
name = board['name']
flasher = board['flasher']
@ -1568,6 +1540,7 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]:
err_count = 0
failed_tests = []
board_wide_fail = False # re-run the whole board, not a subset of its tests
rows = [] # list of (row_label, {example: status}, duration) — one row per build variant
# a -t/-bt filtered run times only a subset; report no duration so an accumulate
# re-run keeps the previous full-run value
@ -1587,10 +1560,36 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]:
random.Random(f'{shuffle_seed}:{name}:{vname}').shuffle(run_list)
if run_list[0] == prev_last:
run_list[0], run_list[-1] = run_list[-1], run_list[0]
cells = {}
if run_list and run_list[0] == prev_last and not skip_flash:
# Same example (same PID) still repeats across the boundary: a one-test
# list (the common case for a -bt scoped run) leaves nothing to swap
# with. Park on board_test first - it disables the board's USB, so the
# PID goes away and the next flash must re-enumerate to be seen.
t_park = time.monotonic()
park_ec, park_status, _ = test_example(board, vname, 'device/board_test')
if park_ec or park_status == 'skip':
# Boundary not cleared: the previous variant's device may still be
# enumerated under the same PID, so this variant's tests could pass
# against its firmware. Skip them - a false green proves nothing and
# is worse than a gap - and record the boundary itself as the failure
# (a visible ❌ cell, mirroring the board-lock row above) so the report
# matches the exit code instead of rendering all-green.
why = 'no board_test binary' if park_status == 'skip' else 'park flash failed'
log_line(f'{vname:40} {"same-PID boundary":30} {STATUS_FAILED}: not cleared ({why}); '
f'skipping {len(run_list)} test(s) on this variant')
err_count += 1
cells[BOUNDARY_CELL] = 'fail'
# blaming run_list[0] would re-run an innocent test that then passes,
# leaving the boundary unretested; re-run the whole board instead
board_wide_fail = True
# leave prev_last alone: the board still holds the previous variant's
# firmware, so the next variant must attempt the park again
run_list = []
t_board += time.monotonic() - t_park # park is teardown, not board cost
if run_list:
prev_last = run_list[-1]
t_variant = time.monotonic()
cells = {}
for test in run_list:
ec, status, metric = test_example(board, vname, test)
err_count += ec
@ -1609,7 +1608,7 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]:
if not skip_flash:
test_example(board, variants[0]['name'], 'device/board_test')
return name, err_count, sorted(set(failed_tests)), rows, t_total
return name, err_count, [] if board_wide_fail else sorted(set(failed_tests)), rows, t_total
finally:
if _lock_fh:
try:
@ -1704,11 +1703,13 @@ def render_matrix(rows_all: list) -> str:
return summary + '\n\n' + '\n'.join([header, sep] + body)
def accumulate_report(mret: list, report_dir: Path, fresh: bool) -> str:
def accumulate_report(mret: list, report_dir: Path, fresh: bool, scope: str = '') -> str:
"""Merge this run's results into hil_report.json in report_dir, then (re)write
the markdown matrix to hil_report.md. `fresh` (a full run, no --accumulate/-bt)
the markdown matrix to hil_report.md. `fresh` (a first run, no --accumulate)
starts a new report; otherwise a re-run accumulates so boards/tests that
already passed are preserved while re-run cells are updated. Returns the md."""
already passed are preserved while re-run cells are updated. `scope` names the
board filter, if any, so a scoped table is not mistaken for a full one.
Returns the md."""
acc = {} # ordered {row_label: [cells dict, duration str|None]}
jpath = report_dir / REPORT_JSON
if not fresh and jpath.is_file():
@ -1736,6 +1737,10 @@ def accumulate_report(mret: list, report_dir: Path, fresh: bool) -> str:
del acc[name]
for row_label, cells, dur in rows:
row = acc.setdefault(row_label, [{}, None])
# the boundary cell is only ever written on failure, so a re-run of this
# variant that cleared the boundary must drop the previous attempt's ❌
if BOUNDARY_CELL not in cells:
row[0].pop(BOUNDARY_CELL, None)
row[0].update(cells)
if dur is not None:
row[1] = dur
@ -1745,6 +1750,10 @@ def accumulate_report(mret: list, report_dir: Path, fresh: bool) -> str:
for k, (c, d) in acc.items()]}, indent=2) + '\n')
md = render_matrix([(k, c, d) for k, (c, d) in acc.items()])
if scope:
# a scoped run's small table is otherwise indistinguishable from a full one,
# and it replaces the previous full table in the sticky PR comment
md = f'_Scoped run: {scope}. Boards/tests not listed were not run._\n\n' + md
(report_dir / REPORT_MD).write_text(md + '\n', encoding='utf-8')
return md
@ -1831,13 +1840,14 @@ def main() -> None:
# HIL report sidecar (hil_report.json/.md) and the .failed re-run spec live in
# report_dir (CI keys it by run id, so it persists across run attempts but is
# private to one run). A full run starts fresh; a re-run (--accumulate / -bt,
# i.e. the .failed file) merges so already-passed boards/tests are preserved.
# Clear prior state up front on a fresh run so a crash mid-run can't leave a
# stale report or re-run spec to be consumed by a retry.
# private to one run). A full run starts fresh; a re-run (--accumulate, which
# the generated .failed spec always starts with) merges so already-passed
# boards/tests are preserved. Clear prior state up front on a fresh run so a
# crash mid-run can't leave a stale report or re-run spec for a retry.
# -bt alone is not a re-run marker: PR-scoped first attempts pass -bt too.
report_dir = Path(os.environ.get('HIL_REPORT_DIR', '.'))
failed_fname = report_dir / (config_file.name + '.failed')
fresh = not (args.accumulate or args.board_test)
fresh = not args.accumulate
if fresh:
report_dir.mkdir(parents=True, exist_ok=True)
for f in (REPORT_JSON, REPORT_MD):
@ -1935,7 +1945,11 @@ def main() -> None:
print(f'warning: cannot persist controller hints to {CONTROLLER_CACHE}: {e}')
# board x test result matrix -> hil_report.md (accumulates across re-runs) + stdout
report = accumulate_report(mret, report_dir, fresh)
# -b/-bt in play means a filtered run (PR selection or a re-run spec): say so in the
# report, which otherwise looks exactly like a full run that happened to be small
scoped = sorted(set(args.board) | set(board_test))
scope = f'{len(scoped)} board(s) — {", ".join(scoped)}' if scoped else ''
report = accumulate_report(mret, report_dir, fresh, scope)
print()
print(report)
print(f'\nReport written to {(report_dir / REPORT_MD).resolve()}')

542
test/hil/test_hil_select.py Normal file
View File

@ -0,0 +1,542 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: MIT
# Unit tests for hil_select.py — pure logic, no hardware, no git. Run directly:
# python3 test/hil/test_hil_select.py
import glob
import json
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import hil_select
from hil_examples import device_tests, dual_tests, host_test
REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
def real_rosters():
"""The actual rig rosters, for regression tests that need real-world data
(a specific board/family/only-list) rather than the synthetic ROSTER above."""
rosters = []
for name in ('tinyusb.json', 'hfp.json'):
path = os.path.join(REPO, 'test/hil', name)
with open(path) as f:
rosters.append((f'test/hil/{name}', json.load(f)['boards']))
return rosters
def on_roster(tc, *names):
"""The subset of `names` currently in the live rig rosters, skipping the test
when none are. Parking/unparking a board is routine rig maintenance and must not
fail this suite: CI runs it right before the selector and treats a failure as
'selector unusable', dropping PR scoping and annotating the run."""
have = {b['name'] for _, boards in real_rosters() for b in boards}
got = [n for n in names if n in have]
if not got:
tc.skipTest(f'not in the rig roster: {", ".join(names)}')
return got
ROSTER = [
# device-only, rp2040 family
{'name': 'raspberry_pi_pico', 'uid': 'u1', 'flasher': {'name': 'openocd'},
'tests': {'device': True, 'host': True, 'dual': True}},
# device-only, stm32f4 family
{'name': 'stm32f407disco', 'uid': 'u2', 'flasher': {'name': 'jlink'},
'tests': {'device': True, 'host': False, 'dual': False}},
# host-only board
{'name': 'raspberry_pi_pico2', 'uid': 'u3', 'flasher': {'name': 'openocd'},
'tests': {'device': False, 'host': True, 'dual': False}},
# only-list board (espressif-style), flashed by the CI leg that splits on esptool
{'name': 'espressif_s3_devkitm', 'uid': 'u4', 'flasher': {'name': 'esptool'},
'tests': {'only': ['device/cdc_msc_freertos', 'host/device_info']}},
]
ROSTERS = [('test/hil/tinyusb.json', ROSTER)]
def sel(files):
return hil_select.classify(files, REPO, ROSTERS)
class TestPortRule(unittest.TestCase):
def test_dcd_rp2040_selects_pico_family_only(self):
s = sel(['src/portable/raspberrypi/rp2040/dcd_rp2040.c'])
self.assertFalse(s['full'])
self.assertIn('raspberry_pi_pico', s['boards'])
self.assertNotIn('stm32f407disco', s['boards'])
self.assertNotIn('espressif_s3_devkitm', s['boards'])
# device role: no host tests in pico's list
self.assertTrue(all(not t.startswith('host/') for t in s['boards']['raspberry_pi_pico']))
# host-only boards drop out entirely on a device-role change
self.assertNotIn('raspberry_pi_pico2', s['boards'])
def test_shared_port_file_is_both_roles(self):
s = sel(['src/portable/synopsys/dwc2/dwc2_common.c'])
self.assertFalse(s['full'])
self.assertNotIn('raspberry_pi_pico', s['boards']) # rp2040 is not a dwc2 family
self.assertIn('stm32f407disco', s['boards']) # stm32f4 is
class TestCoreRoleRule(unittest.TestCase):
def test_usbd_selects_all_device_tests_everywhere(self):
s = sel(['src/device/usbd.c'])
self.assertFalse(s['full'])
self.assertNotIn('raspberry_pi_pico2', s['boards']) # host-only board dropped
pico = s['boards']['raspberry_pi_pico']
self.assertTrue(set(device_tests).issubset(set(pico)))
self.assertTrue(set(dual_tests).issubset(set(pico))) # dual survives device role
self.assertTrue(all(not t.startswith('host/') for t in pico))
# only-list board: selection intersects its only-list
esp = s['boards']['espressif_s3_devkitm']
self.assertEqual(esp, ['device/cdc_msc_freertos'])
def test_host_change_drops_device(self):
s = sel(['src/host/usbh.c'])
self.assertFalse(s['full'])
self.assertIn('raspberry_pi_pico2', s['boards'])
self.assertNotIn('stm32f407disco', s['boards']) # device-only board dropped
class TestClassRule(unittest.TestCase):
def test_cdc_device_selects_cdc_examples_only(self):
s = sel(['src/class/cdc/cdc_device.c'])
self.assertFalse(s['full'])
pico = s['boards']['raspberry_pi_pico']
self.assertIn('device/cdc_msc', pico)
self.assertIn('device/cdc_dual_ports', pico)
self.assertNotIn('device/msc_dual_lun', pico) # CFG_TUD_CDC 0 there
self.assertNotIn('device/usbtest', pico) # CFG_TUD_CDC 0 there
self.assertTrue(all(not t.startswith('host/') for t in pico))
def test_msc_host_selects_host_side(self):
s = sel(['src/class/msc/msc_host.c'])
self.assertFalse(s['full'])
self.assertNotIn('stm32f407disco', s['boards']) # device-only board
pico2 = s['boards']['raspberry_pi_pico2']
self.assertIn('host/msc_file_explorer', pico2)
self.assertTrue(all(not t.startswith('device/') for t in pico2))
class TestClassIncludeEdges(unittest.TestCase):
"""A class header another class includes reaches that class's examples too.
src/class/midi/midi{,2}_{device,host}.h include class/audio/audio.h, so
midi_test's firmware contains audio.h - but the class rule derives macros from
the directory name alone, so an audio.h change used to select only
device/audio_test_freertos. On boards that skip that example the per-board
intersection emptied and an audio.h-only PR ran ZERO HIL on them."""
def test_edges_derived_from_includes(self):
edges = hil_select.class_include_edges(REPO)
self.assertEqual(edges.get('audio/audio.h'), {'midi'})
self.assertEqual(edges.get('cdc/cdc.h'), {'net'})
def test_audio_header_selects_midi_example(self):
s = hil_select.classify(['src/class/audio/audio.h'], REPO, real_rosters())
self.assertFalse(s['full'])
# every board that runs device/midi_test at all must run it here (boards with
# a tests.only list, e.g. espressif, run the freertos examples instead)
by_name = {b['name']: b for _, bs in real_rosters() for b in bs}
checked = 0
for name, tests in s['boards'].items():
if 'device/midi_test' in hil_select.board_tests(by_name[name]):
self.assertIn('device/midi_test', tests, name)
checked += 1
self.assertTrue(checked)
def test_audio_header_reaches_boards_that_skip_audio(self):
# both skip device/audio_test_freertos: without the midi edge their
# intersection is empty and they drop out of the selection entirely
boards = on_roster(self, 'metro_m4_express', 'nrf54lm20dk')
s = hil_select.classify(['src/class/audio/audio.h'], REPO, real_rosters())
for board in boards:
self.assertEqual(s['boards'].get(board), ['device/midi_test'], board)
def test_edge_is_per_header_not_per_class(self):
# midi includes audio.h, not audio_device.h: an audio_device change must
# not drag midi's examples in
s = hil_select.classify(['src/class/audio/audio_device.c'], REPO, real_rosters())
self.assertFalse(s['full'])
for tests in s['boards'].values():
if tests != 'all':
self.assertNotIn('device/midi_test', tests)
class TestFallbackRules(unittest.TestCase):
def test_unknown_tool_is_full(self):
s = sel(['tools/random_new_script.py'])
self.assertTrue(s['full'])
def test_docs_only_is_empty_not_full(self):
s = sel(['docs/info/contributing.rst', 'README.rst'])
self.assertFalse(s['full'])
self.assertEqual(s['boards'], {})
def test_bsp_family_selects_family_boards(self):
s = sel(['hw/bsp/rp2040/family.cmake'])
self.assertFalse(s['full'])
self.assertIn('raspberry_pi_pico', s['boards'])
self.assertEqual(s['boards']['raspberry_pi_pico'], 'all')
self.assertNotIn('stm32f407disco', s['boards'])
def test_bsp_board_narrows_to_board(self):
s = sel(['hw/bsp/rp2040/boards/raspberry_pi_pico/board.h'])
self.assertFalse(s['full'])
self.assertEqual(list(s['boards'].keys()), ['raspberry_pi_pico'])
def test_example_change_selects_that_example(self):
s = sel(['examples/device/cdc_msc/src/main.c'])
self.assertFalse(s['full'])
self.assertEqual(s['boards']['raspberry_pi_pico'], ['device/cdc_msc'])
def test_core_common_is_full(self):
for f in ['src/tusb.c', 'src/common/tusb_fifo.c', 'src/osal/osal_freertos.h']:
self.assertTrue(sel([f])['full'], f)
def test_board_test_example_is_full(self):
# board_test is the park/teardown firmware hil_test.py flashes on every board,
# not an unlisted example: a regression there must not skip the whole rig
for f in ['examples/device/board_test/src/main.c',
'examples/device/board_test/CMakeLists.txt']:
self.assertTrue(sel([f])['full'], f)
def test_harness_is_full(self):
for f in ['test/hil/hil_test.py', '.github/workflows/build.yml', 'hw/mcu/nxp/x.c', 'lib/foo/x.c']:
self.assertTrue(sel([f])['full'], f)
def test_mixed_roles_no_pruning(self):
s = sel(['src/device/usbd.c', 'src/host/usbh.c'])
self.assertFalse(s['full'])
self.assertIn('raspberry_pi_pico2', s['boards'])
self.assertIn('stm32f407disco', s['boards'])
def test_cmakelists_and_requirements_are_full(self):
for f in ['src/CMakeLists.txt', 'examples/CMakeLists.txt',
'examples/device/CMakeLists.txt', 'test/hil/requirements.txt']:
self.assertTrue(sel([f])['full'], f)
def test_docs_txt_is_noncode(self):
s = sel(['docs/info/changelog.txt'])
self.assertFalse(s['full'])
self.assertEqual(s['boards'], {})
class TestArgsEmission(unittest.TestCase):
def test_args_for_scoped_selection(self):
s = sel(['src/portable/raspberrypi/rp2040/dcd_rp2040.c'])
args = hil_select.selection_args(s, ROSTERS)
a = args['tinyusb.json']
self.assertIn('-b raspberry_pi_pico', a)
self.assertNotIn('stm32f407disco', a)
self.assertIn('-bt raspberry_pi_pico:', a) # device-only subset of a device+host board
def test_args_full_is_empty(self):
s = sel(['tools/random_new_script.py'])
self.assertEqual(hil_select.selection_args(s, ROSTERS), {'tinyusb.json': ''})
def test_args_all_board_gets_bare_b(self):
s = sel(['hw/bsp/rp2040/boards/raspberry_pi_pico/board.h'])
a = hil_select.selection_args(s, ROSTERS)['tinyusb.json']
self.assertIn('-b raspberry_pi_pico', a)
self.assertNotIn('-bt', a)
def test_args_by_flasher_splits_esp_from_the_rest(self):
s = sel(['src/device/usbd.c'])
per = hil_select.selection_args_by_flasher(s, ROSTERS)['tinyusb.json']
self.assertIn('espressif_s3_devkitm', per['esptool'])
self.assertIn('raspberry_pi_pico', per['openocd'])
self.assertNotIn('espressif_s3_devkitm', per.get('openocd', '') + per.get('jlink', ''))
def test_args_by_flasher_omits_a_flasher_with_no_selected_board(self):
# the esp CI leg must see no args at all here, not a filter matching zero boards
s = sel(['hw/bsp/rp2040/boards/raspberry_pi_pico/board.h'])
per = hil_select.selection_args_by_flasher(s, ROSTERS)['tinyusb.json']
self.assertEqual(per, {'openocd': '-b raspberry_pi_pico'})
def test_args_by_flasher_full_is_empty(self):
s = sel(['tools/random_new_script.py'])
self.assertEqual(hil_select.selection_args_by_flasher(s, ROSTERS), {'tinyusb.json': {}})
def test_cli_diff_file(self):
import subprocess, tempfile, json as j
with tempfile.NamedTemporaryFile('w', suffix='.txt', delete=False) as f:
f.write('src/class/cdc/cdc_device.c\n')
path = f.name
r = subprocess.run([sys.executable, os.path.join(REPO, 'test/hil/hil_select.py'),
'--diff-file', path, os.path.join(REPO, 'test/hil/tinyusb.json')],
capture_output=True, text=True)
self.assertEqual(r.returncode, 0, r.stderr)
out = j.loads(r.stdout)
self.assertFalse(out['full'])
self.assertIn('tinyusb.json', out['args'])
self.assertTrue(any('cdc_device' in line for line in out['reasons']))
os.unlink(path)
class TestRealRosterPortFamilies(unittest.TestCase):
"""Regression for port_families() missing espressif's dwc2 reference, which
lives in a component CMakeLists.txt rather than family.cmake/family.mk."""
def test_dwc2_change_selects_espressif_boards(self):
boards = on_roster(self, 'espressif_s3_devkitm', 'espressif_p4_function_ev')
s = hil_select.classify(['src/portable/synopsys/dwc2/dcd_dwc2.c'], REPO, real_rosters())
self.assertFalse(s['full'])
for board in boards:
self.assertIn(board, s['boards'])
class TestOptionGatedPort(unittest.TestCase):
"""Regression: family_support.cmake compiles some ports from a build option
(MAX3421_HOST=1 -> hcd_max3421.c), so a board's family file never names them."""
# host-side option board (max3421 as host controller), off any max3421 family
OPT_ROSTER = [('test/hil/opt.json', [
{'name': 'fake_dual_board', 'uid': 'o1', 'flasher': {'name': 'jlink'},
'build': {'args': ['MAX3421_HOST=1']},
'tests': {'device': True, 'host': False, 'dual': True}},
{'name': 'fake_host_board', 'uid': 'o2', 'flasher': {'name': 'jlink'},
'variant': [{'name': 'fake_host_board', 'flags': '-DMAX3421_HOST=1'}],
'tests': {'device': False, 'host': True, 'dual': False}},
{'name': 'fake_off_board', 'uid': 'o3', 'flasher': {'name': 'jlink'},
'variant': [{'name': 'fake_off_board', 'defines': ['MAX3421_HOST=0']}],
'tests': {'device': True, 'host': True, 'dual': True}},
])]
def test_real_roster_max3421_selects_option_board(self):
boards = on_roster(self, 'metro_m4_express')
s = hil_select.classify(['src/portable/analog/max3421/hcd_max3421.c'], REPO, real_rosters())
self.assertFalse(s['full'])
for board in boards:
self.assertIn(board, s['boards'])
def test_option_selects_via_args_defines_and_flags(self):
s = hil_select.classify(['src/portable/analog/max3421/hcd_max3421.c'], REPO, self.OPT_ROSTER)
self.assertFalse(s['full'])
self.assertIn('fake_dual_board', s['boards']) # build.args
self.assertIn('fake_host_board', s['boards']) # variant flags
self.assertNotIn('fake_off_board', s['boards']) # variant defines, but =0
def test_device_role_port_does_not_pull_host_only_option_board(self):
s = hil_select.classify(['src/portable/analog/max3421/dcd_max3421.c'], REPO, self.OPT_ROSTER)
self.assertFalse(s['full'])
self.assertNotIn('fake_host_board', s['boards']) # host-only board, device change
self.assertIn('fake_dual_board', s['boards']) # device-capable option board
def test_gates_parsed_from_family_support(self):
self.assertEqual(hil_select.port_option_gates(REPO).get('analog/max3421'),
{'MAX3421_HOST'})
def test_board_cmake_option_counts(self):
"""A board can enable a gated port in its own BSP rather than via the roster
(hw/bsp/espressif/boards/*/board.cmake -> set(MAX3421_HOST 1)); board_options()
must see those too, or such a board joining the roster is silently dropped."""
self.assertIn('MAX3421_HOST',
hil_select.bsp_board_options('adafruit_feather_esp32s3', REPO))
self.assertIn('CFG_TUH_RPI_PIO_USB',
hil_select.bsp_board_options('adafruit_fruit_jam', REPO))
# commented-out `# set(MAX3421_HOST 1)` must not count
self.assertNotIn('MAX3421_HOST',
hil_select.bsp_board_options('feather_nrf52840_express', REPO))
def test_board_cmake_option_selects_off_family_board(self):
# adafruit_feather_esp32s3 is not on any rig roster; stand it in as one to
# prove the BSP-sourced option alone pulls a max3421 change onto the board
roster = [('test/hil/opt.json', [
{'name': 'adafruit_feather_esp32s3', 'uid': 'o1', 'flasher': {'name': 'esptool'},
'tests': {'device': False, 'host': True, 'dual': False}}])]
s = hil_select.classify(['src/portable/analog/max3421/hcd_max3421.c'], REPO, roster)
self.assertFalse(s['full'])
self.assertIn('adafruit_feather_esp32s3', s['boards'])
def test_board_mk_option_is_ignored(self):
"""Make-only options must not select: HIL CI builds with CMake exclusively, so
hw/bsp/nrf/boards/nrf5340dk/board.mk's MAX3421_HOST compiles nothing here."""
roster = [('test/hil/opt.json', [
{'name': 'nrf5340dk', 'uid': 'o1', 'flasher': {'name': 'jlink'},
'tests': {'device': False, 'host': True, 'dual': False}}])]
s = hil_select.classify(['src/portable/analog/max3421/hcd_max3421.c'], REPO, roster)
self.assertFalse(s['full'])
self.assertEqual(s['boards'], {})
class TestPortFamiliesCmakeOnly(unittest.TestCase):
"""port_families() is CMake-only (HIL CI never builds with Make) and matches on
'port_dir/' so a port dir is not a prefix of a sibling."""
def test_make_only_family_is_not_a_family(self):
# hw/bsp/pic32mz has family.mk but no family.cmake
self.assertEqual(hil_select.port_families('microchip/pic32mz', REPO), set())
def test_prefix_port_does_not_inherit_sibling_families(self):
# bare-substring matching let 'microchip/pic' match '.../microchip/pic32mz/...'
self.assertEqual(hil_select.port_families('microchip/pic', REPO), set())
def test_make_only_port_forces_full(self):
s = sel(['src/portable/microchip/pic32mz/dcd_pic32mz.c'])
self.assertTrue(s['full'])
self.assertTrue(any('no board family' in r for r in s['reasons']), s['reasons'])
def test_cmake_families_still_found(self):
self.assertEqual(hil_select.port_families('raspberrypi/rp2040', REPO), {'rp2040'})
self.assertIn('stm32f4', hil_select.port_families('synopsys/dwc2', REPO))
class TestPortFamiliesCoverage(unittest.TestCase):
"""Systematic guard: every real dcd_*/hcd_* port directory should map to at
least one board family, so a future family.cmake/CMakeLists.txt layout that
port_families() doesn't scan fails loudly instead of silently dropping boards
(as espressif's dwc2 reference did - see TestRealRosterPortFamilies)."""
# Ports with no board family: not a bug, just not wired into any rig board.
# Add here (with a reason) only if port_families() legitimately can't find one.
# A port listed here force-fulls (fail-open), so it is never under-selected.
NO_FAMILY = {
'template', # reference/example port, not built by any board
# hw/bsp/pic32mz has family.mk only (no family.cmake), and port_families()
# is CMake-only because HIL CI builds every board with CMake - so this port
# is compiled for no HIL board.
'microchip/pic32mz',
'microchip/pic', # same: only ever referenced from pic32mz's family.mk
}
@staticmethod
def _dcd_hcd_ports():
portable_root = os.path.join(REPO, 'src/portable')
ports = []
for entry in sorted(os.listdir(portable_root)):
d = os.path.join(portable_root, entry)
if not os.path.isdir(d):
continue
if glob.glob(os.path.join(d, 'dcd_*.c')) or glob.glob(os.path.join(d, 'hcd_*.c')):
ports.append(entry)
continue
for sub in sorted(os.listdir(d)):
sd = os.path.join(d, sub)
if os.path.isdir(sd) and (glob.glob(os.path.join(sd, 'dcd_*.c')) or
glob.glob(os.path.join(sd, 'hcd_*.c'))):
ports.append(f'{entry}/{sub}')
return ports
def test_every_port_maps_to_a_family(self):
ports = self._dcd_hcd_ports()
self.assertTrue(ports) # sanity: the scan itself found something
for port in ports:
if port in self.NO_FAMILY:
continue
fams = hil_select.port_families(port, REPO)
self.assertTrue(fams, f'{port}: no family references this port '
f'(port_families() scan gap, or add to NO_FAMILY)')
class TestRealRosterOnlyListTests(unittest.TestCase):
"""Regression for roster-only-list tests (e.g. espressif's hid_composite_freertos)
being invisible to the selector because it only knew the shared hil_examples lists."""
def test_only_list_example_change_selects_it(self):
boards = on_roster(self, 'espressif_s3_devkitm', 'espressif_p4_function_ev')
s = hil_select.classify(['examples/device/hid_composite_freertos/src/main.c'], REPO, real_rosters())
self.assertFalse(s['full'])
for board in boards:
self.assertEqual(s['boards'][board], ['device/hid_composite_freertos'])
def test_class_change_includes_only_list_boards(self):
boards = on_roster(self, 'espressif_s3_devkitm', 'espressif_p4_function_ev')
s = hil_select.classify(['src/class/hid/hid_device.c'], REPO, real_rosters())
self.assertFalse(s['full'])
for board in boards:
self.assertIn(board, s['boards'])
class TestPortAndCoreRoleUseExtras(unittest.TestCase):
"""Regression: the port rule and core-role rule must thread the roster-only
test universe (extras) the same way the class rule already does, so a DCD
or device-stack change doesn't silently drop espressif's only-list tests
(e.g. hid_composite_freertos) that aren't in the shared device_tests list."""
def test_dcd_change_includes_only_list_test(self):
boards = on_roster(self, 'espressif_s3_devkitm', 'espressif_p4_function_ev')
s = hil_select.classify(['src/portable/synopsys/dwc2/dcd_dwc2.c'], REPO, real_rosters())
self.assertFalse(s['full'])
for board in boards:
tests = s['boards'][board]
self.assertIn('device/hid_composite_freertos', tests)
self.assertIn('device/cdc_msc_freertos', tests)
self.assertIn('device/audio_test_freertos', tests)
self.assertIn('device/usbtest', tests)
def test_core_device_change_includes_only_list_test(self):
boards = on_roster(self, 'espressif_s3_devkitm', 'espressif_p4_function_ev')
s = hil_select.classify(['src/device/usbd.c'], REPO, real_rosters())
self.assertFalse(s['full'])
for board in boards:
tests = s['boards'][board]
self.assertIn('device/hid_composite_freertos', tests)
self.assertIn('device/cdc_msc_freertos', tests)
self.assertIn('device/audio_test_freertos', tests)
self.assertIn('device/usbtest', tests)
def test_host_change_does_not_leak_device_only_list_test(self):
s = hil_select.classify(['src/host/usbh.c'], REPO, real_rosters())
self.assertFalse(s['full'])
for board, tests in s['boards'].items():
if tests == 'all':
continue
self.assertNotIn('device/hid_composite_freertos', tests, board)
class TestFamilies(unittest.TestCase):
"""`families` exists for consumers that build (not just test) the diff: most
families have no rig board, so `boards` alone would compile nothing for them."""
def test_off_rig_port_still_reports_family(self):
s = sel(['src/portable/microchip/samx7x/dcd_samx7x.c'])
self.assertFalse(s['full'])
self.assertEqual(s['boards'], {}) # no same7x board on the rig
self.assertEqual(s['families'], ['same7x'])
def test_port_families_are_reported(self):
s = sel(['src/portable/raspberrypi/rp2040/dcd_rp2040.c'])
self.assertIn('rp2040', s['families'])
def test_bsp_family_and_board_report_family(self):
self.assertEqual(sel(['hw/bsp/rp2040/family.cmake'])['families'], ['rp2040'])
self.assertEqual(sel(['hw/bsp/rp2040/boards/raspberry_pi_pico/board.h'])['families'],
['rp2040'])
def test_docs_only_has_no_families(self):
self.assertEqual(sel(['docs/info/contributing.rst'])['families'], [])
def test_full_selection_still_reports_families(self):
"""A full-matrix file must not hide the families of the other changed files:
consumers that build from `families` (e.g. /pre-pr) ignore `boards` when full."""
s = sel(['src/common/tusb_fifo.c', 'src/portable/microchip/samx7x/dcd_samx7x.c'])
self.assertTrue(s['full'])
self.assertIn('same7x', s['families'])
# full stays full: every roster board, and no args to narrow the run
self.assertEqual(set(s['boards']), {b['name'] for b in ROSTER})
self.assertTrue(all(v == 'all' for v in s['boards'].values()))
self.assertEqual(hil_select.selection_args(s, ROSTERS), {'tinyusb.json': ''})
self.assertEqual(hil_select.selection_args_by_flasher(s, ROSTERS), {'tinyusb.json': {}})
def test_family_order_does_not_matter(self):
# same as above with the full-matrix file last (was the only order that worked)
s = sel(['src/portable/microchip/samx7x/dcd_samx7x.c', 'src/common/tusb_fifo.c'])
self.assertTrue(s['full'])
self.assertIn('same7x', s['families'])
class TestGitDiffArgv(unittest.TestCase):
def test_diff_disables_rename_detection(self):
"""Without --no-renames git reports only a rename's destination, so moving an
HIL-relevant file to a non-code path would be classified as non-code only."""
self.assertIn('--no-renames', hil_select.GIT_DIFF_ARGV)
class TestPortWithoutFamilyIsFull(unittest.TestCase):
"""A port dir no family file references must widen (full matrix), not silently
contribute zero boards — the fail-open contract."""
def test_unreferenced_port_forces_full(self):
orig = hil_select.port_families
hil_select.port_families = lambda port_dir, repo_root: set()
try:
s = sel(['src/portable/vendor/newip/dcd_newip.c'])
finally:
hil_select.port_families = orig
self.assertTrue(s['full'])
self.assertTrue(any('no board family' in r for r in s['reasons']), s['reasons'])
if __name__ == '__main__':
unittest.main(verbosity=1)