static_flow API tests silently do not run on pull requests from forks

[robjarawan]

static_flow/flow_maint_test.sh fetches three example scripts from MetPX/sarracenia at ${GITHUB_HEAD_REF}:

wget https://raw.githubusercontent.com/MetPX/sarracenia/${GITHUB_HEAD_REF}/sarracenia/examples/moth_api_consumer.py

GITHUB_HEAD_REF is set on pull_request events only, and it names a branch in the head repository. The URL hardcodes the owner as MetPX. So for a pull request raised from a fork, that ref resolves only if MetPX happens to carry a branch of the same name. Usually it does not.

What that costs

Two things, and the second is the one that concerns me more.

The check is permanently red for fork contributors, and it tests nothing. All three fetches 404, moth_api_producer.py is never written, and tests 2 and 3 fail. They do not fail because something is wrong with the pull request — they fail before any SR3 code executes. So on every fork pull request that check is noise, and the Python API path it exists to cover is not exercised at all.

From MetPX/sarracenia run 33986304253, 2026-09-05, on both matrix legs:

--2026-09-05 22:56:55--  https://raw.githubusercontent.com/MetPX/sarracenia/fix/sender-dry-run-local-dir/sarracenia/examples/moth_api_consumer.py
HTTP request sent, awaiting response... 404 Not Found
python3: can't open file '.../moth_api_producer.py': [Errno 2] No such file or directory
test  2 FAILURE: moth_api_consumer.py example should consume 5 messages.
test  3 FAILURE: blank results! flow_api_consumer.py should consume 5 messages, found

A push of that same commit 3d95cd91 passes, because GITHUB_HEAD_REF is empty there and the script falls back to development. Same code, same commit, different event.

Since 29b03a96 there have been 30 pull_request runs of flow_basic.yml, of which 14 failed. All the failures are fork pull requests. Most are mine, but Maksim-Burtsev and mmaxjr have raised fork pull requests over the same period.

Where a branch name does collide, it fails silently instead. If a fork's head branch happens to share a name with a MetPX branch — and people do raise pull requests from a branch called development — the fetch succeeds, pulls MetPX's copy of the examples, and the test passes on content that is not the pull request's. That is a wrong answer rather than a missing one, and nothing in the log says so.

Where it came from

29b03a96 moved this from GITHUB_BASE_REF to GITHUB_HEAD_REF so a pull request would exercise its own examples. That is the right intent and it works for same-repository pull requests. It just cannot work through a hardcoded owner when the head branch lives in a fork.

Line 50 also has if [ ! "${GITHUB_HEAD_REF}"] — missing a space before the bracket. It raises [: missing ']' on every pull request and makes the file unparseable by shellcheck. It happens not to change the outcome, but it is in the logs above.

What I propose

Under Actions the fetch is unnecessary: the sarracenia checkout is already on the runner at ${GITHUB_WORKSPACE}, and travis/flow_autoconfig.sh pip installs from that same tree. Taking the examples from there tests the pull request's own examples for fork and same-repository pull requests alike, guarantees the examples and the installed library are the same revision, and removes the network from the common path entirely.

Keep fetching for anyone running the script outside Actions, but settle the ref once rather than per file, so a genuinely deleted or renamed example still fails loudly instead of being quietly replaced by the development copy.

Happy to be told the simpler answer is to always fetch development and drop the ref logic — the examples change rarely enough that it would not lose much. I went the other way because it also closes the silent-wrong-content case.

@petersilva @andreleblanc11 @reidsunderland