Skip to content

kaizen: tier step dies on a non-JSON GitHub reply — a 60-min merge group lost at step 7; retry, then fall back to the full tier #5399

Open

Description

@noahgift

Loss

Merge-group run 38738529476, shard 3 (job 116433916212, yoga-build3) died at step 7 "Decide the test tier (BSE-17, PMAT-1077)" after 1.5 s with invalid character 'S' looking for beginning of value. That killed a ~60-min merge group for #5377 (cop report 13:3xZ; log read 13:5xZ).

Where

ci/sections.yml (step source, merge_group) arm, lines ~392-396 on origin/main): two bare gh api ... --jq calls, the PR head sha (pulls/${pr}) and the workspace-test check-run conclusion. The step runs under set -euo pipefail, so a non-JSON reply (an S... body, most likely "Service Unavailable" or "Server error") makes --jq fail and the whole step exits non-zero. The log shows the line before the error ("first parent available") and no Decide output after it. I did not read the HTTP status itself; the 'S' reply body is inferred from the error text.

Five whys

  1. Why did the MG die? Step 7 exited non-zero.
  2. Why? gh api ... --jq could not parse a non-JSON reply.
  3. Why does a bad reply fail the step? set -e plus no status check and no retry on a call whose only job is to refine the tier.
  4. Why is that call allowed to be fatal? The push arm already treats a failed lookup as "never a red step" (ci_mg_workspace_result.sh ... || mg=""); the merge_group arm was written without that rule.
  5. Why was it not caught? No test feeds the step a non-JSON API reply.

Countermeasure

In the merge_group arm, check the HTTP status and retry (3 tries with backoff) and, if the lookup still fails, fall back to tier=full (the script's own fail-closed answer, --pr-head omitted), never fail the step. Same shape as the push arm. A test or self-test case with a non-JSON reply.

Owner

aprender-a3. This is a .github/workflows / ci/sections.yml change, so it goes to the cop before any PR (per the cop's note).

Done when

A forced non-JSON reply in the step yields tier=full and a green step in a dry run, and no MG dies at step 7 for 7 days.

Activity

  1. added this to the 0.73.0 milestone on Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions