Skip to content

ci.md

Latest commit

 

History

History
661 lines (600 loc) · 46.8 KB

ci.md

File metadata and controls

661 lines (600 loc) · 46.8 KB
Raw

CI and releases

Workflow YAML under .github/workflows/ is the source of truth. This page explains the boundaries; it does not duplicate every job or pinned version.

Issue triage

Issue forms apply needs-triage plus bug or feature according to the selected form. issue-triage.yml provides an asynchronous fallback for newly opened issues submitted through other paths. It skips opening events already carrying needs-triage, then checks the current issue before writing. Closed issues and issues with any lifecycle status are left unchanged. It only adds needs-triage; it never replaces other labels or runs in response to later label edits.

Failed API operations are retried up to three attempts, with a fresh status check before each write attempt. Runs for the same issue are serialized; issues with different numbers run independently. The workflow uses only issues: write and does not check out or execute repository or issue content.

To repair missed intake, a maintainer with write access can select Actions > Issue triage > Run workflow on the default branch and enter one positive issue number. Manual runs use the same lifecycle checks and reject pull request numbers. They do not scan or relabel the backlog.

Actions must be enabled and the run must succeed. Issues created using GITHUB_TOKEN do not trigger this workflow automatically; their creators must set needs-triage, or a maintainer can use the manual repair. Status reads and label writes are separate API calls, so a maintainer edit can still race a write. Who can change labels is controlled by repository permissions, not this workflow.

Pull-request gates

ci.yml always classifies the diff (from the merge base with the base branch, so a branch behind main does not inherit main's newer files as its own changes), and every job runs only when a path it reads changed. The classifier puts each changed path in one class:

Class Paths Jobs that run
documentation docs/**/*.md (.mdx, .rst, .adoc), changelog.d/*.md, changelog.d/release.json, the root README.md, AGENTS.md, CLAUDE.md, CHANGELOG.md, CONTRIBUTING.md, CODE_OF_CONDUCT.md, LICENSE, LICENSE.md the always-on guards (Classify Changes, Check AGENTS.md Links, Check Workflow Action Pins, Fix Regression Gate, PR Title, Storage Upgrade Compatibility, Dependency Guard (cargo deny); of these only Check AGENTS.md Links reads documentation, through scripts/check-docs.py)
GQT cases .gqt files anywhere under crates/omnigraph-gqt/cases/ (recursive discovery) the guards plus GQ Logic Tests (run_gqt)
deployment Dockerfile, .dockerignore, docker/**, deploy/** the guards plus Azure Contract Guards, Container Entrypoint, Azure Deployment Validation (run_deployment)
engine input every other path: crates/** (a text fixture under a crate is source code; only the .gqt corpus is a class of its own), tools/**, Cargo.toml, Cargo.lock, rust-toolchain.toml, .cargo/**, scripts/**, .github/**, anything unlisted every job (run_full_ci, which also sets run_gqt and run_deployment)

A class exists only when the set of jobs reading its paths is closed; a path outside every class is engine input and runs every job. The .gqt corpus is read by the omnigraph-gqt crate (its include_str! cases and its harness, all under GQ Logic Tests, whose dst-clippy job compiles the crate so Lint (clippy) owes it nothing), by scripts/check-fix-regression.py (Fix Regression Gate, always on), and by the seam guard crates/omnigraph-seams/tests/failpoint_names_guard.rs, which counts a case's at: name as arming a seam: Test Workspace runs it on engine input and GQT (ordinary) runs it on run_gqt, so a cases-only PR that drops the last case arming a seam turns the guard red where the PR can see it. The fixture parity test in crates/omnigraph-bench/src/branch_merge.rs also reads generated_branch_merge_dataset.gqt from the corpus. GQT (ordinary) runs it with the other benchmark unit tests on run_gqt, including cases-only changes, while Test Workspace covers it on engine input. No crate reads a deployment file. scripts/check-change-classes.py keeps the literal-spelling half of this true: it replays the classifier over fixture diffs and fails when a string literal in a Rust or TOML file under crates/ or tools/ spells a class path (root-relative or ../-relative) without the file being a listed reader; comments are not read, and a path assembled from pieces at runtime is invisible to it, so a reader of that shape is the reviewer's to list. It runs in Check Workflow Action Pins with its self-test. A dispatch, a tag, an unknown base or an empty diff runs every job. The diff is taken from the merge base, so a branch behind main does not see main's newer files as its own. Workflows that gate on the classification carry a verbatim copy of the Classify Changes job (a job cannot depend on another workflow's job); scripts/check-classify-copy.py discovers every copy under .github/workflows/ (.yml and .yaml, any line whose first token is classify_changes:) and holds it identical to ci.yml; a copy it cannot read fails the check rather than dropping out of it. It runs in Check Workflow Action Pins (with its self-test) and in every GQT qualification job.

Merge queue entries trigger ci.yml, gq-logic-tests.yml, and dst.yml through merge_group (checks_requested). These runs check out the combined queue commit; the classifier diffs the group's base and head, so a documentation-only entry skips the same work it skips on a pull request. Workspace tests fail fast as they do on pull requests. The PR metadata gates stay on pull_request_target; on the queue Fix Regression Gate reports a pass and PR Title checks the squash subject the entry is about to land; the vocabulary audit, Azurite, the format fence, the RustFS shards and the deployment jobs keep their pull-request or post-merge schedule and never run on the queue; the DST pinned suite runs there as a reporting context. Queue runs never publish releases or save the main branch's caches. Details: branch-protection.md, Merge queue.

Branch protection requires five contexts: CI Gate, Check Workflow Action Pins, GQ Logic Tests, Fix Regression Gate and PR Title. CI Gate is the ci_gate job of ci.yml: it needs classify_changes, check_agents_md, workflow_action_pins, cargo_deny, graph_vocabulary_guard, fmt, lint, test, storage_upgrade_compatibility and test_aws_feature, runs under if: always(), and scripts/ci_gate.py turns their results into the verdict: failure and cancelled are red, and skipped is accepted only where the job's own if: was false on that event, as the script's table restates it. Those ten jobs are what blocks main; a new blocking job joins needs: and the table in the same pull request, and scripts/check-merge-group-triggers.py holds the two equal and pins the gate's wiring. Check Workflow Action Pins is required on its own so a malformed gate cannot hide that validator's red. GQ Logic Tests, Fix Regression Gate and PR Title are their own workflows and keep their own names. Details and the recovery routes: branch-protection.md, CI Gate.

Storage Upgrade Compatibility (storage_upgrade_compatibility in ci.yml) runs on every change and in the merge queue. It builds the genuine stamp-13 CLI from the immutable STAMP_13_SOURCE_COMMIT in a detached worktree into the job's target directory, copies the binary out, cleans the path-package artifacts through both manifests (the predecessor and current packages share names and versions) and exports OMNIGRAPH_V13_BIN. The step Install released v0.10.0 and v0.11.0 CLIs downloads the two releases with scripts/install.sh and exports OMNIGRAPH_V6_BIN and OMNIGRAPH_V011_BIN: 0.11.0 writes stamp 9, and 0.10.0 followed by the 0.11.0 upgrade gives the stamp-8 source (--to-format 8) and the stamp-9 source its default target leaves. The step trusts the release assets exactly as the test job's v0.9.0 and v0.10.0 installs do: install.sh downloads the archive and its .sha256 from the same GitHub release and verifies the one against the other before extracting; no digest is pinned in the workflow. Each download gets three attempts (install_release), the only retry in the job; a failed attempt leaves no binary behind. The job's added runtime (two downloads and the five journeys) is UNVERIFIED against its timeout-minutes: 90 until the first hosted run; the timeout was not changed. With OMNIGRAPH_REQUIRE_STORAGE_UPGRADE_TESTS=1 a missing binary fails the five genuine journeys (genuine_v13_storage_upgrade_preserves_history, genuine_v0_11_0_storage_upgrade_preserves_history, genuine_v0_11_0_storage_upgrade_after_predecessor_cleanup, genuine_v0_10_0_to_stamp_8_storage_upgrade_preserves_history, genuine_v0_10_0_to_stamp_9_by_default_storage_upgrade_preserves_history) instead of skipping them; the v6 format fence in the test job keeps its own resolver and skips, never panics, without OMNIGRAPH_V6_BIN. Four scopes then run, each checked against its log by scripts/check-storage-upgrade-ci.py --check-log: the storage_upgrade cases of crossversion_upgrade.rs (a module of the CLI's integration binary, run by the exact names --list-cases crossversion --compiled takes from a -- --list of the same feature graph, which must contain every source-derived name, generated to a file before the run so an empty inventory fails the step, and the run's log must equal that file), the engine db::upgrade::tests, lance_version_columns (a module of the engine's integration binary, selected as lance_version_columns::) and forbidden_apis. All four select the Test Workspace packages (--workspace --exclude omnigraph-gqt --exclude omnigraph-gqt-served --exclude omnigraph-dst --features "$FAILPOINT_FEATURES"), so the job resolves one graph, the one Test Workspace builds, and the engine scopes reuse the crossversion scope's build; a scope that selected -p omnigraph-engine --features failpoints resolved a third graph, about 20 minutes warm. The script's --self-test pins the scopes, the predecessor build and install scripts (compared exactly) and the required case names, so removing or altering one fails Check Workflow Action Pins and this job. The 90-minute budget covers two cold builds into one target directory: the predecessor CLI alone (one package, one bin, no test features), then the current tree's test targets, whose dev-dependency features (lance-io defaults and test-util) differ, so only dependency artifacts with matching features are reused.

The released 0.12 cluster-ledger journey is separate manual release qualification. The complete current-version live-apply CLI journey runs in Tests slow nightly. Test Workspace retains the focused deployment and delivery-loss tests.

GQ Logic Tests (gq-logic-tests.yml) owns the complete .gqt corpus as a required context aggregating four qualification jobs. GQT (ordinary) checks unavailable-DST refusal and the unit tests of omnigraph-gqt, omnigraph-gqt-core and omnigraph-bench under an empty RUSTFLAGS, then runs the seam guard (crates/omnigraph-seams/tests/failpoint_names_guard.rs) in the same flagless shape. Both of its engine invocations select those same three packages, so they resolve one feature graph and the engine builds once: each crate alone resolves a different graph (omnigraph-gqt enables the engine's test-util and depends on omnigraph-dst), and three selections cost three engine builds, about 20 minutes each warm. The guard is a source walk whose crate declares no workspace crate (its dev-dependencies are yaml_serde, syn, tempfile and toml), so it adds no second engine build. The benchmark units include the fixture parity test against the generated branch-merge corpus case. The dispatch owner also checks external-store admission and persistence. Automatic corpus runs supply no --store, so schema-less query files are refused there; schema-and-seed datasets with zero steps are admitted. --measure refuses a selection with no DST environment. GQT (dst) runs the whole package, on engine v2, the one engine; a step's --- expect same as v1 comparison runs inside it, so no job selects an engine. GQT (dst-clippy) checks all package targets with Clippy. GQT (served) runs crates/omnigraph-gqt-served/tests/gqt_served_conformance.rs: every corpus case that declares the omnigraph-server target runs again through an in-process server, its in-process verdict must be green and its served verdict must equal it; gqt_served_count.rs in the same job totals the served and skipped cases and fails when no case is served. The same cell builds omnigraph, omnigraph-server and omnigraph-gqt, then explicitly runs omnigraph-cli/tests/integration/gqt_served_smoke.rs. That test reuses the CLI's cluster/server process helpers and runs a corpus case through the GQT command with a bearer token. It requires one passing served attempt, token redaction in the retained report and replay refusal. It is ignored outside this cell because the GQT binary must be built first. All three steps select the same four packages under an empty RUSTFLAGS, and the binaries are built with cargo test --no-run so every step resolves the same dev-dependency feature world (cargo build --bins would compile omnigraph-cluster again without failpoints and test-util). This is the one job that builds the tests-only omnigraph-gqt-served crate, so Test Workspace excludes it beside omnigraph-gqt. All four run from the repository root under the workspace Cargo configuration, which enables the seeded Tokio runtime. Each job has its own 60-minute budget and cache key. The budget is at least twice the observed 27-minute cold ordinary build. The roughly 30 minutes recorded for dst.yml cover a different package selection and workload; they do not establish a bound for GQT dst or dst-clippy. Exact cold timings for those jobs remain unverified. See the cache rule under Full correctness graphs for the budget policy. Matrix fail-fast cancels the remaining jobs when one fails; Cargo retains its default fail-fast between test targets. The required context fails if classification or any qualification fails, is cancelled, or is skipped. A successful run still requires all three jobs to pass; fail-fast never turns incomplete qualification into success. Test jobs upload invocation reports, and all three jobs upload available Cargo build timings separately, including on failure. Every corpus case is enrolled, including cases whose required graph behavior currently fails. Test Workspace excludes this separately tested package; it does not silently skip DST cases. GQ Logic Tests runs when its input changed (engine input or a .gqt case, the classifier's run_gqt) and otherwise reports success without building, the way the AWS job does; its workflow carries the Classify Changes copy under the name Classify Changes (GQ Logic Tests). Fix Regression Gate (fix-regression-gate.yml) holds every issue the PR body closes by keyword (Closes #N, Closes ModernRelay/omnigraph#N, or the issue URL) to a regression in the diff: a .gqt case anywhere in the corpus or a #[test]-attributed issue_N function, added or strengthened, in a top-level test target (tests/<name>.rs), tests/integration/main.rs or its direct modules, or a src/ module under crates/*, tools/* or skills/ (an owner test not yet named for the issue is renamed to carry issue_N when extended). An integration module requires a bare, top-level mod <name>; declaration in that root at the range's head commit. Attributed declarations, path overrides and deeper modules are not resolved. A PR whose diff changes no path under crates/, tools/ or skills/ (Markdown files there aside) and neither root Cargo.toml nor Cargo.lock passes unexamined, with a notice annotation saying so: a fix in a workflow, a script, a document, or a deployment file has no logic or Rust test that could witness it. Owners the gate does not recognize inside those paths (helper and fixture modules, a script under a crate, a rustdoc-only change) go through the no-repro label, which a maintainer applies to waive the check per PR; scripts/check-fix-regression.py is the check. A failure names the code paths that made the gate look, the ways through, any near miss in the diff (a case whose header says # issue: N under another name, a test named with the bare number, moved, under a leading _, or in a helper module, an issue-named function with no test attribute), and a case skeleton, as a log line and as a GitHub error annotation. It is a policy check, so it runs on pull_request_target: the workflow and the script come from main, and the pull request head is fetched only as data for the diff range, never checked out or executed. It runs on body edits and label changes as well as pushes, builds nothing, and runs for every change class. On the merge queue's branch it reports a pass without a check (branch-protection.md, Merge queue). PR Title (pr-title.yml) holds the pull request title to type(scope)!: description: a type from feat fix perf refactor docs test ci build revert rfc release, an optional lowercase scope list, ! for a breaking change, and a description that starts lowercase (a backtick or digit also passes), carries no trailing period, and keeps the title within 100 characters with no whitespace but the plain space; docs(rfc): is refused in favour of rfc:, and GitHub's Revert "…" title in favour of revert: … (the rule and its examples: CONTRIBUTING.md, Pull Requests). scripts/check-pr-title.py is the check; it runs its --self-test first and prints each independently detectable problem with a stable diagnostic code and a repair instruction, both in the log and in a GitHub error annotation. Length failures include the actual length, the limit and how many characters must be removed. Text output uses red failures, yellow repairs and green successes on a terminal or in Actions. Use --color never or set NO_COLOR to a nonempty value to disable automatic color; --color always forces it. Agents can use --format json with either input option for one JSON object. Version 1 carries schema_version, offset_unit, input_kind, valid, title, length, max_length, excess and errors. Each error has a closed code, its field, a span with start and end, the exact found text, expected, message and fix. Ranges are zero-based, end-exclusive Unicode code-point offsets in the checked title; for a squash message, that means the extracted title. A zero-width range marks a missing token; an unclosed scope marks its unmatched opening parenthesis. The human report names the same range using one-based character positions, or an insertion offset. --schema prints the JSON Schema with the allowed codes, fields and record shapes. Enum types close the code, field, input, format and color sets; immutable records validate ranges and their original-title slices at runtime. Numeric facts and the validity flag are derived from the report. JSON contains no color or Actions annotations. Both formats exit 0 for an accepted title and 1 for a refused title; invalid command-line arguments exit 2. For example:

python3 scripts/check-pr-title.py --title "fix(merge): find the merge base over contracted lineage" --format json
python3 scripts/check-pr-title.py --schema

The check uses Python 3.11 or later and the standard library. Its --self-test entry point runs scripts/test_check_pr_title.py, including an exact range and field fixture for every error code, Unicode/length boundaries, parser refusals, runtime type invariants and each output/CLI mode. When editing the checker, also run python3 -m mypy --strict scripts/check-pr-title.py in a development environment with mypy installed. The enum dispatches are exhaustive.

It is a policy check on pull_request_target like the regression gate, re-run on every title edit, and the pull request head is never fetched. On the merge queue's branch it checks the first line of the group commit's message, minus the (#N) suffix: the subject the queue lands, fixed when the group is built, so a title edit after queueing cannot change what lands unchecked (branch-protection.md, Merge queue).

The Check AGENTS.md Links context also runs scripts/check-docs.py, which validates local documentation links, user/developer audience boundaries, RFC location and metadata, registry agreement, and the absence of committed merge-conflict markers in Markdown. Before the documentation checks run, the same context also rejects any pull request whose own diff adds a conflict-marker line in any file type (a pull-request-only step; the merge-group run skips it, the pull request run having covered the diff), annotating each offending file and line; markers already on the base branch never fail an unrelated pull request. There is no exemption; a document that must quote a conflict block indents the markers one space. After the documentation checks, the same context runs typos (crate-ci/typos, pinned by commit) over every tracked text file, hidden paths such as .github/ included; the tool itself skips Cargo.toml manifests, lock files and binaries. It matches each word against a list of known misspellings, not a dictionary: an unknown word never fires, and an identifier fires only when one of the words it splits into is on the list (a CamelCase fragment or a short abbreviation can be one). A flagged token that is correct where it occurs gets one commented line in .typos.toml: under [default.extend-words] when it appears in prose and code alike, under [type.rust.extend-words] (or extend-identifiers for one exact identifier) when it exists only in Rust sources, so the same misspelling in Markdown still fires; a hyphenated prefix goes in extend-ignore-re; a generated text file gets an extend-exclude glob. Every exemption lives in that one file: the job refuses a sibling typos.toml or _typos.toml, and CI ignores a config file in a subdirectory (a local run inside that subdirectory would not).

Graph Vocabulary Guard remains a dependency of CI Gate, but its substrate-sized audit steps are currently disabled everywhere (decision of 2026-08-28; the job-level VOCABULARY_AUDIT_ENABLED variable in ci.yml is the single switch). The job still runs its unit tests and reports success so the exact-SHA release gates stay wired. When re-enabled it checks OpenAPI, Rust presentation strings, and public Rust against the reviewed terminology inventory after merge, on tags, and by manual dispatch. User documentation is intentionally outside this exact-occurrence audit and is owned by scripts/check-docs.py. The AWS job reports a successful skip when no engine input changed; formatting and Clippy are also skipped by the classifier without leaving required contexts pending.

Automatic edge and versioned publication are jobs in the same CI run and cannot start until that run's vocabulary audit succeeds. Each publishing workflow then re-verifies the authorizing CI run, resolves its source once to an immutable commit, and builds only that commit. Version tags are checked again immediately before publication; a stale main audit cannot move the rolling edge tag backward. A manual backfill must already have a successful non-PR vocabulary audit for the exact commit. The current manual workflows therefore fail closed for historical pre-guard tags rather than offering a force bypass.

GitHub evaluates a tag-push workflow from the commit selected by that tag. A new v* tag must not be created against a commit that predates these gates, because no later workflow edit can retroactively replace that commit's old publisher definitions. Enforcing that administrative boundary against tag creators requires repository tag policy in addition to the checked-in workflow gate.

Formatting and Clippy use the repository's pinned toolchain. Lints remain warnings in the workspace; CI applies -D warnings. Clippy runs both the default and failpoint-superset graphs.

Repository metadata gates also check:

  • immutable commit SHAs for external Actions and reusable workflows;
  • agreement between container and package binary sets;
  • the dependency direction around omnigraph-azure-admission;
  • the dependency graph, in Dependency Guard (cargo deny): cargo deny --locked check holds Cargo.lock and every manifest, all features enabled, to the allowlists in deny.toml (registry and git sources, licenses, RustSec advisories and yanked versions, wildcard version specs; dev-dependencies included; the rules and their reasons live there). It runs on every pull request, documentation-only ones included. A new source, license, or advisory exemption is an edit to deny.toml in the same pull request, reviewed as such (a git-form [patch] entry is refused like any git source). Three refusals name no remedy: a path dependency without version in a publishable crate is a wildcard (add version); a workspace member without a license field is unlicensed (add license = "MIT"); a crate whose license text cargo-deny cannot read needs a [[licenses.clarify]] entry. A path copy of a crate, bare or behind [patch], and a .cargo/config.toml source replacement keep no source for cargo-deny to check; scripts/check-dependency-sources.py, run in the same job, refuses them: the source-less Cargo.lock packages are exactly the workspace members by name and version, no manifest declares a [patch] table, and neither .cargo/config.toml nor the deprecated .cargo/config declares a source replacement or path override. Build scripts are outside both checks. An exemption that no longer matches anything fails the check, so the bump that clears an advisory or drops a license's last holder also removes its deny.toml row. The job blocks main through CI Gate (see branch-protection.md). The RustSec database is fetched at run time; dependency-guard-nightly.yml runs the same check on main daily, so an advisory published overnight shows there first and then turns every open pull request red at its next push; the fix is a lockfile bump or a deny.toml exemption in its own pull request, not a rerun.

Container entrypoint and Azure deployment-validation jobs test argument composition, non-destructive Bicep validation, bootstrap readiness/admission modes, and non-root image ownership. They and the container/package binary-set and admission-direction guards run when engine input or a deployment file changed (the classifier's run_deployment).

Full correctness graphs

The workspace suite (Test Workspace) runs on every pull request, merge-queue entry and push to main that changes engine input, on release tags, and by manual dispatch. GQT has its own configured owner above. The main, tag, and dispatch form (a pull request and a merge-queue entry drop --no-fail-fast):

cargo nextest run --workspace --exclude omnigraph-gqt --exclude omnigraph-gqt-served --exclude omnigraph-dst --locked \
  --profile ci --no-fail-fast \
  --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints

On a pull request and on the merge queue's branch it blocks main through CI Gate (branch-protection.md) and it fails fast: the queue waits for it before merging. On main, tags, and dispatch it is the post-merge detection channel and keeps --no-fail-fast, so every independent failure stays attributable; a red run there is stop-the-line. The job compiles in one step (cargo test --no-run) and runs in the next, so compile and run wall clock read apart in the log.

The run step uses cargo-nextest (binary version pinned in the job through the commit-pinned taiki-e/install-action) rather than cargo test: cargo test runs the workspace's test binaries one after another, so the run's wall clock is the sum of the binaries' durations, and a binary whose remaining tests are few and process-bound (the CLI, server and upgrade journeys) holds the machine while they finish. nextest schedules every test of every binary through one queue, a process per test, so another binary's tests fill that time. Which journeys form the tail is a matter for the suite's per-test timings, not asserted here. nextest runs no doctest, so the step keeps a cargo test --doc call on the same graph after it. The ci profile in .config/nextest.toml writes target/nextest/ci/junit.xml with captured output, and scripts/check-workspace-test-owners.py reads that report for the owners the job requires by name (the two upgrade journeys and the CLI/server/proxy journeys listed in the step) and for the upgrade journeys' captured notices, in place of grepping a --nocapture log. The local command in testing.md stays cargo test; the two run the same binaries.

Every run, red or green, uploads that report as test-workspace-junit and the compile step's --timings page as test-workspace-build-timings, both kept 14 days: per-test durations and per-crate compile and link times on the hosted runner, which any change to this job's scheduling is sized from. The ci profile terminates a test that runs ten minutes (slow-timeout), so a hang fails as a named test instead of holding the job to its 60-minute limit, and a test still running after one minute is reported as SLOW in the log.

The main run also seeds the dependency cache that pull requests restore. Every Swatinem/rust-cache step in ci.yml, gq-logic-tests.yml, and dst.yml saves only from main (save-if): a save from any other ref, a pull-request branch or a tag, is restorable by no pull request and only evicts shared entries under the repository cache cap. The pull-request-path jobs also save when red (cache-on-failure): dependency artifacts are valid whatever the test verdict, and a red seed run would otherwise leave every pull request cold until main is green again.

The active PR-path Rust jobs also install sccache through the commit-pinned mozilla-actions/sccache-action, with the binary version pinned to v0.16.0. Each job sets RUSTC_WRAPPER=sccache and SCCACHE_GHA_ENABLED=true. SCCACHE_GHA_RW_MODE is READ_WRITE on refs/heads/main and READ_ONLY on every other ref, including pull requests, merge-queue branches and tags. This keeps per-compilation uploads under the same main-only policy as the archive cache; rust-cache's save-if does not control sccache. Both caches share the repository's Actions cache quota. The vocabulary audit (nightly toolchain, disabled) and the release and publish workflows carry no sccache step.

rust-cache restores an archive keyed on the toolchain, build environment, manifests and Cargo.lock, with prefix fallback when dependencies change. When Cargo still invokes rustc, sccache can reuse retained compilations whose source, compiler, flags, paths and dependency hashes match. rust-cache exports CARGO_INCREMENTAL=0, so eligible workspace library compilations can be cached too. Linking remains uncached, including binaries, test executables and proc-macro crates; unchanged source alone does not guarantee a hit. Adding RUSTC_WRAPPER changes rust-cache's environment hash, including its prefix restore key, so existing archives do not seed the first run. Changing the pinned sccache version also changes its cache namespace. Cargo can print Compiling on a cache hit: use the action's post-step hit/miss counts and build timings to measure reuse. The speedup and cache retention need hosted-run evidence.

A cache is derived state, and no job may need one to fit its budget. The repository's caches exceed GitHub's cap, so every save evicts the least recently used entries: within one main run, the jobs that finish first are evicted first by the saves of the jobs that finish last. A run cancelled by its timeout does not save a rust-cache archive (issue #755: every run after the cancelled main run restored nothing). Completed sccache uploads can survive a later cancellation, but neither cache is guaranteed. Each GQ Logic Tests budget therefore covers a cold build with margin, and a second cargo invocation in a job must not change the dependency graph: it selects packages whose graph the first invocation already built, or it rebuilds every crate whose features differ (issue #755 was GQT (ordinary) running the seam guard as an engine integration test, whose dev-dependencies resolve a second graph, 49 minutes cold against 45; the Azurite job ran its owners under six per-crate selections, four of them with the Lance stack, 89 minutes on main). A job that runs several owners names one selection once and passes each owner only its target.

Every Rust job in those three workflows installs the rust-toolchain.toml pin with a bare rustup toolchain install; the rustc version is part of every cache key, so the pin is what keeps caches warm across Rust releases. The release and publish workflows still build on the floating stable action and save their caches from the tag ref; they are outside this rule.

The remaining jobs own contracts that need special infrastructure. They run after merge, on tags, and by manual dispatch; four of them, the format fence, the RustFS S3 integration, the Azurite Azure integration, and the AWS feature build, also run on pull requests:

  • Graph vocabulary audit checks OpenAPI, Rust presentation strings, and public Rust against the reviewed terminology inventory (audit steps currently disabled; see above).
  • V5 ↔ V10 format fence builds the immutable final-v5 CLI and proves mutual refusal plus the documented export/init/load rebuild. It also runs on every pull request that changes engine input, as a reporting context: the rebuild check compares the rebuilt export against the predecessor's, so a loss or a spelling change in what it compares reports on the pull request; wait for it before clicking Merge when ready. A red fence on a pull request that touched neither the export, the loader, nor the format is inherited from main: compare with the latest main run before reading it as the pull request's.
  • RustFS S3 integration runs configured engine, server, cluster, CLI, and recovery owners. A configured test that skips is a failure. It also runs on every pull request that changes engine input, as a reporting context: the configured S3 owners run nowhere else, so a contract change that updates only the local-FS twin of an object-store test reports on the pull request instead of first appearing on main; wait for both shards before clicking Merge when ready. A red shard on a pull request that touched no object-store code, or one that names no test (the 60-minute ceiling, the image pull, RustFS readiness), is inherited from main or from infrastructure: compare with the latest main run before reading it as the pull request's. To reproduce locally, the job's env block and its Start RustFS and Create RustFS test bucket steps in ci.yml are the complete recipe.
  • Azurite Azure integration exercises configured storage, admission-lease, recovery, cluster, server, and CLI owners against a digest-pinned Azurite image, then verifies that control objects, Lance data, and the admission object use the declared container. A configured test that skips is a failure, and every owner it runs by name must report ok, so a deleted or renamed owner fails the job rather than passing vacuously; scripts/check-ci-cells.py, a step of Check Workflow Action Pins, refuses such a name on the pull request without building any Rust. The job runs on every pull request that changes engine input, as a reporting context, and after merge, on tags, and by manual dispatch: the configured Azure owners run nowhere else, so a change that removes or renames one reports on the pull request instead of first appearing on main; wait for it before clicking Merge when ready. Every owner runs under the Test Workspace selection (--workspace --exclude omnigraph-gqt --exclude omnigraph-gqt-served --exclude omnigraph-dst --features "$FAILPOINT_FEATURES") and names only its target, so the job builds one Lance graph. Its 90-minute ceiling is the cold-cache envelope; a warm run takes minutes. A red run on a pull request that touched no object-store code, or one that names no test (the image pull, Azurite readiness), is inherited from main or from infrastructure: compare with the latest main run before reading it as the pull request's.
  • AWS feature builds and tests omnigraph-server with --features aws; it also runs on every pull request and blocks main through CI Gate.

Azure remains a qualification preview. Emulator coverage and the completed managed-identity smoke proof do not replace the pending adversarial live-Azure matrix, and every mutation-capable Azure server must retain the admission-wrapper boundary.

CI checks OpenAPI drift but never rewrites openapi.json. Regenerate an intentional API change locally as described in testing.md.

DST tiers

Two workflows own the simulator's pinned tests and generated fleets. The omnigraph-dst crate builds with the workspace --cfg tokio_unstable like every other crate; the default test jobs exclude it by name, and neither workflow sets RUSTFLAGS (an env RUSTFLAGS would replace the configured list):

  • dst.yml (per PR and main push that changes engine input, through its own Classify Changes (DST) copy; a superseded PR run is cancelled, which loses no cache uploads because PRs write neither cache): the pinned deterministic suite — every failure line carries the universe seed, so a red run is reproducible locally from the log alone. The job also lints the shipped engine shape (-p, no dst feature), which workspace feature unification hides from the default Clippy job. Whether the suite blocks a merge is the branch-protection required-contexts list.
  • dst-nightly.yml (cron 03:00 UTC + manual dispatch): matrix-sharded deterministic and concurrent fleets over date-derived, mutually disjoint seed intervals. Failures are logs with seed rows, not required contexts; the concurrent fleet's wild mode makes no replay claim.

gq-logic-tests.yml separately owns authored GQT execution through DST. Every step runs from the repo root under the workspace Cargo configuration; the refusal step clears RUSTFLAGS to build the one flagless shape, and the seam guard step runs under the same empty RUSTFLAGS to share its artifacts. An unavailable-runtime refusal test does not replace executing the DST cases. gqt-slow-nightly.yml (cron 03:30 UTC + manual dispatch) runs the cases under crates/omnigraph-gqt/cases_slow/ through the omnigraph-gqt binary, one at a time; it is not a required context, and it is its own workflow because the GQT runner refuses the pool-quiescing variables dst-nightly.yml sets. tests-slow-nightly.yml (cron 04:15 UTC + manual dispatch) runs the #[ignore]d Rust tests enrolled by exact name in its list, the nightly: deadline and journey tests and the heavy-repro: scale repros (AGENTS.md, ignore species), through cargo nextest run --run-ignored only on the workspace graph with --no-fail-fast, then check-workspace-test-owners.py over its JUnit report, so an enrolled test that was renamed, filtered out or red fails the run; it is not a required context.

Local pre-push checks

For Rust changes:

cargo fmt --all --check
cargo clippy --workspace --all-targets --locked -- -D warnings -W clippy::dbg_macro
cargo clippy --workspace --all-targets --locked \
  --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints \
  -- -D warnings -W clippy::dbg_macro
cargo test --workspace --exclude omnigraph-gqt --exclude omnigraph-gqt-served --exclude omnigraph-dst --locked \
  --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints
cargo test -p omnigraph-gqt --locked --lib --test runner_dispatch

From crates/omnigraph-gqt, also run the complete configured package:

cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked
cargo clippy -p omnigraph-gqt -p omnigraph-gqt-core --all-targets --locked -- -D warnings -W clippy::dbg_macro

For repository metadata and workflow changes, first activate the documentation environment:

bash scripts/check-agents-md.sh
python3 scripts/check-docs.py
python3 scripts/check-workflow-action-pins.py
python3 scripts/check-storage-upgrade-ci.py --self-test
python3 scripts/check-merge-group-triggers.py --self-test
python3 scripts/ci_gate.py --self-test
python3 scripts/check-pr-title.py --self-test
python3 scripts/check-pr-title.py --title "<the title you will give the pull request>"
python3 scripts/check-ci-cells.py --self-test
python3 scripts/check-release-vocabulary-gates.py
python3 scripts/check-container-binary-contract.py
python3 scripts/check-azure-admission-boundary.py
python3 scripts/check-dependency-sources.py
cargo deny --locked check   # from the repository root, after Cargo.lock is current; allowlist in deny.toml
typos                       # from the repository root; a subdirectory run scans only that subtree
actionlint .github/workflows/*.yml
shellcheck scripts/*.sh

typos (cargo install typos-cli --locked --version 1.50.1, the version ci.yml pins; the misspelling list grows per release, so a newer local binary can flag words CI accepts), cargo-deny (cargo install cargo-deny --locked --version 0.20.2, the version the pinned cargo-deny-action bundles; deny.toml uses the unsound scope field, which needs 0.19 or newer; run it from the repository root once Cargo.lock is current, since --locked refuses a stale lockfile and a subdirectory run scopes the graph to that package and reports the root's ignores as unmatched; the advisory database grows daily, so a local run can report an advisory CI has not seen yet or the reverse), actionlint and shellcheck are developer tools, not workspace dependencies. Run the applicable subset when a change does not touch their surface.

Release workflows

Check AGENTS.md Links validates release-note inputs, runs the focused Python composer tests and uploads a release-notes-preview artifact with links pinned to its selected commit. The checker and composer use the same pinned CommonMark dependencies from scripts/requirements-docs.txt, installed in a temporary environment. Its full-history checkout supplies the explicit previous release and pinned migration source. Note-only changes use the documentation class. The authoring guide describes permanent fragments, the supported link format and snapshot generation.

For v0.12.0 and later, the stable publisher requires a generated release snapshot whose complete note manifest and configuration still match its audited source. The recorded input commit is informational, allowing squash merges; release base and migration source ancestry are still required. The publisher installs the same pinned parser in a temporary environment after the historical cutoff. It derives the GitHub body from that same snapshot with links pinned to the release tag. Earlier manual backfills keep their asset-only route. Edge is unchanged.

Workflow Trigger and output
release-edge.yml Called by a main CI run that changed engine input, after its vocabulary audit, or manually for an already-audited current main; updates the rolling edge release and platform archives. A main push that changes only documentation, .gqt cases or deployment files makes no edge release: the binaries did not change (the container image is published from tags).
release.yml Called by audited v* tag CI or manually for an already-audited tag; builds platform archives, publishes the GitHub release, updates Homebrew when credentials are available, and smoke-tests the Windows installer.
publish-crates.yml Called by audited v* tag CI or manually for an already-audited tag; publication remains paused until the registry-ownership policy changes.
publish-image.yml Called by audited v* tag CI or manually for an already-audited tag; builds the bookworm-compatible public server image for GHCR and, when configured, Docker Hub. Manual backfills do not move latest.
package.yml / omnigraph-package.yml Manual AWS CodeBuild packaging for default and AWS-feature artifacts, with checksums, digests, and attestations.
refresh-docs-site.yml Documentation changes on main or manual dispatch; requests a docs-site redeploy.

Release archives and containers include the CLI, server, and Azure admission wrapper where their packaging contract requires all three. Keep the reusable package workflow, Dockerfile, and binary-contract check aligned.

Every release build sets RUSTFLAGS itself (release.yml, release-edge.yml, publish-image.yml, omnigraph-package.yml, scripts/install-source.sh, the documented source build): empty everywhere except the macOS archive row's -C code-model=large. A set RUSTFLAGS, empty included, replaces the workspace .cargo/config.toml [build] rustflags, so shipped binaries never carry --cfg tokio_unstable. A plain cargo build --release outside these paths still inherits the cfg. The cfg serves the DST suite and the GQT DST runner only.

Changing CI

  1. Preserve a reporting path for every branch-protection context on every pull request. A job that blocks main is listed in the ci_gate job's needs: and in the table of scripts/ci_gate.py in the same pull request, with a skip rule that restates the job's own if:; scripts/check-merge-group-triggers.py refuses the two lists apart and any continue-on-error on a job the gate needs.
  2. Keep external Actions and reusable workflows pinned to full commit SHAs.
  3. Update the classifier when adding a documentation format or a change class; a class needs a grep-verified closed set of reading jobs, every path outside a class runs every job, and never classify by extension outside the approved paths.
  4. Keep configured object-store jobs fail-closed on accidental skips.
  5. Keep every automatic artifact publisher transitively behind a successful exact-SHA vocabulary audit; a skipped pull-request context never authorizes publication.
  6. Update branch-protection.md only when the declared required contexts or policy actually change.
  7. Keep every required context reporting on the merge queue's temporary branch too: a workflow that owns one lists merge_group under on:; a job condition that admits pull_request by name (== 'pull_request') also admits merge_group; a negative gate written for post-merge venues (!= 'pull_request') also excludes merge_group, so the queue runs only what blocks it. scripts/check-merge-group-triggers.py enforces the first two (branch-protection.md, Merge queue), and a job the gate needs that skips on the queue where its if: says it runs makes CI Gate red there.