feat: deterministic manifest schema v2 — the manifest hash is the content address

[Fieldnote-Echo]

Wave U1 of the OrdinalDB post-mortem hardening (builder post-mortem F3; reconciled plan + implementation spec in the OrdinalDB planning docs; trust co-design: the deterministic manifest is the future Ed25519 seal surface — see ordinaldb docs/roadmap/ordinaldb-trust-spec.md).

Breaking schema change (v2, zero back-compat by decision — nothing published):

• manifest_id and created_at removed from IndexManifest; creation writes build: None (no more per-write UUIDs/timestamps) → two writes of identical content produce byte-identical manifest.json, so sha256(manifest.json) is a stable content address.
• Schema version bumped to ordvec.index_manifest.v2; old-schema manifests fail with a clear "older/newer manifest schema" error (version probed on parse failure).
• Canonical-bytes format contract: aux entries sorted by (name, path) at creation; canonical bundle-relative forward-slash paths enforced at both create and verify (red-team fix: creation previously could embed paths its own verification rejects); golden byte fixture + determinism tests are the merge gates.
• VerificationReport.manifest_id and the sqlite registry's manifest_id column removed (forced fallout; cache identity already keyed on manifest_sha256).

Adversarial 3-lens review pass done pre-PR: 5 medium findings found and fixed (create/verify canonicality asymmetry, non-escaping .. acceptance, Windows absolute-path contract split, stale README/examples, missing CHANGELOG). Tests: 108 passed (-p ordvec-manifest --all-features), root suite 256 passed; clippy/fmt clean; CI matrix extended to run manifest tests across OSes.

Note: uuid dep retained (still used by row-identity db_id validation).

Stacked: feat/typed-verification-codes (U2) builds on this branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BQ1JzAocstWiqtCF5J2V7K

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[qodo-code-review]

PR Summary by Qodo

Deterministic manifest schema v2 (hash = content address) + cache migration

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Bump index manifest schema to v2, removing per-write IDs/timestamps for deterministic bytes.
• Enforce canonical bundle-relative paths at both create and verify, with clearer schema errors.
• Drop manifest_id from verification reports + sqlite registry, adding migrations and determinism
 gates.

Diagram

graph TD
A["Index builder"] --> B["Manifest create (v2)"] --> C["manifest.json bytes"] --> D["sha256 content ID"]
C --> E["Load + verify"] --> F["SQLite report cache"]
G["CI (multi-OS)"] --> E

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Keep v1 parsing as a compatibility shim

• ➕ Smoother upgrades if any v1 manifests exist in the wild
• ➕ Allows gradual migration tooling instead of a hard cutover
• ➖ Adds ongoing maintenance burden and ambiguous behavior around determinism guarantees
• ➖ Risk of subtly different validation/path rules across schema generations

2. Add an explicit `manifest_sha256` field instead of relying on canonical bytes

• ➕ Identity becomes visible without re-hashing bytes
• ➕ Could decouple identity from JSON formatting decisions
• ➖ Bootstrapping problem: the field changes the bytes being hashed unless excluded
• ➖ Creates a second source of truth that must be re-validated

3. Adopt a standardized JSON canonicalization scheme (e.g., RFC 8785/JCS)

• ➕ More portable/verifiable canonical JSON definition across implementations
• ➕ Reduces risk of serde-specific formatting becoming a long-term contract
• ➖ Additional dependency/complexity; may still be a schema-version event to introduce
• ➖ Current approach already documents write_manifest_file as the canonical byte surface

Recommendation: The PR’s approach (make write_manifest_file the explicitly documented canonical byte surface, remove entropy fields, and gate determinism via golden bytes/tests) is the right fit given the stated goal (content-addressed identity) and the pre-release decision for zero back-compat. The main longer-term consideration is whether to later switch to a standards-based canonical JSON scheme; if that happens, it should remain a schema-versioned change as this PR already formalizes.

Files changed (10) +645 / -116

Enhancement (2) +200 / -98

lib.rsImplement v2 deterministic manifest + canonical path enforcement +176/-51

Implement v2 deterministic manifest + canonical path enforcement

• Bumps 'SCHEMA_VERSION' to v2, removes 'manifest_id'/'created_at' from 'IndexManifest', and makes creation emit 'build: None' while sorting auxiliary artifacts by '(name, path)' to guarantee stable bytes. Adds schema-version probing on parse failures for clearer older/newer-schema errors, enforces canonical embedded paths across artifact/row_identity/aux/calibration/encoder-distortion refs, and hardens create-time path embedding to never mint a manifest that verify would reject (including Windows absolute/verbatim path normalization).

ordvec-manifest/src/lib.rs

sqlite.rsDrop manifest_id from sqlite registry schema and cache lookups +24/-47

Drop manifest_id from sqlite registry schema and cache lookups

• Removes 'manifest_id' from 'verification_reports' and 'active_manifest' tables, updates cache lookup/store paths accordingly, and forces migration if a stale 'manifest_id' column exists (due to NOT NULL constraints). Adjusts migration logic and related query parameter ordering to keep report caching keyed on manifest/location/options hashes.

ordvec-manifest/src/sqlite.rs

Refactor (1) +1 / -8

main.rsRemove manifest_id from CLI display and success output +1/-8

Remove manifest_id from CLI display and success output

• Updates 'inspect' and verification success output to stop referencing 'manifest_id', aligning the CLI with the v2 schema and content-addressed identity model.

ordvec-manifest/src/main.rs

Tests (4) +409 / -4

test_manifest_bindings.pyUpdate Python binding tests for v2 schema fields +0/-2

Update Python binding tests for v2 schema fields

• Removes 'manifest_id' and 'created_at' from the test manifest fixture to match the v2 IndexManifest schema.

ordvec-manifest-python/tests/test_manifest_bindings.py

deterministic.rsAdd determinism merge gates + canonical-path and schema-error tests +314/-0

Add determinism merge gates + canonical-path and schema-error tests

• Introduces tests that assert byte-identical manifests for identical inputs, golden-byte matching, sensitivity to artifact/aux changes, and aux declaration-order invariance. Adds coverage for clear v1 schema rejection messaging, canonical path validation behavior (including policy-governed absolutes), and create-time rejection of non-embeddable paths.

ordvec-manifest/tests/deterministic.rs

manifest.v2.jsonCheck in golden canonical manifest bytes for v2 +39/-0

Check in golden canonical manifest bytes for v2

• Adds a fixed v2 manifest JSON fixture used as the canonical serialization gate; any serializer/format change becomes a deliberate schema-version event.

ordvec-manifest/tests/golden/manifest.v2.json

manifest.rsUpdate manifest tests for v2 + add canonical profile-ref checks +56/-2

Update manifest tests for v2 + add canonical profile-ref checks

• Adjusts existing assertions that previously depended on 'manifest_id', updates build-info setup for validation tests, and adds coverage ensuring calibration and encoder-distortion profile paths must be canonical. Extends sqlite migration tests to assert the 'manifest_id' column is removed.

ordvec-manifest/tests/manifest.rs

Documentation (2) +30 / -6

CHANGELOG.mdDocument breaking deterministic manifest schema v2 +21/-1

Document breaking deterministic manifest schema v2

• Updates the Unreleased section with the v2 schema change rationale, determinism contract, canonical path rules, and sqlite registry fallout/migration. Clarifies that serialization changes are schema-version events.

CHANGELOG.md

README.mdRefresh README for v2 determinism + canonical path rules +9/-5

Refresh README for v2 determinism + canonical path rules

• Updates schema version references to v2 and explains determinism/content-addressing expectations. Removes 'manifest_id' from report examples and documents the new canonical-path failure code for auxiliary artifacts.

ordvec-manifest/README.md

Other (1) +5 / -0

ci.ymlRun ordvec-manifest tests on non-Ubuntu OS lanes +5/-0

Run ordvec-manifest tests on non-Ubuntu OS lanes

• Adds a dedicated 'cargo test -p ordvec-manifest' step in the multi-OS workflow. This increases coverage for OS-sensitive path handling without duplicating the Ubuntu feature-matrix lane.

.github/workflows/ci.yml

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. active_manifest not migrated ✓ Resolved 🐞 Bug ☼ Reliability

Description

init() migrates only verification_reports and then uses CREATE TABLE IF NOT EXISTS for
active_manifest, so existing databases with the legacy active_manifest schema are not updated.
activate() now inserts into active_manifest without manifest_id, which can fail against
pre-existing DBs due to schema mismatch / legacy NOT NULL constraints.

Code

ordvec-manifest/src/sqlite.rs[R150-154]

        CREATE TABLE IF NOT EXISTS active_manifest(
            id INTEGER PRIMARY KEY CHECK(id = 1),
-            manifest_id TEXT NOT NULL,
            manifest_path TEXT NOT NULL,
            activated_at TEXT NOT NULL,
            forced INTEGER NOT NULL

Relevance

⭐⭐⭐ High

SQLite schema-migration robustness changes were previously accepted (legacy schema
detection/migration work in PR #203; cache schema work in #178).

PR-#203
PR-#178

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

init() performs a migration only for verification_reports and then uses `CREATE TABLE IF NOT
EXISTS for active_manifest, meaning existing active_manifest` schemas are not changed.
activate() assumes the new schema by inserting without manifest_id, which is incompatible with
legacy tables that still require it.

ordvec-manifest/src/sqlite.rs[94-158]
ordvec-manifest/src/sqlite.rs[44-92]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
SQLite upgrade logic drops `manifest_id` from `active_manifest`, but `init()` never migrates existing `active_manifest` tables—only `verification_reports` is migrated. Since `CREATE TABLE IF NOT EXISTS` does not alter existing tables, upgraded installs can break at runtime when `activate()` runs its new INSERT statement.

## Issue Context
After upgrading, older databases may still have an `active_manifest` table that includes a `manifest_id` column (and potentially a NOT NULL constraint). The new code inserts only `(id, manifest_path, activated_at, forced)`, so activation can fail on upgraded DBs.

## Fix Focus Areas
- ordvec-manifest/src/sqlite.rs[94-158]
- ordvec-manifest/src/sqlite.rs[44-92]

### Concrete fix direction
- Add a migration check for `active_manifest` similar to `verification_reports_needs_migration()`.
- If legacy columns are detected, migrate via `ALTER TABLE ... RENAME TO ...; CREATE TABLE ...; INSERT INTO new_table(...) SELECT ... FROM old_table; DROP old_table;`.
- Ensure migration runs before any `activate()` insert and is safe/idempotent.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Backslash path bypass ✓ Resolved 🐞 Bug ≡ Correctness

Description

validate_manifest_shape() skips canonical-path validation when is_manifest_path_absolute()
returns true, but is_manifest_path_absolute() treats any leading \ as absolute, letting
non-canonical backslash paths bypass v2’s forward-slash-only enforcement. On Unix this can then be
resolved as a relative path (because resolution uses Path::is_absolute()), allowing manifests with
backslash-containing bundle-relative paths to verify successfully and violating the
canonical/deterministic path contract.

Code

ordvec-manifest/src/lib.rs[R4187-4196]

+fn is_manifest_path_absolute(path: &str) -> bool {
+    if path.starts_with('/') || path.starts_with('\\') {
+        return true;
+    }
+    let bytes = path.as_bytes();
+    bytes.len() >= 3
+        && bytes[0].is_ascii_alphabetic()
+        && bytes[1] == b':'
+        && (bytes[2] == b'/' || bytes[2] == b'\\')
+}

Relevance

⭐⭐⭐ High

Team historically accepts manifest path hardening/security fixes (e.g., verifier path/TOCTOU
hardening in PRs #203, #255).

PR-#203
PR-#255

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The manifest shape validator only runs canonical checks when is_manifest_path_absolute is false;
the absolute detector currently flags any leading backslash as absolute, and later resolution uses
Path::is_absolute() (Unix treats leading backslash as relative). This combination allows a
backslash-containing path to evade canonicality validation and still be resolved/verified on Unix.

ordvec-manifest/src/lib.rs[353-385]
ordvec-manifest/src/lib.rs[4169-4196]
ordvec-manifest/src/lib.rs[2403-2444]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`is_manifest_path_absolute()` currently returns true for any path starting with a single backslash (`\`). Because canonicality checks are skipped for “absolute” paths, this allows backslash-containing paths (non-canonical in v2) to bypass `*_path_not_canonical` validation.

## Issue Context
Verification path resolution later uses `Path::is_absolute()` (platform semantics), so on Unix a leading-backslash path is treated as *relative* and can still be resolved within `base_dir`, meaning the manifest can pass verification with a non-canonical (backslash) path.

## Fix Focus Areas
- ordvec-manifest/src/lib.rs[4169-4196]
- ordvec-manifest/src/lib.rs[353-385]
- ordvec-manifest/src/lib.rs[2403-2444]

### Concrete fix direction
- Update `is_manifest_path_absolute()` to only treat UNC/verbatim forms as absolute for backslashes (e.g., `path.starts_with("\\\\")`), not a single leading `\`.
- Consider using the same absolute-path classification consistently in resolution (or explicitly rejecting Windows/UNC-looking absolute strings on non-Windows when `allow_absolute_paths` is false) so policy and canonicality decisions can’t diverge across OSes.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[Fieldnote-Echo]

Pushed three things beyond the earlier canonical-path/sqlite fixes:

• 0.7.0 lockstep version bump (c30bcf3): the release-publish invariants require the current version to have a dated changelog section with an empty [Unreleased] — schema v2 is a breaking change on top of the release-shaped 0.6.0 tree, so it forces the bump the invariant is designed to force. All 9 lockstep fields + README quickstart + THREAT_MODEL status + both lockfiles; changelog entry moved into a dated 0.7.0 section.
• crossbeam-epoch 0.9.20 in both lockfiles (RUSTSEC-2026-0204, same as fix: bump crossbeam-epoch to 0.9.20 for RUSTSEC-2026-0204 #291) so this branch's cargo-deny job passes independently of the merge base.
• Platform-proofing the new OS-matrix coverage (22bd683): Windows checked the golden fixture out through autocrlf (CRLF ≠ committed LF bytes → golden byte-identity test failed) — pinned tests/golden/ -text via .gitattributes; macOS APFS rejects non-UTF-8 filenames with EILSEQ, so the non-UTF-8 base-path test is now cfg(target_os = "linux").

[Fieldnote-Echo]

Follow-up audit round (adversarial re-review of the full stack, 3 independent reviewers + triangulation). Two fixes landed on this branch in 2640224:

• sqlite report-registry migration is now atomic (MEDIUM). The v1→v2 RENAME/CREATE/INSERT/DROP ran as a bare execute_batch (each statement auto-committed), so an interruption after the RENAME left a stray verification_reports_old that wedged every future open (rename target already exists), and two processes could race it. Now one IMMEDIATE transaction + DROP TABLE IF EXISTS ..._old first + an in-lock re-check so a lost race commits a no-op. Regression test: seeded leftover _old table still migrates.
• load_manifest_file rejects an unsupported schema_version explicitly. Real v1 manifests already failed the parse (their manifest_id/created_at trip deny_unknown_fields), but a v2-shaped doc that only mislabels its version would load and fail solely at verify — now the loader enforces the version as the CHANGELOG documents. Regression test added.
• Plus doc accuracy: golden-bytes test message no longer misattributes an .ovrq format change or an editor newline-edit as a schema-version event; CHANGELOG now says creation omits build (not build: null) and distinguishes the migrated verification_reports cache from the reset active_manifest pointer.

Security review came back clean (no high/critical under the untrusted-manifest threat model; canonicalization symmetric + fails closed, reads declared-size-bounded, SQL parameterized). Lower-severity items deferred to #293/#294/#295. ordvec-manifest --all-features 102 passed.