fix(upgrade-journey-raft): read post-restart sanity checks at QUORUM

[jfrancoa]

Motivation

The upgrade-journey-raft chaos suite intermittently failed its post-restart sanity checks — e.g. the dystopian cross-ref filter returning fewer than 5 authors, or the multitenancy near_text under-counting. The data was intact on the cluster; the reads were just landing on a replica that hadn't finished reconciling yet.

The root cause is a behavioral change in core: async replication became default-on for RF>1 in weaviate#11214. A CL=ONE read issued immediately after the rolling restart can be served by any single replica, including one still mid-reconciliation, which under-counts and trips the assertions.

Approach

Switch the three read-path sanity checks (Books, Authors, and the multitenancy near_text) from ConsistencyLevel.ONE to ConsistencyLevel.QUORUM. With QUORUM the read is satisfied by a majority of replicas, so it reflects converged data rather than one arbitrary replica's not-yet-reconciled view. This makes the checks assert what they actually intend — "the data survived the upgrade + restart" — instead of "this one replica happens to be caught up."

Scoped deliberately to the reads only: no assertions were changed, and the import write consistency level is left as-is, so the test still exercises the same write path and verifies the same expected counts.

Key areas for review

• apps/upgrade-journey-raft/books.py — _books_sanity_checks and _authors_sanity_checks, the two aggregate/query reads now at QUORUM
• apps/upgrade-journey-raft/multitenancy.py — _books_sanity_checks, the per-tenant near_text read now at QUORUM

Risks and mitigations

• Masking a real replication regression: QUORUM still requires a majority of replicas to agree, so it does not paper over genuine data loss or a stuck reconciliation — it only tolerates a single lagging replica, which is exactly the transient state expected right after a restart. If reconciliation were truly broken, the majority would also under-count and the check would still fail.
• Behavioral coupling to core defaults: the fix tracks core's default-on async replication for RF>1. The inline comments reference weaviate#11214 so the rationale stays discoverable if the default changes again.

Testing

This is a fix to the chaos test harness itself; correctness is demonstrated by the suite no longer flaking on the post-restart reads. The change is confined to read consistency level — no schema, import, or assertion logic was touched, so the expected-count checks remain the verification of record.

🤖 Generated with Claude Code

[orca-security-eu]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	SAST	0   0   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca