fix(db): hold back only the index entity whose publication keeps failing

[xav-db]

Stacked on #1156; the base branch is async-index-queue, and it should be retargeted to main once #1156 merges. Fixes HEL-956.

Problem

In #1156, if planning one entity's queued index operation failed the same way every time, its whole (scope, index, generation) stopped publishing for good. The error was retried forever, the queue filled, writes to that index got 429, and blocked_index_entity_count stayed at 0.

A real trigger exists: data written by released versions (HEL-952) can contain a self-link, and the vector insert searches never excluded the inserting node, so planning failed every time with ContainsOwner.

Fix

1. A node is never its own neighbour.

• Both insert searches and entry-point resolution now exclude the inserting node.
• A self-link already in storage is dropped in memory when its row loads, instead of failing every mutation. The row is rewritten without it only when a mutation actually changes that row.
• Each guard has a test that fails without it, and a property test over damaged graphs checks that no new self-link is ever created.

2. Only the failing entity is held back.

publish attempt fails
  FailureKind::of(error)                  // exhaustive match on HelixDbError, no string matching
    Fatal         (closed / fenced)        → stop
    Transient     (I/O, conflict, cancel)  → retry the generation with backoff   (unchanged)
    Deterministic (corrupt input, invariant)
      vector → hold the entity at the failed position
      text   → halve the epoch until one entity fails alone, then hold it
      held entity: never acknowledged or dropped; counted in blocked_index_entity_count
      retried alone after 60 s, or immediately when a newer write for it arrives
      every other entity keeps publishing

• Held state lives in memory only. After a restart, the entity is held back again on its first failed attempt instead of stalling the generation.
• When several entities fail in a row, which suggests the whole generation is damaged, attempts back off instead of spinning.

Behaviour changes

• blocked_index_entity_count (in stats and health) now also counts entities held back because their planning failed, not only entities too large to fit a publication. The docs and the proto/OpenAPI descriptions say so.
• A held entity that keeps being written still counts toward the index's shared queue budget, so it can eventually cause 429s for the whole index. This is documented, and bounding it is HEL-969.

Testing (EC2)

• fmt and clippy (workspace) are clean.

• cargo test -p db --lib: 2,058 passed.

• Release production targets (queue publication, internal contracts, text queue, index-operation queue, text correctness regressions): 92 passed.

• cargo test -p server passes.

• The release queue soak passes with 5 seeds.

• The new isolation tests cover:

  • a vector entity failing mid-batch;
  • text epoch splitting;
  • a transient failure, where nothing is held back;
  • a restart while an entity is held.

  They also cover repair writes and the 60 s retry.

• The text correctness regressions now expect a damaged root to hold back only its own entity, with the search still failing closed, instead of retrying forever.

No stored format changes. The only effect on stored data is that a damaged row loses its self-link when a mutation rewrites it.

Expected CI follow-up: the DB coverage fingerprint will change. Take the new count and sha from this PR's ARM64 "DB production thresholds" job.

The PR should wait for the production coverage baseline update so its required CI gate can pass.

Findings

1.  Coverage baseline remains unchanged ▶

Summary

The PR makes queued index publication hold back an entity whose planning fails deterministically while continuing to publish other entities. It also prevents HNSW inserts from choosing their own node as a neighbor and tolerates stored self-links during mutation.

• Vector failures identify the failing effect; text failures narrow an epoch until one entity remains.
• Failed entities remain queued and become eligible for a timed retry or a newer write.
• The production coverage baseline still needs updating for the changed source.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Select queued entities] --> B[Plan publication]
  B -->|Success| C[Commit effects and acknowledge]
  B -->|Transient failure| D[Retry batch with backoff]
  B -->|Deterministic vector failure| E[Hold identified entity]
  B -->|Deterministic text failure| F[Narrow epoch]
  F -->|One entity fails| E
  E --> G[Publish other entities]
  E --> H[Retry on newer write or timer]

Loading

_{Reviews (1) · Last reviewed commit: "docs: count failed-planning holds in blo..."}

[greptile-apps]

Coverage baseline remains unchanged

These new publication branches change the uncovered non-vector source lines used by the production coverage job, but this PR does not update the tracked count and SHA-256 in db-production-coverage-baselines.json. The job requires both values to match its generated report, so the coverage gate will fail until the baseline is updated.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
helix	🟢 Ready	View Preview	Oct 5, 2026, 10:16 AM

💡 Tip: Enable Automations to automatically generate PRs for you.