Lexical/semantic similarity clustering for memory roll-up (contingent on the semantic memory module)

[StarshipSuperjam]

The engine opened this item itself — you didn't create it.

What this is. A second cross-session relatedness signal for memory roll-up — grouping old notes by the SIMILARITY of their wording (lexical), and eventually their meaning (semantic) — deferred until the syntactic/semantic memory module exists. Issue #235 asked for a richer relatedness signal beyond same-session grouping; that is DELIVERED, via cross-session shared-topic-tag clustering (Slices A and B). A lexical text-similarity clustering signal was explored as a second, complementary signal and deliberately shelved here.

• Why it was shelved: the reuse-first mechanism (bm25 relevance via the existing scent lookup) collapses on short memory notes — a topic's defining words are not rare within a topic-heavy corpus, so bm25 gives genuinely-similar notes near-zero scores while unrelated notes pick up spurious signal from common words (there is no stopword filtering). No fixed bm25 threshold separates them.
• Why not a stopword-list fix: a content-term-overlap metric separates cleanly in testing, but it needs a hardcoded stopword list — a perpetual tweak-list to maintain, the wrong shape for the engine's floor.
• Where it belongs: the design splits memory into a lexical floor plus a separate, deferred SEMANTIC module (decision D-030). Lexical and semantic similarity clustering are that module's job — it is planned to retain chat verbatim, the substrate a real similarity signal needs — not a bm25 bolt-on to roll-up.

What happens next. This is CONTINGENT on the syntactic/semantic memory module being built; it is not actionable until then.

• When that module exists, add a cross-session lexical- (and semantic-) similarity clustering signal to roll-up's grouping, alongside the shared-topic-tag clustering already shipped.
• The roll-up grouping already carries the scaffold for it: a fixed-precedence grouping pass in detect_rollup_candidates, and a reserved sim:<id8> cross-session cluster sentinel (records.SIM_SESSION_PREFIX, already recognized by the erasure-veto credit in forget.earned_consolidated_raw). A future slice fills in the similarity metric and the pass.
• No engine behavior changes until then; shared-tag clustering remains the richer relatedness signal in v1.

More detail.

• Issue #235 (richer relatedness — delivered via tags)
• Slice A — consolidation assigns topic tags (PR #380)
• Slice B — cross-session shared-tag clustering (PR #381)
• Memory design — lexical floor + semantic module (D-030)

[StarshipSuperjam]

Reframed by the memory overhaul (pending PR #607).

This issue asks for a lexical/semantic similarity-clustering signal to group notes for roll-up. PR #607 (pending your merge) records eADR-0038, which retires the curation lifecycle — including summary-to-gist roll-up — from the memory substrate. With roll-up removed, this enhancement's premise no longer holds.

Kept open as a tracker; it will be closed by the slice that removes the roll-up machinery (the curation-removal slice of the overhaul). No work is owed here in the meantime.

[StarshipSuperjam]

Closing this: the thing it was waiting for has arrived, and the thing it was for is being removed.

This item was shelved until a semantic memory module existed, because a real similarity signal needs meaning, not word-overlap. That module is now being built — it is the second half of the memory overhaul's current slice, adding a local similarity index behind the existing search interface.

But the work described here was to enrich roll-up's grouping — the pass that folds older summaries into a shorter combined summary. The overhaul's recorded end state (eADR-0038) makes the exact conversation the canonical record and retires the whole automatic summarising layer above it, roll-up included. So the consumer this signal was going to feed is scheduled for deletion, and a better grouping signal for a layer that is going away is work that would be thrown out on arrival.

What that means concretely:

• The scaffold comes out. The reserved sim: cross-session cluster marker (records.SIM_SESSION_PREFIX) has never had a producer — nothing in the engine mints one. It is removed in the current slice, along with the other pieces that were waiting on a stage that changed shape underneath them.
• Nothing of value is lost. Cross-session grouping by shared topic tag — the richer relatedness signal issue Add a richer relatedness signal to memory roll-up (beyond same-session grouping) #235 actually asked for — shipped and works. It stays until the summarising layer itself is retired.
• The capability this was really reaching for is being delivered, elsewhere. Finding related material by meaning rather than by wording is exactly what the semantic search leg does — but at recall time, where the overhaul puts all meaning-work, rather than baked into a maintenance pass that runs behind your back.

No action is owed. If grouping by similarity turns out to matter again once the substrate settles, it would be a new item against the new shape, not this one.