Skip to content

fix: preserve surface hashes and scope evidence reuse by memory file - #238

Open
FocalFactotum wants to merge 5 commits into
kunchenguid:mainfrom
FocalFactotum:fm/upstream-bp-196
Open

fix: preserve surface hashes and scope evidence reuse by memory file#238
FocalFactotum wants to merge 5 commits into
kunchenguid:mainfrom
FocalFactotum:fm/upstream-bp-196

Conversation

@FocalFactotum

@FocalFactotum FocalFactotum commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Preserve the full memory-surface hash for targeted runs while selecting only the named memory file for editing.
  • Require cached evidence to match the analyzed memory file, allowing primary-target reuse while reanalyzing when switching to a separate file; bump the analysis-index version.
  • Document cache behavior and add regression coverage for target transitions, evidence folding, and restricted staging.

Risk Assessment

✅ Low: The change preserves whole-surface hashing, distinguishes selected-memory evidence freshness, and retains narrow staging and proposal gates with behavioral regression coverage.

Testing

Focused tests passed after correcting disposable CommonJS fixture setup. Real CLI checks demonstrated cache reuse, cross-file evidence rejection, and whole-surface invalidation. CLI artifacts were captured; missing acpx prevented live fresh analysis and completed synthesis. No broad suite or static checks ran.

  • Live validation: ⚠️ inconclusive - 3 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Target the primary memory file and retain whole-surface cached evidence through proposal folding ✅ pass live cli-scenarios.json: Primary-target cached analysis; Primary-target proposal admits full-surface evidence
Switch memory files and reject evidence analyzed against the other instruction index ✅ pass live cli-scenarios.json: Secondary target rejects primary instruction evidence; Untargeted proposal rejects secondary instruction evidence
Edit the non-target memory file and invalidate previously cached primary evidence ✅ pass live cli-scenarios.json: Editing non-target memory invalidates primary evidence
Reanalyze after switching memory files in both directions and obtain foldable evidence ⏸️ untested no The focused test passed with a stand-in harness. Live cached analysis worked, but fresh model analysis requires acpx, which is absent from PATH. Downloads are forbidden. Provide a repository-local acp…
Complete a targeted proposal while staging only the allowed write surface ⏸️ untested no The real CLI admitted evidence and reached synthesis, then stopped because acpx was unavailable. Targeted proposal and staging tests passed with a stand-in harness. Provide repository-local acpx and i…
Evidence: Live CLI outputs and persisted fold summaries 
[
  {
    "label": "Primary-target cached analysis",
    "args": [
      "analyze",
      "--target",
      "AGENTS.md"
    ],
    "status": 0,
    "stdout": "{\n  \"memoryFile\": \"AGENTS.md\",\n  \"transcripts\": 1,\n  \"summary\": {\n    \"total\": 1,\n    \"cached\": 1,\n    \"analyzed\": 0,\n    \"skipped\": 0,\n    \"failed\": 0,\n    \"usage\": [],\n    \"staleMemoryHash\": 0,\n    \"quotesNotInTrace\": 0\n  }\n}\n",
    "stderr": "targeting AGENTS.md; existing skills are read-only this run\n",
    "summary": null
  },
  {
    "label": "Primary-target proposal admits full-surface evidence",
    "args": [
      "propose",
      "--target",
      "AGENTS.md",
      "--synthesis-agent",
      "pi"
    ],
    "status": 1,
    "stdout": "",
    "stderr": "targeting AGENTS.md; existing skills are read-only this run\n· synthesizing with pi effort=high\nerror cannot verify acpx adapter configuration for pi: spawn ~/.no-mistakes/worktrees/4c4fc99317e2/01M60RCWD0C6YH5ZS3ATP6C7QT/.phase-test-tmp/live/unavailable-acpx ENOENT\n  upgrade acpx or omit the model and effort override\n",
    "summary": {
      "version": 1,
      "generatedAt": "2026-10-28T08:02:54.010Z",
      "analyzedSessions": 1,
      "analyzedByInteraction": {
        "interactive": 0,
        "non-interactive": 1
      },
      "totals": {
        "positive": 1,
        "negative": 0,
        "gapSightings": 0,
        "gapClusters": 0,
        "reportOnlyGapClusters": 0,
        "reportOnlyByReason": {
          "majorityOrchestration": 0,
          "belowFloorMixed": 0,
          "tooFewProjects": 0
        },
        "droppedGapSingletons": 0,
        "orchestrationGapSightings": 0,
        "instructionsWithNegatives": 0,
        "usedRawTranscript": 0,
        "crossSurfaceDuplicates": 0
      },
      "instructions": [
        {
          "instruction": "AG-001",
          "positive": 1,
          "negative": 0,
          "harmSessions": 0,
          "sessions": 1,
          "relevance": 1,
          "relevanceByInteraction": {
            "interactive": 0,
            "non-interactive": 1
          },
          "relevanceByProject": {
            "~/.no-mistakes/worktrees/4c4fc99317e2/01M60RCWD0C6YH5ZS3ATP6C7QT/.phase-test-tmp/live/repo": 1
          },
          "tokens": 8,
          "section": "Instructions",
          "known": true,
          "parentId": null,
          "nonCompliance": 0,
          "nonComplianceSessions": 0,
          "quotes": [
            {
              "polarity": "positive",
              "text": "Ran make build.",
              "effect": "followed",
              "moment": "start",
              "source": "pi · demo · 2026-10-28"
            }
          ]
        }
      ],
      "sources": [
        "pi · demo · 2026-10-28"
      ],
      "sourceProjects": {
        "pi · demo · 2026-10-28": "~/.no-mistakes/worktrees/4c4fc99317e2/01M60RCWD0C6YH5ZS3ATP6C7QT/.phase-test-tmp/live/repo"
      },
      "sourceObservers": {
        "pi · demo · 2026-10-28": "ac8c17454bc385831d6e3a475230675e51b56852ff85c244b34ee79286bcc843"
      },
      "parentHarmSessions": {},
      "gaps": [],
      "crossSurfaceDuplicates": [],
      "reportOnlyGaps": [],
      "oversized": [],
      "consolidation": {
        "skipped": "fewer than two open gaps"
      }
    }
  },
  {
    "label": "Secondary target rejects primary instruction evidence",
    "args": [
      "propose",
      "--target",
      "CLAUDE.md"
    ],
    "status": 1,
    "stdout": "",
    "stderr": "targeting CLAUDE.md; existing skills are read-only this run\nerror no loss calculated yet: nothing to run gradient descent on\n  run `backpass analyze` first, or `backpass` for the full pass\n",
    "summary": {
      "version": 1,
      "generatedAt": "2026-10-28T08:02:54.125Z",
      "analyzedSessions": 0,
      "analyzedByInteraction": {
        "interactive": 0,
        "non-interactive": 0
      },
      "totals": {
        "positive": 0,
        "negative": 0,
        "gapSightings": 0,
        "gapClusters": 0,
        "reportOnlyGapClusters": 0,
        "reportOnlyByReason": {
          "majorityOrchestration": 0,
          "belowFloorMixed": 0,
          "tooFewProjects": 0
        },
        "droppedGapSingletons": 0,
        "orchestrationGapSightings": 0,
        "instructionsWithNegatives": 0,
        "usedRawTranscript": 0,
        "crossSurfaceDuplicates": 0
      },
      "instructions": [
        {
          "instruction": "AG-001",
          "positive": 0,
          "negative": 0,
          "harmSessions": 0,
          "sessions": 0,
          "relevance": 0,
          "relevanceByInteraction": {
            "interactive": 0,
            "non-interactive": 0
          },
          "tokens": 6,
          "section": "Separate",
          "known": true,
          "parentId": null,
          "nonCompliance": 0,
          "nonComplianceSessions": 0,
          "quotes": []
        }
      ],
      "sources": [],
      "sourceProjects": {},
      "sourceObservers": {},
      "parentHarmSessions": {},
      "gaps": [],
      "crossSurfaceDuplicates": [],
      "reportOnlyGaps": [],
      "oversized": [],
      "consolidation": {
        "skipped": "fewer than two open gaps"
      }
    }
  },
  {
    "label": "Secondary cached analysis",
    "args": [
      "analyze",
      "--target",
      "CLAUDE.md"
    ],
    "status": 0,
    "stdout": "{\n  \"memoryFile\": \"CLAUDE.md\",\n  \"transcripts\": 1,\n  \"summary\": {\n    \"total\": 1,\n    \"cached\": 1,\n    \"analyzed\": 0,\n    \"skipped\": 0,\n    \"failed\": 0,\n    \"usage\": [],\n    \"staleMemoryHash\": 0,\n    \"quotesNotInTrace\": 0\n  }\n}\n",
    "stderr": "targeting CLAUDE.md; existing skills are read-only this run\n",
    "summary": {
      "version": 1,
      "generatedAt": "2026-10-28T08:02:54.125Z",
      "analyzedSessions": 0,
      "analyzedByInteraction": {
        "interactive": 0,
        "non-interactive": 0
      },
      "totals": {
        "positive": 0,
        "negative": 0,
        "gapSightings": 0,
        "gapClusters": 0,
        "reportOnlyGapClusters": 0,
        "reportOnlyByReason": {
          "majorityOrchestration": 0,
          "belowFloorMixed": 0,
          "tooFewProjects": 0
        },
        "droppedGapSingletons": 0,
        "orchestrationGapSightings": 0,
        "instructionsWithNegatives": 0,
        "usedRawTranscript": 0,
        "crossSurfaceDuplicates": 0
      },
      "instructions": [
        {
          "instruction": "AG-001",
          "positive": 0,
          "negative": 0,
          "harmSessions": 0,
          "sessions": 0,
          "relevance": 0,
          "relevanceByInteraction": {
            "interactive": 0,
            "non-interactive": 0
          },
          "tokens": 6,
          "section": "Separate",
          "known": true,
          "parentId": null,
          "nonCompliance": 0,
          "nonComplianceSessions": 0,
          "quotes": []
        }
      ],
      "sources": [],
      "sourceProjects": {},
      "sourceObservers": {},
      "parentHarmSessions": {},
      "gaps": [],
      "crossSurfaceDuplicates": [],
      "reportOnlyGaps": [],
      "oversized": [],
      "consolidation": {
        "skipped": "fewer than two open gaps"
      }
    }
  },
  {
    "label": "Untargeted proposal rejects secondary instruction evidence",
    "args": [
      "propose"
    ],
    "status": 1,
    "stdout": "",
    "stderr": "warn CLAUDE.md is a separate memory file and will NOT be updated - only AGENTS.md is optimized. To cover both, consolidate: move its content into AGENTS.md and make CLAUDE.md a pointer (a single line: @AGENTS.md).\nerror no loss calculated yet: nothing to run gradient descent on\n  run `backpass analyze` first, or `backpass` for the full pass\n",
    "summary": {
      "version": 1,
      "generatedAt": "2026-10-28T08:02:54.289Z",
      "analyzedSessions": 0,
      "analyzedByInteraction": {
        "interactive": 0,
        "non-interactive": 0
      },
      "totals": {
        "positive": 0,
        "negative": 0,
        "gapSightings": 0,
        "gapClusters": 0,
        "reportOnlyGapClusters": 0,
        "reportOnlyByReason": {
          "majorityOrchestration": 0,
          "belowFloorMixed": 0,
          "tooFewProjects": 0
        },
        "droppedGapSingletons": 0,
        "orchestrationGapSightings": 0,
        "instructionsWithNegatives": 0,
        "usedRawTranscript": 0,
        "crossSurfaceDuplicates": 0
      },
      "instructions": [
        {
          "instruction": "AG-001",
          "positive": 0,
          "negative": 0,
          "harmSessions": 0,
          "sessions": 0,
          "relevance": 0,
          "relevanceByInteraction": {
            "interactive": 0,
            "non-interactive": 0
          },
          "tokens": 8,
          "section": "Instructions",
          "known": true,
          "parentId": null,
          "nonCompliance": 0,
          "nonComplianceSessions": 0,
          "quotes": []
        }
      ],
      "sources": [],
      "sourceProjects": {},
      "sourceObservers": {},
      "parentHarmSessions": {},
      "gaps": [],
      "crossSurfaceDuplicates": [],
      "reportOnlyGaps": [],
      "oversized": [],
      "consolidation": {
        "skipped": "fewer than two open gaps"
      }
    }
  },
  {
    "label": "Editing non-target memory invalidates primary evidence",
    "args": [
      "propose",
      "--target",
      "AGENTS.md"
    ],
    "status": 1,
    "stdout": "",
    "stderr": "targeting AGENTS.md; existing skills are read-only this run\nerror no loss calculated yet: nothing to run gradient descent on\n  run `backpass analyze` first, or `backpass` for the full pass\n",
    "summary": {
      "version": 1,
      "generatedAt": "2026-10-28T08:02:54.380Z",
      "analyzedSessions": 0,
      "analyzedByInteraction": {
        "interactive": 0,
        "non-interactive": 0
      },
      "totals": {
        "positive": 0,
        "negative": 0,
        "gapSightings": 0,
        "gapClusters": 0,
        "reportOnlyGapClusters": 0,
        "reportOnlyByReason": {
          "majorityOrchestration": 0,
          "belowFloorMixed": 0,
          "tooFewProjects": 0
        },
        "droppedGapSingletons": 0,
        "orchestrationGapSightings": 0,
        "instructionsWithNegatives": 0,
        "usedRawTranscript": 0,
        "crossSurfaceDuplicates": 0
      },
      "instructions": [
        {
          "instruction": "AG-001",
          "positive": 0,
          "negative": 0,
          "harmSessions": 0,
          "sessions": 0,
          "relevance": 0,
          "relevanceByInteraction": {
            "interactive": 0,
            "non-interactive": 0
          },
          "tokens": 8,
          "section": "Instructions",
          "known": true,
          "parentId": null,
          "nonCompliance": 0,
          "nonComplianceSessions": 0,
          "quotes": []
        }
      ],
      "sources": [],
      "sourceProjects": {},
      "sourceObservers": {},
      "parentHarmSessions": {},
      "gaps": [],
      "crossSurfaceDuplicates": [],
      "reportOnlyGaps": [],
      "oversized": [],
      "consolidation": {
        "skipped": "fewer than two open gaps"
      }
    }
  }
]
Evidence: Focused automated tests 
TAP version 13
# warn CLAUDE.md is a separate memory file and will NOT be updated - only AGENTS.md is optimized. To cover both, consolidate: move its content into AGENTS.md and make CLAUDE.md a pointer (a single line: @AGENTS.md).
# warn CLAUDE.md is a separate memory file and will NOT be updated - only AGENTS.md is optimized. To cover both, consolidate: move its content into AGENTS.md and make CLAUDE.md a pointer (a single line: @AGENTS.md).
# warn CLAUDE.md is a separate memory file and will NOT be updated - only AGENTS.md is optimized. To cover both, consolidate: move its content into AGENTS.md and make CLAUDE.md a pointer (a single line: @AGENTS.md).
# Subtest: selected memory transitions reanalyze under the same surface hash and remain foldable
ok 1 - selected memory transitions reanalyze under the same surface hash and remain foldable
  ---
  duration_ms: 926.201852
  type: 'test'
  ...
# Subtest: a memory target reuses whole-surface evidence without staging the other memory file
ok 2 - a memory target reuses whole-surface evidence without staging the other memory file
  ---
  duration_ms: 297.374419
  type: 'test'
  ...
# warn CLAUDE.md is a separate memory file and will NOT be updated - only AGENTS.md is optimized. To cover both, consolidate: move its content into AGENTS.md and make CLAUDE.md a pointer (a single line: @AGENTS.md).
# Subtest: memory targets retain the full surface hash while selecting the named file
ok 3 - memory targets retain the full surface hash while selecting the named file
  ---
  duration_ms: 8.677189
  type: 'test'
  ...
# Subtest: a memory-file target stages no existing skill, still measures a new extract, and refuses re-creating a skill
ok 4 - a memory-file target stages no existing skill, still measures a new extract, and refuses re-creating a skill
  ---
  duration_ms: 19.187025
  type: 'test'
  ...
# Subtest: a skill target stages that skill only; the proposal keeps the whole-surface budget and refuses every other file
ok 5 - a skill target stages that skill only; the proposal keeps the whole-surface budget and refuses every other file
  ---
  duration_ms: 17.807748
  type: 'test'
  ...
1..5
# tests 5
# suites 0
# pass 5
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 1018.126482
- Outcome: ⚠️ 1 warning across 1 run (2m36s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

⏭️ **Rebase** - skipped

Step was skipped.

✅ **Review** - passed

✅ No issues found.

⚠️ **Test** - 1 warning
  • ⚠️ live validation verdict: inconclusive (3 of 5 scenarios were driven live against the product); untested: Reanalyze after switching memory files in both directions and obtain foldable evidence, Complete a targeted proposal while staging only the allowed write surface
  • Live validation: ⚠️ inconclusive - 3 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Target the primary memory file and retain whole-surface cached evidence through proposal folding ✅ pass live cli-scenarios.json: Primary-target cached analysis; Primary-target proposal admits full-surface evidence
Switch memory files and reject evidence analyzed against the other instruction index ✅ pass live cli-scenarios.json: Secondary target rejects primary instruction evidence; Untargeted proposal rejects secondary instruction evidence
Edit the non-target memory file and invalidate previously cached primary evidence ✅ pass live cli-scenarios.json: Editing non-target memory invalidates primary evidence
Reanalyze after switching memory files in both directions and obtain foldable evidence ⏸️ untested no The focused test passed with a stand-in harness. Live cached analysis worked, but fresh model analysis requires acpx, which is absent from PATH. Downloads are forbidden. Provide a repository-local acp…
Complete a targeted proposal while staging only the allowed write surface ⏸️ untested no The real CLI admitted evidence and reached synthesis, then stopped because acpx was unavailable. Targeted proposal and staging tests passed with a stand-in harness. Provide repository-local acpx and i…
  • node --import ./test/helpers/hermetic-config.js --test --test-name-pattern='selected memory transitions|a memory target reuses whole-surface|memory targets retain|a memory-file target|a skill target' test/analyze-reuse.test.js test/synthesis-cli.test.js test/target.test.js with workspace-local TMPDIR
  • node .phase-test-tmp/live.mjs: six isolated real CLI invocations using disposable transcripts and seeded evidence
  • command -v acpx, command -v pi, command -v claude
  • Removed disposable fixtures and confirmed git status --short was clean
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@greptile-apps

greptile-apps Bot commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Reviews (1) · Last reviewed commit: "no-mistakes(document): Replace stale tar..." · Reviewed by Greptile

@kunchenguid kunchenguid mentioned this pull request Oct 28, 2026
Open
@kunchenguid

kunchenguid commented Oct 29, 2026

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate: thanks @FocalFactotum for picking up #196. I read the diff on 49b2548. Keeping the full memory-surface hash for a memory target and narrowing only staging and the proposal gate is the fix direction recorded on the issue, and scoping reuse by the analyzed memory file is a sensible guard, since a separate memory file has its own instruction IDs. CI is green.

This is waiting on you for two things:

  1. Live evidence for the two untested scenarios in your table: reanalyzing after switching memory targets in both directions, and a complete targeted proposal that stages only the allowed file. Please run them against a real setup and post the output here.
  2. A question on ANALYSIS_INDEX_VERSION going from 4 to 5. That makes every user reanalyze every transcript on upgrade, including untargeted runs whose keys this PR does not change, and each reanalysis is a paid model call. As far as I can tell, old records cannot be wrongly reused under the new rules, because old separate-file records were keyed to the narrowed hash and the new path check runs on every reuse. If there is a case where an old record would be wrongly reused, please name it here; otherwise please drop the bump.

No rebase needed from you.

VISION per-rule:

  • Evidence is the only currency: aligns (whole-surface evidence stays usable for the primary target; evidence from one file's instruction index is never folded into another's).
  • The human owns the weights: aligns (the target still names the one file that may be written; nothing widens).
  • Nothing the model says is taken on faith: aligns.
  • The budget is the constraint: aligns with the version-bump question above (unneeded reanalysis costs money but does not grow the file).
  • It reads what you already have and owns nothing: aligns.
  • Failure is loud and named: aligns (removes the misleading "no loss calculated" after an untargeted run).
  • Scope: aligns.

Contract-class: restore. The README promised that a targeted run keeps whole-surface analysis, evidence, and state; that was specified and broken for multi-file surfaces. The README edit for separate-file targets narrows a promise the code could never meet, since another file's instruction IDs differ.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@FocalFactotum @kunchenguid