diff --git a/.claude/skills/backscroll/SKILL.md b/.claude/skills/backscroll/SKILL.md index aab4193..af46f87 100644 --- a/.claude/skills/backscroll/SKILL.md +++ b/.claude/skills/backscroll/SKILL.md @@ -10,6 +10,11 @@ allowed-tools: Backscroll is the primary local episodic index for coding-agent work. Run it before starting feature, bug, test, refactor, or decision work that may have history. A hit is evidence from indexed rows. An empty result is only query/index uncertainty: it does not prove the event, file, or decision never existed. +Every operational command validates active manifests and attempts one incremental +sync before executing. Session, plan, and Markdown files are ingestion inputs; +SQLite is the perennial record used by search, list, patterns, status, and validate. +Use `--source-path` on search as a filter, paired with query text, for database-backed retrieval scoped to a known input path. + ## 1) Preflight (required) ```bash @@ -36,7 +41,7 @@ Invoke `/skill:backscroll` automatically for: - Writing tests: query the test subject and related module. - Refactoring: query the module, interface, or previous pattern. - Decision questions: query the decision topic and alternatives. -- Debugging execution: query command names, paths, flags, exit codes, and use `backscroll search` with `--content-type tool`. +- Debugging execution: query command names, paths, flags, exit codes, and use the search command with `--content-type tool` and query text. Spanish trigger equivalents include "ya lo hicimos", "qué hicimos con", "qué error dio", "dónde corrí", and "qué decidimos". Do not wait for explicit recall requests; missed lookup cost is rework and duplicate decisions. @@ -54,7 +59,7 @@ where `` is the OS config directory, or `BACKSCROLL_CONFIG_DIR`. The Use machine-readable, budgeted output: -- `--robot`: emits `result_N_field=value` lines. +- Robot mode on search emits `result_N_field=value` lines; search string values escape backslash as `\\`, carriage return as `\r`, and newline as `\n`. - `--fields minimal`: returns `source_path`, `snippet`, `score`, `role`, and `timestamp`. - `--fields full`: use only for a selected source-path drill. - `--max-tokens `: declare and enforce the output budget. @@ -128,7 +133,7 @@ backscroll search "go test" --all-projects --content-type tool --robot --fields ```bash SOURCE_PATH="" -backscroll search "" --all-projects --indexed-only --source-path "$SOURCE_PATH" --robot --fields full --max-tokens 4000 +backscroll search --text "$QUERY" --all-projects --source-path "$SOURCE_PATH" --robot --fields full --max-tokens 4000 ``` 2. **Use the artifact's vocabulary.** For transcripts, logs, reports, and pasted artifacts, query literal speaker names, boilerplate, IDs, exact errors, paths, and the artifact language. A translated or paraphrased query is secondary evidence only. @@ -140,20 +145,20 @@ backscroll search --help backscroll list --help ``` -4. **Two empty searches prove nothing.** Before concluding content is absent from the index: retry with artifact-literal terms; broaden to `--all-projects`; if a path or UUID is known, probe existing indexed rows; run one normal search without `--indexed-only` so normal search autosync can run; repeat the indexed-only probe; then collect diagnostics and report the gap. +4. **Two empty searches prove nothing.** Before concluding content is absent from the index: retry with artifact-literal terms; broaden to `--all-projects`; if a path or UUID is known, drill down with search `--source-path` plus query text; rely on mandatory startup sync to refresh active manifests; then collect diagnostics and report the gap. ```bash -backscroll search "literal speaker or error" --all-projects --indexed-only --robot --fields minimal --max-tokens 2000 -backscroll search "" --all-projects --indexed-only --source-path "*SESSION-UUID*" --json --fields minimal --limit 1 backscroll search "literal speaker or error" --all-projects --robot --fields minimal --max-tokens 2000 -backscroll search "" --all-projects --indexed-only --source-path "*SESSION-UUID*" --json --fields minimal --limit 1 +backscroll search --text "artifact literal" --all-projects --source-path "*SESSION-UUID*" --json --fields minimal --limit 1 +backscroll search "literal speaker or error" --all-projects --content-type tool --robot --fields minimal --max-tokens 2000 +backscroll search --text "$QUERY" --all-projects --source-path "*SESSION-UUID*" --json --fields full --max-tokens 4000 backscroll status -backscroll validate --indexed-only +backscroll validate ``` Report the source path or UUID, literal probes, scopes used, and full diagnostic output as an indexing gap when the probe remains absent. -5. **Raw-file boundary.** `cat`, `jq`, Python, or direct `backscroll read` is not a normal retrieval fallback. Do not use raw JSONL parsing, directory listings for session hunting, or direct file reads unless the user explicitly authorizes indexing-bug diagnosis after you report the gap and the indexed commands attempted. +5. **Raw-file boundary.** `cat`, `jq`, Python, or filesystem session hunting is not a normal retrieval fallback. Do not use raw JSONL parsing, directory listings for session hunting, or direct file inspection unless the user explicitly authorizes indexing-bug diagnosis after you report the gap and the indexed commands attempted. Database-backed search with `--source-path` and query text is the supported drill-down path. ## 6) Degradation and troubleshooting @@ -161,12 +166,12 @@ Report the source path or UUID, literal probes, scopes used, and full diagnostic ```bash backscroll status -backscroll validate --indexed-only +backscroll validate ``` If a search warns about scope, content type, or compatibility, follow the hint and rerun a corrected current command once. -**No results:** follow the hard rules: literal artifact vocabulary, all-projects scope, source-path/UUID probe, one normal search for autosync, repeated indexed-only probe, then status and validate. Report uncertainty; do not convert empty rows into proof of absence. +**No results:** follow the hard rules: literal artifact vocabulary, all-projects scope, source-path/UUID probe through mandatory startup sync, then status and validate. Report uncertainty; do not convert empty rows into proof of absence. **Tool-query tokenizer limits:** the tool index uses a trigram tokenizer. Prefer exact flags, paths, command names, and error fragments of at least three characters, for example `"--content-type tool"`, `"go test"`, or `"BUSY"`. @@ -209,7 +214,7 @@ backscroll search "query" --all-projects --robot --fields minimal --max-tokens 2 ## Pattern discovery: census, not retrieval -`backscroll search` answers “find what I can already name.” For discovery — “what recurs that nobody named?” — use census commands. BM25 pattern queries usually yield anecdotes, not counts. +Search answers “find what I can already name.” For discovery — “what recurs that nobody named?” — use census commands. BM25 pattern queries usually yield anecdotes, not counts. | Question | Command | |---|---| @@ -223,7 +228,7 @@ Agent-grade census output: ```bash backscroll patterns --kind corrections --pending --batch 50 --robot -backscroll patterns --kind commands --all-projects --indexed-only --robot +backscroll patterns --kind commands --all-projects --robot ``` Interpret the complete table returned. The census did the counting; the agent's job is judgment, not sampling. diff --git a/.claude/skills/backscroll/ref-context-mode.md b/.claude/skills/backscroll/ref-context-mode.md index 596b0f2..cb56c40 100644 --- a/.claude/skills/backscroll/ref-context-mode.md +++ b/.claude/skills/backscroll/ref-context-mode.md @@ -2,11 +2,13 @@ Use this only for `/skill:backscroll --context`. Produce a recovery brief with: Backscroll evidence, optional Rootline live state, and gaps. +Backscroll retrieval uses active manifests, mandatory startup sync, perennial SQLite, and database-backed query. Raw `cat`, `jq`, Python, or filesystem session hunting is not a normal retrieval fallback; drill into known paths with search `--source-path` plus query text. + ## Required Backscroll Retrieval ```bash -backscroll validate --indexed-only -backscroll status --indexed-only +backscroll validate +backscroll status backscroll list --limit 10 --all-projects --json ``` @@ -23,6 +25,13 @@ If this returns no useful results, run one broader session search: backscroll search "$PROJECT_SLUG" --source session --all-projects --max-tokens 4000 ``` +If a result includes a useful `source_path`, drill into it before leaving the indexed boundary: + +```bash +SOURCE_PATH="" +backscroll search --text "$QUERY" --source-path "$SOURCE_PATH" --all-projects --max-tokens 4000 +``` + For empty results or suspected gaps, follow the main skill's search discipline rather than raw-file fallback. ## Optional Rootline State diff --git a/CLAUDE.md b/CLAUDE.md index 5ae8b5c..76bb3d3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,7 @@ Backscroll is a Go CLI tool that indexes Claude Code, Pi, OpenCode, and declarat **Status**: Go port complete — `main` branch is the active Go implementation. The Rust implementation is frozen in the `v0` branch. -Implemented: `internal/config`, `internal/input_config`, `internal/models`, `internal/readers`, `internal/sync`, `internal/tagging`, `internal/plans`, `internal/sources`, `internal/storage`, `internal/projects`, `internal/reader`, `internal/templates`, `internal/corrections`, `internal/categories`, `internal/sequences`, `internal/compat`, `internal/recovery`. CLI commands in `cmd/backscroll/` (11 v2 commands via cobra). +Implemented: `internal/config`, `internal/input_config`, `internal/models`, `internal/readers`, `internal/sync`, `internal/tagging`, `internal/plans`, `internal/sources`, `internal/storage`, `internal/projects`, `internal/templates`, `internal/corrections`, `internal/categories`, `internal/sequences`, `internal/chunking`, `internal/embedding`, `internal/hybrid`, `internal/compat`, `internal/recovery`. CLI commands in `cmd/backscroll/` (10 v2 commands via cobra). Stack: cobra, go-toml/v2, goldmark, modernc.org/sqlite (pure Go, no CGO), stdlib testing. @@ -32,7 +32,7 @@ Run a single test: `go test -run TestName ./internal/...` **Coverage**: the release-blocking gate is **aggregate** statement coverage ≥85%, checked identically by CI (crossbeam `go-ci` light profile) and the local pre-push hook via `just ci`. Per-package floors in `.coverage-floors.toml` (default 85%) remain available as an advisory quality check via `just coverage-check` (pkcov), but are **not** release-blocking — individual packages may dip below 85% as long as the aggregate holds. backscroll conforms to [coverage-spec v1.0](https://github.com/pablontiv/picokit/blob/main/docs/coverage-spec.md). -Tests use stdlib `testing` + subprocess or direct `run()` invocation. Unit tests are co-located in each package. Integration tests in `cmd/backscroll/main_test.go` (CLI integration via direct `run()` invocation). Additional unit tests: `internal/storage/unit_test.go`, `internal/sync/noise_test.go`, `internal/reader/semantic_test.go`. The push gate and CI both enforce aggregate coverage ≥85% (`just ci`); `just coverage-check` (pkcov per-package floors) is advisory. Tests must be hermetic — scrub machine state with `testEnv(t)` / `t.Setenv("HOME", tempDir)` so they pass in CI's bare environment, which `just ci` reproduces via a scrubbed `HOME`/`BACKSCROLL_CONFIG_DIR`. +Tests use stdlib `testing` + subprocess or direct `run()` invocation. Unit tests are co-located in each package. Integration tests in `cmd/backscroll/main_test.go` (CLI integration via direct `run()` invocation). Additional unit tests include `internal/storage/unit_test.go` and `internal/sync/noise_test.go`. The push gate and CI both enforce aggregate coverage ≥85% (`just ci`); `just coverage-check` (pkcov per-package floors) is advisory. Tests must be hermetic — scrub machine state with `testEnv(t)` / `t.Setenv("HOME", tempDir)` so they pass in CI's bare environment, which `just ci` reproduces via a scrubbed `HOME`/`BACKSCROLL_CONFIG_DIR`. ## Architecture @@ -41,18 +41,17 @@ Tests use stdlib `testing` + subprocess or direct `run()` invocation. Unit tests ``` cmd/backscroll/ ├── main.go — entrypoint; run(stdout, stderr, args) for testability -├── list.go — list command (v2: --input, --order, --type, --tool) -├── search.go — search command (v2: --text, --input) -├── read.go — read command (v2: --path, --tail, --semantic, --pretty) -├── patterns.go — patterns command (v2: --kind commands|failures|templates|sequences|corrections [--pending] [--batch N] [--trend], --project, --tag, --min-support, --min-confidence, --min-length, --max-length, --json, --robot) +├── list.go — list command (v2: --project, --all-projects, --recent N, --order, --limit, --offset, --json, --robot) +├── search.go — search command (v2: --text, --project, --all-projects, --source, --source-path, --after, --before, --role, --content-type, --tag, --fields, --max-tokens, --lexical-only, --similarity-threshold, --json, --robot) +├── patterns.go — patterns command (v2: --kind commands|failures|templates|sequences|corrections [--pending] [--batch N] [--trend], --project, --all-projects, --tag, --min-support, --min-confidence, --min-length, --max-length, --json, --robot) ├── annotate.go — annotate command (F3b: --uuid --kind --label; validates message existence; upsert semantics) ├── recover.go — recover command (--from, --dry-run; lossless active+stranded database union) ├── status.go — status command -├── validate.go — validate command (--indexed-only) +├── validate.go — validate command (--json) ├── rebuild.go — rebuild command (replaces reindex) ├── purge.go — purge command ├── config.go — config command (shows effective config + inputs) -└── sync_helpers.go — shared auto-sync helpers (maybeAutoSync, runSync) +└── sync_helpers.go — shared startup sync helpers (maybeAutoSync, reader registry wiring) internal/ ├── config/ — config resolution: backscroll.toml → ~/.config → env → defaults ├── compat/ — stateless schema-shape inspection, release lineage catalog, migration plans, and canonical recovery planning @@ -63,30 +62,33 @@ internal/ ├── plans/ — Markdown plan parser (split by ## headers, goldmark) ├── sources/ — external source parsers (ke, decision, memory, rule, spec, backlog) + SourceRegistry ├── projects/ — project identity registry: LoadGlobalRegistry(), Identify(), LoadLocalHint() -├── reader/ — direct reading and filtering of individual session files ├── readers/ — SessionReader interface, Registry, ClaudeReader (text+tool_use+tool_result), PiReader (text+toolCall+custom results), OpenCodeReader (text+tool state.input+state.output), MarkdownDocumentReader (`markdown_document`), MarkdownSectionsReader (`markdown_sections`); toolfmt serializer ├── recovery/ — stranded database recovery orchestration, verified active+stranded union, durable backup, and atomic replacement ├── templates/ — F2 Drain-inspired miner: Miner, ProcessLine, ExtractErrorLines, deterministic signature via SHA256 ├── corrections/ — F3 correction detection: bilingual lexicon, interrupt flags, denial heuristics, rephrase-similarity; detector registry + implementations ├── categories/ — F4 category map loader (rule engine, versioning, tool→category mapping) with embedded default preset +├── chunking/ — Token-aware text chunking helpers for embedding preparation +├── embedding/ — Embedding provider interfaces, mock provider, and ONNX provider implementation +├── hybrid/ — Reciprocal Rank Fusion helpers for merged lexical/vector retrieval ├── sequences/ — F4 PrefixSpan mining (deterministic discovery of frequent tool-call sequences per session) └── storage/ — SQLite adapter (dual FTS5 indexes: tool_fts + messages_fts, BM25, WAL mode, migrations v1–v13, search_items, session_tags, tool_events, message_templates, template_matches, correction_signals, annotations, AggregateCommands, AggregateFailures, AggregateTemplates, AggregateCorrections, UpsertAnnotation, LoadToolSequences) ``` -Eleven v2 CLI commands: `list [--project] [--all-projects] [--order timestamp:desc|asc] [--limit] [--offset] [--json]`, `search [--text ] [--project] [--all-projects] [--after] [--before] [--limit] [--offset] [--indexed-only] [--json]`, `read --path [--tail ] [--semantic] [--pretty]`, `patterns --kind commands|failures|templates|sequences|corrections [--pending] [--batch N] [--project] [--all-projects] [--tag] [--trend] [--min-support N] [--min-confidence F] [--min-length N] [--max-length N] [--limit] [--offset] [--indexed-only] [--json] [--robot]`, `annotate --uuid --kind --label [--path

--ordinal ]`, `recover --from [--dry-run]`, `status`, `validate [--indexed-only]`, `rebuild`, `purge --before `, `config [--json]`. +Ten v2 CLI commands: `list [--project] [--all-projects] [--recent N] [--order timestamp:desc|asc] [--limit] [--offset] [--json] [--robot]`, `search [--text ] [--project] [--all-projects] [--source] [--source-path] [--after] [--before] [--role] [--content-type] [--tag] [--limit] [--offset] [--fields minimal|full] [--max-tokens N] [--lexical-only] [--similarity-threshold F] [--json] [--robot]`, `patterns --kind commands|failures|templates|sequences|corrections [--pending] [--batch N] [--project] [--all-projects] [--tag] [--trend] [--after] [--before] [--min-support N] [--min-confidence F] [--min-length N] [--max-length N] [--limit] [--offset] [--json] [--robot]`, `annotate --uuid --kind --label [--path

--ordinal ]`, `recover --from [--dry-run]`, `status [--json]`, `validate [--json]`, `rebuild`, `purge --before `, `config [--json]`. The `SearchEngine` interface is the port; `internal/storage` is the adapter. Database opened lazily. `OpenReadOnly()` provides read-only access for external consumers. ### Core Pipeline ``` -JSONL files → fs.WalkDir → SHA-256 dedup → ParseSessions() ─┐ +Active manifests → mandatory startup sync ───────────────────┐ +JSONL files → fs.WalkDir → SHA-256 dedup → ParseSessions() ─┤ Markdown plans → DiscoverPlanFiles() → ParsePlan() ──────────┤ Declarative Markdown inputs → markdown readers ──────────────┤ ▼ - SyncFiles() → SQLite FTS5 + SyncFiles() → perennial SQLite FTS5 │ -CLI query → Search() → BM25 → format_results() +search/list/patterns/status/validate → database-backed query/output ``` ### Declarative Markdown Source Types @@ -98,6 +100,7 @@ External knowledge sources are configured with active `*.inputs.toml` manifests - **Defensive parsing**: `SessionRecord` wrapper with `json.RawMessage` for fields handles legacy schemas and noise. - **Noise filtering**: Excludes `system-reminder`, `task-notification`, and subagent sessions by default. - **External FTS5**: Uses `search_items` as content table with SQLite triggers, `snippet()` extraction, and Porter stemmer tokenizer for morphological matching. +- **Mandatory root startup sync**: Every operational command validates active manifests and attempts one incremental sync before executing. Session, plan, and Markdown files are ingestion inputs; SQLite is the perennial record used by search, list, patterns, status, and validate. Use `--source-path` on search as a filter, paired with query text, for database-backed retrieval scoped to a known input path. - **Incremental sync**: SHA-256 hash per file stored in `indexed_files` table; unchanged files are skipped. - **Plan indexing**: Markdown plans from `~/.claude/plans/` split by `##` headers, each section indexed as a separate search item with `source='plan'`. - **Declarative Markdown inputs**: `markdown_document` and `markdown_sections` readers reuse `internal/sources` parsers, flow through the normal input manifest registry/autosync path, and store the manifest `source` value on indexed rows. They do not parse YAML frontmatter into structured metadata. @@ -114,7 +117,7 @@ External knowledge sources are configured with active `*.inputs.toml` manifests - **Schema migration rule**: Every new table or column MUST be introduced as a new migration version (increment the version number and add a new version-check block in `SetupSchema()`). Never modify existing migration blocks — existing databases that already passed that version will never re-run them. Migration v5 drops the phantom `session_events` table (and its indexes `idx_session_events_order` and `idx_session_events_project`) — the table was write-only dead weight after structured-stats filtering was removed. Migration v6 drops the phantom `search_items.source_metadata` column via `ALTER TABLE ... DROP COLUMN` — it had a setter but zero production callers and was never read. Migration v13 adds indexes on `template_matches(source_path)` and `correction_signals(source_path)` to optimize backfill discovery queries (reduces O(N·M) subquery scans to O(N·log M) index lookups). - **F0a rich capture (migration v8)**: readers extract per-message identity and tool metadata BEFORE serialization/cleaning destroys the evidence — `uuid` (record uuid; tool blocks get stable `#tN`/`#rN` suffixes by block index), `tool_name`, `command_head`, `is_error` (`*bool`, three-valued: tool_result blocks carry it and it is paired back onto the tool_use message cross-record via `tool_use_id`), and `was_interrupted` (detected on raw content before `CleanContent` strips "Request interrupted"). Persisted to `search_items` (`extraction_version`, `was_interrupted` columns) and the perennial `tool_events` satellite table (`UNIQUE(source_path, ordinal)`, no CASCADE lifecycle — only `purge` deletes from it, explicitly). Claude reader only; Pi/OpenCode emit zero values and stay on the legacy path. Design: `docs/superpowers/specs/2026-07-17-pattern-discovery-northstar-design.md`. - **F0a.1 command-head extraction v2 (extraction_version bump to 2)**: The `commandHead()` function in `internal/readers/claude_reader.go` now strips leading POSIX variable assignments (tokens containing `=`) before extracting the actual command name. Example: `"SP=/path/to/code; go test ./..."` extracts `"go"` instead of `"SP=/path/to/code;"`. This reduces noise in sequence mining and command pattern discovery. The bump to `extraction_version=2` triggers B1 backfill on next sync, incrementally re-extracting command_head from stored text for files indexed before this change (up to 200 files/run). -- **F0b perennial sync**: the DB is the perennial event store — session JSONL files expire (~30 days), indexed sessions survive them. Session files where EVERY message has a uuid sync append-only (no DELETE; `INSERT OR IGNORE` + UNIQUE constraints; row ids stable forever), with a one-time transition cleanup of legacy uuid-NULL rows per file. Files with any uuid-less message (Pi/OpenCode, legacy Claude) keep wipe-and-reload. `rebuild` is NON-destructive: re-derives both FTS indexes from `search_items` via FTS5 external-content `'rebuild'` in one transaction (`RebuildFTS()`), then runs incremental sync — it never deletes rows and never re-reads disk as source of truth. `purge --before` is the only deletion path and deletes `tool_events` satellites explicitly in the same transaction (no CASCADE). +- **F0b perennial sync**: the DB is the perennial event store — session JSONL files expire (~30 days), indexed sessions survive them. Session files where EVERY message has a uuid sync append-only (no DELETE; `INSERT OR IGNORE` + UNIQUE constraints; row ids stable forever), with a one-time transition cleanup of legacy uuid-NULL rows per file. Files with any uuid-less message (Pi/OpenCode, legacy Claude) keep wipe-and-reload. `rebuild` is NON-destructive: after the mandatory root startup sync has prepared the database, the handler re-derives both FTS indexes from `search_items` via FTS5 external-content `'rebuild'` in one transaction (`RebuildFTS()`) and performs no second sync — it never deletes rows and never re-reads disk as source of truth. `purge --before` is the only deletion path and deletes `tool_events` satellites explicitly in the same transaction (no CASCADE). - **F1 exit code mining**: the **reader** parses the exit code from the FULL tool output before `toolfmt` truncates it (`SerializeToolOutput` caps at 4000 runes and a Bash exit code is usually on the last line, so parsing the capped text lost exactly the codes worth having). `ExtractExitCodeText` does the parsing with no tool gate, because a `tool_result` block lives in a different JSONL record than its `tool_use` and the tool name is not knowable at that point; the Bash gate and the code are applied when result and use are paired by `tool_use_id` — the same cross-record pairing already used for `is_error`. `SyncFiles` persists `IndexedMessage.ExitCode` verbatim and MUST NOT re-derive a code from `msg.Text`, which is truncated. The extracted exit code is stored in the `tool_events.exit_code` column (migration v8; NULL for non-Bash tools or no match). The `patterns` command aggregates tool_events by (tool_name, command_head) for commands or (tool_name, is_error, exit_code) for failures, returning top N sorted by frequency with optional filters by project, session tag, and time window. Coverage metric reports the count of events with non-NULL is_error (signalled events) against the total failure count in the result set. - **F2 template mining (migration v10)**: unsupervised Drain-inspired template miner (`internal/templates/Miner`) discovers recurring error patterns from tool output during sync. Miner uses fixed-depth token prefix clustering (depth=2) to group messages; beyond the prefix, numeric/path/UUID tokens become `<*>` variables. Error-bearing lines (is_error=true) are extracted per tool via `ExtractErrorLines` (calibrated per tool: Bash prefers LAST non-empty line + error-matching; Go test matches "--- FAIL:", "FAIL\t", "error:"; others default to error-matching heuristic) and deterministically mined with SHA256 signature. Templates stored in `message_templates` (signature, normalization_version, template_text, occurrence_count, first_seen, last_seen) joined via `template_matches` (template_id, source_path, ordinal, item_uuid) with UNIQUE constraint for idempotency. Mining runs inside `SyncFiles` transaction; re-syncing increments occurrence_count only for new matches (detected via INSERT OR IGNORE). Query method `AggregateTemplates(opts)` filters by min_support (default 3), project, date range; `patterns --kind templates [--min-support N]` exposes results in text/JSON/robot formats with normalization_version metadata. _Q1 backfill update:_ Backfill mining filters rows by: include a tool-text row only when (it has case-insensitive "error: " prefix OR its ordinal has a tool_events row with is_error=1) AND it is not an input serialization. - **One line-selection predicate for both mining paths**: `shouldMineToolLine(contentType, text, isError)` in `internal/storage/mining.go` is called by sync-time `mineTemplatesForFile` and by backfill. Only the error signal differs — sync reads `IsError` off the message, backfill derives it from an `"error: "` prefix or a `tool_events` row. The predicate exists because the two paths had drifted, and the drift was the bug: sync selected on `ToolName != ""`, which picks the **tool_use** message, whose text *is* the input serialization, while the error text lives on the **tool_result** message that carries no `ToolName`. Sync therefore mined inputs and never errors. Tool results have no tool name, so sync mines them as `"Unknown"`, matching backfill; per-tool line calibration is a follow-up. @@ -124,18 +127,18 @@ External knowledge sources are configured with active `*.inputs.toml` manifests - **F3 correction detection (migration v11)**: deterministic message-level correction detection as a funnel for agent-classification loops. Four detectors: (1) bilingual correction lexicon (es+en, confidence 0.8), (2) interrupt flags from F0a (confidence 0.5), (3) permission denials ("denied"/"rechaza", confidence 0.4), (4) rephrase-similarity via Jaccard ≥0.6 (confidence 0.6). All pure Go, no ML. Detection runs at sync time; results stored in perennial `correction_signals` (UNIQUE(source_path, ordinal, detector), migration v11). Query method `AggregateCorrections(opts)` groups by ordinal, filters by project and min-confidence, returns top N with detector names and max confidence; `uuid` is COALESCEd to empty string in the SELECT because B3-backfilled candidates can sit on NULL-uuid rows (legacy and Pi/OpenCode messages) and a plain-string Scan would otherwise fail. `patterns --kind corrections [--min-confidence F]` exposes candidates for F3b agent labeling. Calibration procedure in `docs/eval/corrections-calibration.md` (hand-label 50 candidates to measure per-detector precision before F3b launch). Known limitation: Spanish false positive "no, eso no es un bug, es esperado" (acceptable v1 trade-off). - **Chat-export filter (Q5)**: the lexicon detector skips pasted chat transcripts, which otherwise flag at the lexicon's 0.8 confidence and dominate the candidate pool. `isChatExport` counts **markers, not lines**: `CleanContent` collapses all whitespace (`strings.Join(strings.Fields(content), " ")`), so by detector time a pasted transcript is a single line and any line-based rule could never fire. Three or more sender markers mark the span as a transcript; requiring several keeps an incidental timestamp from tripping the filter. The marker regex is calibrated to formats actually observed in the corpus: the hour is `\d{1,2}` (real exports write `[3:06 p.m., 2/7/2026]`, unpadded) and the sender is `[^:\]\n]{1,40}` rather than `\w+` (display names carry spaces — "Pedro Chan:"). A two-digit hour or a single-word sender silently misses the exact false positive this filter exists for. - **Superseded correction signals are cleared on re-sync**: `correction_signals` is append-only with `INSERT OR IGNORE`, so a false positive recorded under older detector rules would outlive any detector fix and keep topping the census. `SyncFiles` deletes a file's signals whose `extraction_version` is NULL or below `CurrentExtractionVersion` before re-detecting, so bumping the extraction version is what propagates a detector fix to already-indexed sessions (drained incrementally by the B1 stale-set, ~200 files/run). Signals already at the current version are untouched, so re-syncing an up-to-date file stays a no-op. **Sessions that neither path reaches are handled by `RederiveSupersededCorrections`** (called from `maybeAutoSync`, bounded per run). A session whose JSONL expired keeps its `indexed_files` row — nothing prunes that table when a file disappears — so it is invisible to `SyncFiles` (not on disk) *and* to `BackfillDerived` (which treats "expired" as absent from `indexed_files`). That dead zone is the common case, not an edge case. Discovery is epoch-only (`extraction_version` NULL or `< CurrentExtractionVersion`), which converges in both directions: a re-derived path is stamped current and drops out, and a path that re-derives to zero signals has no stale rows left to match. Re-derived signals carry `CurrentExtractionVersion`, not the lossy `0`; that stamp *is* the convergence mechanism, and it is safe because nothing reads `correction_signals.extraction_version`. Two things this depends on, both learned the hard way: `LoadMessagesForPath` must COALESCE the nullable columns (`uuid`, `timestamp`, `extraction_version`) — legacy and Pi/OpenCode rows carry NULLs and a plain Scan aborts on them — and the batch must skip an unreadable path rather than fail, because discovery order is deterministic and one bad path otherwise parks itself at the head of every run. With both, the real corpus drains 1201 stale signals across 267 paths to zero in two syncs; without them it made no progress at all. Re-derivation uses **one transaction per path, never one per batch**: it deletes a path's superseded signals before writing the new ones, so that pair must be atomic — and a shared batch transaction would be worse than useless, since a path failing midway leaves its DELETE staged and skipping to the next path still commits it, wiping the signals of the very path the loop reports as skipped. (`0` still marks lossy *`tool_events`*, where metadata really was reconstructed from serialized text; a correction signal read the same stored prose either way, so it is not lossy.) -- **F3 calibration sample quality (no migration)**: Two improvements to v1 calibration precision. (1) **Interrupt min-length guard** (detection-time): cInterruptDetector skips candidates with cleaned text `< 20 Unicode code points`, filtering out stubs like `"[ by user]"` (8 chars). Threshold is unexported const `minInterruptLength` in `internal/corrections/corrections.go`, tunable as a v1 trade-off (may adjust future if false-negative rate high). (2) **Teammate-message wrapper cleanup** (sync-time): CleanContent removes `...` blocks entirely, reducing rephrase false positives from Jaccard similarity on internal notes. Pair added to `tagPatternsWithContent` in `internal/sync/sync.go` (~line 255). Post-merge, users run `backscroll rebuild` to re-derive FTS and re-mine corrections with both guards applied. Existing stored FP signals remain (append-only); re-calibration samples fresh corrections post-rebuild. +- **F3 calibration sample quality (no migration)**: Two improvements to v1 calibration precision. (1) **Interrupt min-length guard** (detection-time): cInterruptDetector skips candidates with cleaned text `< 20 Unicode code points`, filtering out stubs like `"[ by user]"` (8 chars). Threshold is unexported const `minInterruptLength` in `internal/corrections/corrections.go`, tunable as a v1 trade-off (may adjust future if false-negative rate high). (2) **Teammate-message wrapper cleanup** (sync-time): CleanContent removes `...` blocks entirely, reducing rephrase false positives from Jaccard similarity on internal notes. Pair added to `tagPatternsWithContent` in `internal/sync/sync.go` (~line 255). Post-merge, users run `backscroll rebuild` so mandatory startup sync runs first, then the handler re-derives FTS and re-mines corrections with both guards applied. Existing stored FP signals remain (append-only); re-calibration samples fresh corrections post-rebuild. - **F3b classification loop checkpoint semantics (migration v12)**: `annotations` table is append-and-replace (INSERT OR REPLACE on UNIQUE key), keyed by (source_path, ordinal, kind). Agent loop queries `patterns --kind corrections --pending` to get un-annotated candidates; after annotating a batch, the next query automatically resumes from where it left off (LEFT JOIN filter). Crash-safe: re-running the loop command always shows correct pending state. Labels free-form in v1; `label_enum` table (enum constraint) is a future slice post-calibration, added in a new migration that pre-fills the enum from observed labels and rejects new labels outside the enum. - **F4 sequence mining (PrefixSpan)**: discovers frequent tool-call sequences per session using the PrefixSpan algorithm. Tool events are categorized via a versioned category map (v2: `internal/categories` with `inputs/categories.toml` preset + embedded fallback) that maps tool names + command heads to canonical categories: NAV (cd), SEARCH (rg|grep|fd|find), FILE_INSPECT (ls|cat|eza|bat), TEXT_TRANSFORM (sd|jq|awk|sed), DB (sqlite3), REVIEW_TOOL (gentle-ai), TASK_RUNNER (just), SHELL_STATE (echo|export), FS_OP (mkdir|mv|rm), GO_EXEC, GIT, TEST_EXEC, PKG, FILE_READ, FILE_WRITE, SHELL_OTHER. At query time, `LoadToolSequences` loads tool_events, applies categories, and groups by session. `Mine()` algorithm in `internal/sequences` is pure-Go, deterministic, and includes a mandatory maxLen (default 6) to prevent combinatorial explosion on repetitive sessions. Results sorted by support DESC then lexicographic order; --limit/--offset paginate the MINED patterns (the input corpus is never truncated). Query method `patterns --kind sequences [--min-support N] [--min-length N] [--max-length N]` exposes findings in text/JSON/robot formats with share percentage per pattern. - **F4b trend analysis (no migration)**: week-over-week bucketing for command and failure patterns via `--trend` flag. SQL groups by `strftime('%Y-W%W', si.timestamp)` for SQLite %W week numbering (Monday-based, 00-53; not ISO 8601); NULL timestamps are excluded from trend results with exclusion count reported to stderr. Output shapes extended: text shows weeks with patterns nested per week; JSON wraps patterns in week buckets; robot emits `week_N=YYYY-WNN` lines. Only `--kind commands` and `--kind failures` support `--trend`; other kinds reject the flag at early validation (before DB open). Methods `AggregateCommandsTrend()` and `AggregateFailuresTrend()` in `internal/storage` return `CommandPatternWeekly` and `FailurePatternWeekly` structs with week bucketing. - **B1 extraction-version backfill (incremental re-sync)**: Files indexed before v8 carry no rich metadata (uuid NULL, no tool_events, no corrections). A new storage query `StalePaths(currentVersion)` returns paths whose rows have `extraction_version IS NULL` or `< currentVersion`. In `maybeAutoSync`, a stale-set is built once per run and during hash evaluation, unchanged-hash files in the stale-set re-parse anyway (skip the `continue`), up to a per-run cap (default 200 files). The existing perennial path + transition cleanup (sync.go:99-108) then does the right thing automatically — legacy uuid-NULL rows are deleted (one-time), rich uuid-bearing rows replace them via `INSERT OR IGNORE`, and `extraction_version` updates. Repeated `maybeAutoSync` invocations drain the backlog incrementally FIFO (ordered by `last_indexed` ASC). Re-parsing stops when expired JSONL files vanish from disk — the database survives them (perennity contract). - **B2 project fallback identity + historical re-resolution**: when the global registry (`~/.config/backscroll/projects.toml`) does not match a session's cwd, `projects.Identify()` derives a sanitized fallback id from the cwd basename (lowercase, `[a-z0-9-_]`; registry always wins when it matches). `rebuild` runs a re-resolution pass: `ReresolveProjects` decodes Claude's session-dir encoding (dashes-for-slashes; if the decoded path exists on disk its basename wins, else the last dash segment — the encoding is lossy and this heuristic is documented as ambiguous for dir names containing dashes) and relabels rows stuck at `project='unknown'`, returning distinct files resolved. Fallback labels are not revisited once set — followed by registry-aware re-resolution below. - **B2.1 registry-aware re-resolution**: `projects.Identify()` now carries a `FromRegistry` flag to distinguish registry matches from fallback labels. `rebuild` includes a second phase that loads `~/.config/backscroll/projects.toml` and re-resolves historical sessions labeled with fallback IDs. For each session path, it decodes the cwd, applies cross-host normalization, and calls `Identify` with the registry. If a registry entry matches (FromRegistry=true) and differs from the stored fallback, the rows are updated. Only registry matches count — fallback-only paths are not re-labeled (no churn). This allows a future registry entry to correct historical misattribution without touching already-correct sessions. Wired into rebuild via `ReresolveProjectsWithRegistry` (queries.go). -- **B3 retroactive mining over stored text**: for sessions that expire from disk before Template/Correction/Sequence mining runs, `BackfillDerived()` recovers templates, correction signals, and lossy tool_events from stored text in search_items. Stale template re-mining (F2a) is integrated: BackfillDerived first discovers stale paths via `StaleTemplatePaths(CurrentNormalizationVersion)` and re-mines templates under v2 heuristics, updating existing templates to normalization_version=2. Template mining reuses `internal/templates/Miner` over tool-text rows. Correction detection filters input to prose only (role='user' AND content_type IN ('text','code')) to avoid tool_result false positives; lexicon, denial, and rephrase detectors run on prose; interrupt detector runs on all user messages. Lossy tool_events reverse-parse toolfmt input serialization ( ...; heuristic: first token has no '=', at least one subsequent has '=') to extract tool_name and command_head; outputs unattributable without tool_use_id so skipped; extraction_version=0 marks lossy rows (inputs only, unrecoverable once disk source expires). `rebuild` command now: (1) re-derive FTS from DB, (2) BackfillDerived (stale-path discovery + re-mining + expired-file discovery + batch mining + progress reporting), (3) re-resolve projects, (4) incremental sync. -- **Early input validation**: CLI commands validate flag values (e.g. `--format`) before opening the database, so invalid inputs fail fast without side effects. +- **B3 retroactive mining over stored text**: for sessions that expire from disk before Template/Correction/Sequence mining runs, `BackfillDerived()` recovers templates, correction signals, and lossy tool_events from stored text in search_items. Stale template re-mining (F2a) is integrated: BackfillDerived first discovers stale paths via `StaleTemplatePaths(CurrentNormalizationVersion)` and re-mines templates under v2 heuristics, updating existing templates to normalization_version=2. Template mining reuses `internal/templates/Miner` over tool-text rows. Correction detection filters input to prose only (role='user' AND content_type IN ('text','code')) to avoid tool_result false positives; lexicon, denial, and rephrase detectors run on prose; interrupt detector runs on all user messages. Lossy tool_events reverse-parse toolfmt input serialization ( ...; heuristic: first token has no '=', at least one subsequent has '=') to extract tool_name and command_head; outputs unattributable without tool_use_id so skipped; extraction_version=0 marks lossy rows (inputs only, unrecoverable once disk source expires). `rebuild` command now runs after mandatory root startup sync and then: (1) re-derives FTS from DB, (2) BackfillDerived (stale-path discovery + re-mining + expired-file discovery + batch mining + progress reporting), (3) re-resolves projects, and (4) applies registry-aware re-resolution. It performs no second sync. +- **Early input validation**: CLI commands validate positional arguments and semantic flag values through Cobra `Args`, which runs before the root `PersistentPreRunE`; the root also checks required flags and flag groups before invoking the mandatory startup policy. Invalid invocations therefore fail before opening, creating, or synchronizing the database. Keep pure validators reusable from direct `run*` callers so tests and non-Cobra entry points preserve the same boundary. - **Coverage gate**: CI enforces ≥85% aggregate statement coverage via `go test ./... -race -coverprofile`. Local check: `bash scripts/check-coverage.sh`. Tests that depend on local machine state (e.g. `~/.config/backscroll/projects.toml`) must use `t.Setenv("HOME", tempDir)` to stay reproducible on CI. Likewise, `InputsDir` branches requiring `BACKSCROLL_CONFIG_DIR` to be unset must set it to `""` via `t.Setenv`. To test the `Validate` orphan path, insert directly into `search_items` without a matching `indexed_files` row. - **Zero-result guidance**: when `search`/`list` return no rows, actionable suggestions (`--all-projects`, `--content-type tool`, `backscroll status`) are printed to STDERR — never STDOUT, so `--json` stays a clean empty payload. -- **Robot output contract**: `search --robot` emits `result_N_field=value` lines exactly once-wrapped (the robot path writes lines directly; passing pre-formatted lines through the picokit formatter double-wraps them as `result_N=result_N_field=...`). +- **Search robot output contract**: robot mode on search emits `result_N_field=value` lines exactly once-wrapped (the robot path writes lines directly; passing pre-formatted lines through the picokit formatter double-wraps them as `result_N=result_N_field=...`). Search robot string values escape backslash as `\\`, carriage return as `\r`, and newline as `\n`. - **Cross-host project identity**: `projects.Identify()` normalizes session cwd against registry roots by matching root tails (≥2 components, case-insensitive), so `/home/shared/` sessions resolve against `/Users/Shared/` roots on a synced index. Registry roots should keep distinct suffixes — two projects whose roots share the same trailing components could misbucket. - **Recall eval-set**: `docs/eval/queries.toml` (~20 real mined queries with `expected_match` ground truth) + `scripts/eval.sh` compute recall@5; a query counts only if its expected target appears in the top 5. Local regression gate, not a required CI step. @@ -219,10 +222,12 @@ github.com/pablontiv/backscroll/internal/sources — External source parse github.com/pablontiv/backscroll/internal/templates — F2 Drain-inspired miner: Miner, ProcessLine, ExtractErrorLines github.com/pablontiv/backscroll/internal/corrections — F3 correction-signal detectors: lexicon (es+en), interrupt, denial, rephrase-similarity; registry pattern for deterministic, pluggable detectors github.com/pablontiv/backscroll/internal/categories — F4 category map loader (rule engine, versioning, tool→category mapping) with embedded default preset +github.com/pablontiv/backscroll/internal/chunking — Token-aware text chunking helpers for embedding preparation +github.com/pablontiv/backscroll/internal/embedding — Embedding provider interface, mock provider, and ONNX provider implementation +github.com/pablontiv/backscroll/internal/hybrid — Reciprocal Rank Fusion helpers for merged lexical/vector retrieval github.com/pablontiv/backscroll/internal/sequences — F4 PrefixSpan mining (deterministic pattern discovery per session) github.com/pablontiv/backscroll/internal/storage — Database schema, migrations v1–v13, FTS5 indexes github.com/pablontiv/backscroll/internal/projects — Project identity registry -github.com/pablontiv/backscroll/internal/reader — Direct session file reader github.com/pablontiv/backscroll/internal/readers — SessionReader interface, Registry, ClaudeReader (text+tool_use+tool_result), PiReader (text+toolCall+custom results), OpenCodeReader (text+tool state.input+state.output), MarkdownDocumentReader (`markdown_document`), MarkdownSectionsReader (`markdown_sections`); toolfmt serializer github.com/pablontiv/backscroll/internal/recovery — Stranded database recovery orchestration, durable backup, and atomic replacement ``` diff --git a/README.md b/README.md index 42b5871..9ebccd3 100644 --- a/README.md +++ b/README.md @@ -138,15 +138,20 @@ How a file is re-synced depends on whether its messages carry identity. Sessions ### Recover what happened +Every operational command validates active manifests and attempts one incremental +sync before executing. Session, plan, and Markdown files are ingestion inputs; +SQLite is the perennial record used by search, list, patterns, status, and validate. +Use `--source-path` on search as a filter, paired with query text, for database-backed retrieval scoped to a known input path. + ```bash backscroll search --text "QUERY" --project # this project backscroll search --text "QUERY" --all-projects # everywhere backscroll search --text "QUERY" --content-type tool # commands, paths, errors backscroll list --order timestamp:desc --limit 10 # recent sessions -backscroll read --path --tail 45 --semantic # one session, condensed +backscroll search --text "artifact literal" --source-path "*SESSION-ID*" --all-projects --json # filter one known input path ``` -Filters worth knowing: `--after` / `--before` for a date window, `--tag` for auto-detected session categories (debugging, refactoring, testing…), `--source-path` to pin one file, `--role` to keep only what you said. +Filters worth knowing: `--after` / `--before` for a date window, `--tag` for auto-detected session categories (debugging, refactoring, testing…), `--source-path` to pin one stored input path, `--source` to keep one source class, and `--role` to keep only what you said. ### Discover what recurs @@ -173,18 +178,18 @@ Labelled candidates drop out of `--pending`, so the loop resumes wherever it sto ```bash backscroll status # size, counts, last sync -backscroll validate # integrity check +backscroll validate --json # parseable integrity check backscroll rebuild # re-derive search indexes from the database backscroll purge --before # the only deletion path ``` -`rebuild` does not re-read your session files as the source of truth: it rebuilds the search indexes from what is already stored, re-derives templates and correction signals, then runs an ordinary incremental sync. Sessions that vanished from disk survive it untouched. +`rebuild` operates after the mandatory root startup sync has already prepared the database. The handler does not perform a second sync: it re-derives search indexes from stored rows, re-derives templates/correction signals/tool-event satellites where possible, and re-resolves project identities. Sessions that vanished from disk survive it untouched. ### Output for whoever is reading -Output is tab-separated text with no ANSI escapes by default, so it pipes cleanly. +Default output is human-readable text. Machine modes keep stdout parseable: human progress and warnings go to stderr, JSON/robot startup progress is discarded, and structured diagnostics remain parseable. -`--json` is available on `search`, `list`, `patterns`, `status` and `config`. `--robot`, which emits `field=value` lines, is available on `search`, `list` and `patterns` — the three that return result sets. `read` takes `--pretty` instead, and `validate`, `rebuild`, `purge` and `annotate` report in plain text only. +`--json` is available on `search`, `list`, `patterns`, `status`, `validate`, and `config`. JSON mode on search emits a JSON array. `--robot` is available on `search`, `list`, and `patterns`; robot mode on search emits `result_N_field=value` lines, and search robot string values escape backslash as `\\`, carriage return as `\r`, and newline as `\n`. `rebuild`, `purge`, and `annotate` report in plain text only. On `search`, `--fields minimal|full` controls density and `--max-tokens N` caps output for a context window. @@ -203,7 +208,7 @@ Agents use the same commands with `--robot --fields minimal --max-tokens N`. A ` Backscroll separates application configuration from input configuration. - **Application config** (`backscroll.toml`) controls where the database lives. By default, Backscroll creates an index at `~/.backscroll.db`. -- **Input config** (`*.inputs.toml`) controls what files are ingested via `backscroll search` and `backscroll list`. The canonical runtime location is `/backscroll/inputs/*.inputs.toml`, where `` is the OS config directory or `BACKSCROLL_CONFIG_DIR` when set. +- **Input config** (`*.inputs.toml`) controls what files are ingested before operational commands query SQLite. The canonical runtime location is `/backscroll/inputs/*.inputs.toml`, where `` is the OS config directory or `BACKSCROLL_CONFIG_DIR` when set. Override app settings by creating `~/.config/backscroll/config.toml` or `backscroll.toml` in the current directory: @@ -249,7 +254,7 @@ See [Configuration docs](docs/configuration.md) for the full resolution order an | [Sync & Indexing](docs/sync.md) | Incremental sync, noise filtering, project detection | | [Search Engine](docs/search.md) | BM25 ranking, output formats, token limiting | | [Pattern Discovery](docs/patterns.md) | The five censuses, the classification loop, calibration | -| [Indexed Path Lookup](docs/read.md) | DB-backed lookup using `search_items.source_path` | +| [Source Path Retrieval](docs/read.md) | DB-backed lookup using `search_items.source_path` | | [Configuration](docs/configuration.md) | Config resolution, TOML format, environment variables | | [Generic Input Contract](docs/input-contract.md) | Global `*.inputs.toml` contract for provider-neutral ingestion | | [Session Search Research](docs/research/backscroll-session-search-cli.md) | Feasibility study: axioms, evidence tables, capabilities matrix | diff --git a/cmd/backscroll/annotate.go b/cmd/backscroll/annotate.go index d888359..dedaf04 100644 --- a/cmd/backscroll/annotate.go +++ b/cmd/backscroll/annotate.go @@ -26,11 +26,22 @@ func newAnnotateCmd(stdout, stderr io.Writer) *cobra.Command { Long: `Annotate a message with a classification label. Validates message existence before writing. Supports both uuid (preferred) and legacy source_path+ordinal fallback. Re-annotating the same (source_path, ordinal, kind) replaces the label.`, + Args: func(cmd *cobra.Command, args []string) error { + return validateCommandBeforeStartup(cmd, args, func(_ *cobra.Command, args []string) error { + if len(args) > 0 { + return fmt.Errorf("unexpected positional argument %q", args[0]) + } + return nil + }, func() error { + return validateAnnotateRequest(uuid, path, ordinal, kind, label) + }) + }, RunE: func(cmd *cobra.Command, args []string) error { - if len(args) > 0 { - return fmt.Errorf("unexpected positional argument %q", args[0]) + startup := startupResultFrom(cmd) + if startup.Config == nil { + return fmt.Errorf("startup configuration unavailable") } - return runAnnotate(stdout, stderr, uuid, path, ordinal, kind, label) + return runAnnotate(cmd.Context(), stdout, stderr, startup.Config, uuid, path, ordinal, kind, label) }, } @@ -46,24 +57,24 @@ fallback. Re-annotating the same (source_path, ordinal, kind) replaces the label return cmd } -func runAnnotate(stdout, stderr io.Writer, - uuid, path string, ordinal int, kind, label string) (retErr error) { - - // Early flag validation +func validateAnnotateRequest(uuid, path string, ordinal int, kind, label string) error { if uuid == "" && (path == "" || ordinal < 0) { return fmt.Errorf("must provide either --uuid or both --path and --ordinal") } - if kind == "" || label == "" { return fmt.Errorf("--kind and --label are required") } + return nil +} - cfg, err := config.Load() - if err != nil { - return fmt.Errorf("load config: %w", err) +func runAnnotate(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, + uuid, path string, ordinal int, kind, label string) (retErr error) { + + if err := validateAnnotateRequest(uuid, path, ordinal, kind, label); err != nil { + return err } - db, diag, err := prepareIndex(context.Background(), cfg, indexMutation, false) + db, diag, err := prepareIndex(ctx, cfg, indexMutation) if diag != nil { return refuseIndex(stdout, stderr, *diag, false, false) } diff --git a/cmd/backscroll/annotate_test.go b/cmd/backscroll/annotate_test.go index 0940de7..441e91b 100644 --- a/cmd/backscroll/annotate_test.go +++ b/cmd/backscroll/annotate_test.go @@ -32,8 +32,8 @@ func TestAnnotateCommand(t *testing.T) { } _ = db.Close() - // Test: annotate via CLI - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) + // Test: annotate via CLI with hermetic startup inputs/config + setIndexPolicyEnv(t, dbPath, t.TempDir()) var stdout, stderr bytes.Buffer err = run(&stdout, &stderr, []string{"annotate", "--uuid", "u1", "--kind", "correction", "--label", "fixable"}) if err != nil { @@ -57,8 +57,8 @@ func TestAnnotateCommandMissingMessage(t *testing.T) { } _ = db.Close() - // Test: annotate non-existent message - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) + // Test: annotate non-existent message with hermetic startup inputs/config + setIndexPolicyEnv(t, dbPath, t.TempDir()) var stdout, stderr bytes.Buffer err = run(&stdout, &stderr, []string{"annotate", "--uuid", "nonexistent", "--kind", "correction", "--label", "label"}) if err == nil { diff --git a/cmd/backscroll/compat_diagnostics_test.go b/cmd/backscroll/compat_diagnostics_test.go index 1ab11a9..e9a2290 100644 --- a/cmd/backscroll/compat_diagnostics_test.go +++ b/cmd/backscroll/compat_diagnostics_test.go @@ -2,67 +2,60 @@ package main import ( "bytes" + "context" "database/sql" "encoding/json" + "errors" "fmt" + "io" "os" "path/filepath" "strings" "testing" "github.com/pablontiv/backscroll/internal/compat" + "github.com/pablontiv/backscroll/internal/config" "github.com/pablontiv/backscroll/internal/storage" ) -func TestDirectReadRemainsAvailableButClaimsNoIndexFreshness(t *testing.T) { - fixture, err := filepath.Abs(filepath.Join(fixturesDir(), "pi-session.jsonl")) - if err != nil { - t.Fatalf("resolve fixture path: %v", err) - } +func TestStatusUnhealthyReturnsDiagnostic(t *testing.T) { dbPath := newUnsupportedIndexedConsumerDB(t) - before := readDBBytes(t, dbPath) - - stdout, stderr, err := runCmd("read", fixture) - if err != nil { - t.Fatalf("read with unsupported index configured failed: %v\nstderr: %s", err, stderr) - } - for _, want := range []string{"pi manifest fixture signal", "pi visible answer"} { - if !strings.Contains(stdout, want) { - t.Fatalf("read output missing decoded fixture content %q:\n%s", want, stdout) - } - } - for _, forbidden := range []string{"usable", "fresh", "current index"} { - if strings.Contains(strings.ToLower(stdout+stderr), forbidden) { - t.Fatalf("direct read made index-freshness claim %q; stdout=%q stderr=%q", forbidden, stdout, stderr) - } - } - if after := readDBBytes(t, dbPath); !bytes.Equal(after, before) { - t.Fatal("direct read mutated unsupported database bytes") - } -} - -func TestStatusUnhealthyIsReadOnly(t *testing.T) { - dbPath := newUnsupportedIndexedConsumerDB(t) - before := snapshotSQLiteFiles(t, dbPath) stdout, stderr, err := runCmd("status") if err == nil { t.Fatalf("status succeeded on unsupported index; stdout=%q stderr=%q", stdout, stderr) } assertDiagnosticText(t, stdout+stderr, compat.CodeUnsupportedLineage, dbPath) - assertSQLiteFilesUnchanged(t, dbPath, before) } -func TestValidateUnhealthyIsReadOnly(t *testing.T) { +func TestValidateUnhealthyReturnsDiagnostic(t *testing.T) { dbPath := newUnsupportedIndexedConsumerDB(t) - before := snapshotSQLiteFiles(t, dbPath) stdout, stderr, err := runCmd("validate") if err == nil { t.Fatalf("validate succeeded on unsupported index; stdout=%q stderr=%q", stdout, stderr) } assertDiagnosticText(t, stdout+stderr, compat.CodeUnsupportedLineage, dbPath) - assertSQLiteFilesUnchanged(t, dbPath, before) +} + +func TestRecoveryDiagnosticsForIndexHonorsCanceledContext(t *testing.T) { + dbPath := newRecoveryConflictDiagnosticDB(t) + db, err := storage.OpenReadOnly(dbPath) + if err != nil { + t.Fatalf("open recovery diagnostics db: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + diagnostics, err := recoveryDiagnosticsForIndex(ctx, db, dbPath) + if err == nil { + t.Fatalf("recovery diagnostics succeeded with canceled context; diagnostics=%+v", diagnostics) + } + if !errors.Is(err, context.Canceled) { + t.Fatalf("recovery diagnostics error = %v, want context canceled", err) + } } func TestValidateTextReportsSemanticRecoveryDiagnosticsReadOnly(t *testing.T) { @@ -121,7 +114,7 @@ func TestValidateTextReportsMultipleSemanticRecoveryDiagnosticsReadOnly(t *testi assertSQLiteFilesUnchanged(t, dbPath, before) } -func TestStatusHealthyIndexIsReadOnlyInTextAndJSON(t *testing.T) { +func TestStatusHealthyIndexReportsTextAndJSON(t *testing.T) { dbPath := filepath.Join(t.TempDir(), "backscroll.db") db, err := storage.Open(dbPath) if err != nil { @@ -144,7 +137,6 @@ func TestStatusHealthyIndexIsReadOnlyInTextAndJSON(t *testing.T) { t.Fatalf("close current index: %v", err) } setIndexPolicyEnv(t, dbPath, t.TempDir()) - before := snapshotSQLiteFiles(t, dbPath) stdout, stderr, err := runCmd("status") if err != nil { @@ -153,7 +145,6 @@ func TestStatusHealthyIndexIsReadOnlyInTextAndJSON(t *testing.T) { if stderr != "" || !strings.Contains(stdout, "Files indexed: 1") || !strings.Contains(stdout, "Messages indexed: 1") { t.Fatalf("status healthy index text output stdout=%q stderr=%q", stdout, stderr) } - assertSQLiteFilesUnchanged(t, dbPath, before) stdout, stderr, err = runCmd("status", "--json") if err != nil { @@ -179,7 +170,6 @@ func TestStatusHealthyIndexIsReadOnlyInTextAndJSON(t *testing.T) { if !payload.Database.Exists || payload.Database.Size == 0 || !payload.Index.Usable || payload.Index.TotalFiles != 1 || payload.Index.TotalMessages != 1 { t.Fatalf("status --json healthy index payload = %+v, want existing usable one-row index", payload) } - assertSQLiteFilesUnchanged(t, dbPath, before) } func TestAnnotatePathOrdinalFallbackPersistsThroughCobra(t *testing.T) { @@ -228,7 +218,7 @@ func TestAnnotatePathOrdinalFallbackPersistsThroughCobra(t *testing.T) { } } -func TestValidateHealthyIndexIsReadOnlyInTextAndJSON(t *testing.T) { +func TestValidateHealthyIndexReportsTextAndJSON(t *testing.T) { dbPath := filepath.Join(t.TempDir(), "backscroll.db") db, err := storage.Open(dbPath) if err != nil { @@ -238,7 +228,6 @@ func TestValidateHealthyIndexIsReadOnlyInTextAndJSON(t *testing.T) { t.Fatalf("close current index: %v", err) } setIndexPolicyEnv(t, dbPath, t.TempDir()) - before := snapshotSQLiteFiles(t, dbPath) stdout, stderr, err := runCmd("validate") if err != nil { @@ -247,7 +236,6 @@ func TestValidateHealthyIndexIsReadOnlyInTextAndJSON(t *testing.T) { if stderr != "" || !strings.Contains(stdout, "Index validation passed") { t.Fatalf("validate healthy index text output stdout=%q stderr=%q", stdout, stderr) } - assertSQLiteFilesUnchanged(t, dbPath, before) stdout, stderr, err = runCmd("validate", "--json") if err != nil { @@ -266,10 +254,9 @@ func TestValidateHealthyIndexIsReadOnlyInTextAndJSON(t *testing.T) { if !payload.Valid || !payload.DatabaseExists { t.Fatalf("validate --json healthy index payload = %+v, want valid existing database", payload) } - assertSQLiteFilesUnchanged(t, dbPath, before) } -func TestConfigAndStatusDeclarativeInputsVisibleWithoutCreatingIndex(t *testing.T) { +func TestConfigAndStatusDeclarativeInputsVisibleAfterStartup(t *testing.T) { dir := t.TempDir() dbPath := filepath.Join(dir, "backscroll.db") cfgDir := filepath.Join(dir, "config") @@ -321,7 +308,7 @@ func TestConfigAndStatusDeclarativeInputsVisibleWithoutCreatingIndex(t *testing. if err != nil { t.Fatalf("status text with declarative input failed: %v stdout=%q stderr=%q", err, stdout, stderr) } - for _, want := range []string{"Index: Not yet created", "Inputs: 1 active (declarative)", "- claude"} { + for _, want := range []string{"Files indexed: 1", "Messages indexed: 4", "Inputs: 1 active (declarative)", "- claude"} { if !strings.Contains(stdout, want) { t.Fatalf("status text missing %q in:\n%s", want, stdout) } @@ -329,64 +316,58 @@ func TestConfigAndStatusDeclarativeInputsVisibleWithoutCreatingIndex(t *testing. if stderr != "" { t.Fatalf("status text wrote stderr: %q", stderr) } - if _, err := os.Stat(dbPath); !os.IsNotExist(err) { - t.Fatalf("config/status created database: %v", err) + if _, err := os.Stat(dbPath); err != nil { + t.Fatalf("mandatory startup did not prepare configured database: %v", err) } } -func TestStatusAndValidateMissingIndexAreReadOnlyMachineDiagnostics(t *testing.T) { - dbPath := filepath.Join(t.TempDir(), "missing", "backscroll.db") +func TestStatusAndValidateMissingIndexArePreparedByStartup(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "backscroll.db") setIndexPolicyEnv(t, dbPath, t.TempDir()) stdout, stderr, err := runCmd("validate") if err != nil { - t.Fatalf("validate missing index failed: %v stdout=%q stderr=%q", err, stdout, stderr) + t.Fatalf("validate startup-prepared index failed: %v stdout=%q stderr=%q", err, stdout, stderr) } - if stderr != "" || !strings.Contains(stdout, "database not found") { - t.Fatalf("validate missing index text output stdout=%q stderr=%q", stdout, stderr) + if stderr != "" || !strings.Contains(stdout, "Index validation passed") { + t.Fatalf("validate startup-prepared index text output stdout=%q stderr=%q", stdout, stderr) } - if _, err := os.Stat(dbPath); !os.IsNotExist(err) { - t.Fatalf("validate created or touched missing database: %v", err) + if _, err := os.Stat(dbPath); err != nil { + t.Fatalf("validate did not prepare configured database: %v", err) } stdout, stderr, err = runCmd("status") if err != nil { - t.Fatalf("status missing index failed: %v stdout=%q stderr=%q", err, stdout, stderr) + t.Fatalf("status startup-prepared index failed: %v stdout=%q stderr=%q", err, stdout, stderr) } - if stderr != "" || !strings.Contains(stdout, "Index: Not yet created") || !strings.Contains(stdout, "Messages indexed: 0") { - t.Fatalf("status missing index text output stdout=%q stderr=%q", stdout, stderr) - } - if _, err := os.Stat(dbPath); !os.IsNotExist(err) { - t.Fatalf("status created or touched missing database: %v", err) + if stderr != "" || !strings.Contains(stdout, "Files indexed: 0") || !strings.Contains(stdout, "Messages indexed: 0") { + t.Fatalf("status startup-prepared index text output stdout=%q stderr=%q", stdout, stderr) } stdout, stderr, err = runCmd("validate", "--json") if err != nil { - t.Fatalf("validate --json missing index failed: %v stdout=%q stderr=%q", err, stdout, stderr) + t.Fatalf("validate --json startup-prepared index failed: %v stdout=%q stderr=%q", err, stdout, stderr) } if stderr != "" { - t.Fatalf("validate --json missing index wrote stderr: %q", stderr) + t.Fatalf("validate --json startup-prepared index wrote stderr: %q", stderr) } var validatePayload struct { Valid bool `json:"valid"` DatabaseExists bool `json:"database_exists"` } if err := json.Unmarshal([]byte(stdout), &validatePayload); err != nil { - t.Fatalf("validate --json missing index emitted invalid JSON %q: %v", stdout, err) - } - if !validatePayload.Valid || validatePayload.DatabaseExists { - t.Fatalf("validate --json missing index payload = %+v, want valid without database", validatePayload) + t.Fatalf("validate --json startup-prepared index emitted invalid JSON %q: %v", stdout, err) } - if _, err := os.Stat(dbPath); !os.IsNotExist(err) { - t.Fatalf("validate --json created or touched missing database: %v", err) + if !validatePayload.Valid || !validatePayload.DatabaseExists { + t.Fatalf("validate --json startup-prepared index payload = %+v, want valid existing database", validatePayload) } stdout, stderr, err = runCmd("status", "--json") if err != nil { - t.Fatalf("status --json missing index failed: %v stdout=%q stderr=%q", err, stdout, stderr) + t.Fatalf("status --json startup-prepared index failed: %v stdout=%q stderr=%q", err, stdout, stderr) } if stderr != "" { - t.Fatalf("status --json missing index wrote stderr: %q", stderr) + t.Fatalf("status --json startup-prepared index wrote stderr: %q", stderr) } var statusPayload struct { Database struct { @@ -400,13 +381,16 @@ func TestStatusAndValidateMissingIndexAreReadOnlyMachineDiagnostics(t *testing.T } `json:"index"` } if err := json.Unmarshal([]byte(stdout), &statusPayload); err != nil { - t.Fatalf("status --json missing index emitted invalid JSON %q: %v", stdout, err) + t.Fatalf("status --json startup-prepared index emitted invalid JSON %q: %v", stdout, err) + } + if !statusPayload.Database.Exists || statusPayload.Database.Size == 0 { + t.Fatalf("status --json startup-prepared database payload = %+v, want existing database", statusPayload.Database) } - if statusPayload.Database.Exists || statusPayload.Database.Size != 0 || statusPayload.Index.Usable || statusPayload.Index.TotalFiles != 0 || statusPayload.Index.TotalMessages != 0 { - t.Fatalf("status --json missing index payload = %+v, want no index and zero counts", statusPayload) + if statusPayload.Index.TotalFiles != 0 || statusPayload.Index.TotalMessages != 0 { + t.Fatalf("status --json startup-prepared index counts = %+v, want zero files/messages", statusPayload.Index) } - if _, err := os.Stat(dbPath); !os.IsNotExist(err) { - t.Fatalf("status --json created or touched missing database: %v", err) + if statusPayload.Index.Usable { + t.Fatalf("status --json startup-prepared index usable=%v with total_files=0; production contract requires usable=false for empty index", statusPayload.Index.Usable) } } @@ -453,7 +437,7 @@ func TestSearchRobotDiagnosticIsMachineReadableAndPreservesIndexBytes(t *testing dbPath := newUnsupportedIndexedConsumerDB(t) before := readDBBytes(t, dbPath) - stdout, stderr, err := runCmd("search", "sentinel", "--robot", "--indexed-only") + stdout, stderr, err := runCmd("search", "sentinel", "--robot") if err == nil { t.Fatalf("search --robot unsupported index succeeded; stdout=%q stderr=%q", stdout, stderr) } @@ -463,7 +447,7 @@ func TestSearchRobotDiagnosticIsMachineReadableAndPreservesIndexBytes(t *testing for _, want := range []string{ "diagnostic_code=" + string(compat.CodeUnsupportedLineage), "diagnostic_summary=", - "diagnostic_continuation_argv=recover --from " + dbPath + " --dry-run", + fmt.Sprintf(`diagnostic_continuation_argv=["recover","--from","%s","--dry-run"]`, dbPath), } { if !strings.Contains(stdout, want) { t.Fatalf("search --robot diagnostic missing %q in %q", want, stdout) @@ -474,50 +458,86 @@ func TestSearchRobotDiagnosticIsMachineReadableAndPreservesIndexBytes(t *testing } } -func TestLiveWALDiagnosticDoesNotClaimMigrationFailureOrRecovery(t *testing.T) { +func TestLiveWALStartupUsesCompatibleIndexWithoutRecoveryDiagnostic(t *testing.T) { dbPath, closeWriter := newLiveWALDiagnosticDB(t) defer closeWriter() - before := snapshotSQLiteFiles(t, dbPath) - for _, argv := range [][]string{{"status", "--json"}, {"validate", "--json"}} { - t.Run(strings.Join(argv, " "), func(t *testing.T) { - got := runJSONDiagnosticAllowNoContinuation(t, argv, compat.CodeIndexStale) - if len(got.Continuation) != 0 { - t.Fatalf("%v continuation=%v, want none", argv, got.Continuation) - } - summary := strings.ToLower(got.Summary) - for _, want := range []string{"cannot be inspected without side effects", "wal", "uncheckpointed frames"} { - if !strings.Contains(summary, want) { - t.Fatalf("%v summary missing %q: %q", argv, want, got.Summary) - } - } - for _, forbidden := range []string{"migration_failed", "recover --from", "--dry-run"} { - if strings.Contains(strings.ToLower(got.Code+" "+got.Summary+" "+strings.Join(got.Continuation, " ")), forbidden) { - t.Fatalf("%v emitted false recovery/migration guidance %q in %+v", argv, forbidden, got) - } - } - assertSQLiteFilesUnchanged(t, dbPath, before) - }) - } + t.Run("status --json", func(t *testing.T) { + stdout, stderr, err := runCmd("status", "--json") + if err != nil { + t.Fatalf("status --json failed with live WAL; stdout=%q stderr=%q err=%v", stdout, stderr, err) + } + if stderr != "" { + t.Fatalf("status --json wrote stderr: %q", stderr) + } + var payload map[string]any + if err := json.Unmarshal([]byte(stdout), &payload); err != nil { + t.Fatalf("status --json emitted invalid JSON %q: %v", stdout, err) + } + if _, hasCode := payload["code"]; hasCode { + t.Fatalf("status --json unexpectedly emitted diagnostic payload: %q", stdout) + } + database, ok := payload["database"].(map[string]any) + if !ok { + t.Fatalf("status --json missing database payload: %q", stdout) + } + index, ok := payload["index"].(map[string]any) + if !ok { + t.Fatalf("status --json missing index payload: %q", stdout) + } + if exists, _ := database["exists"].(bool); !exists { + t.Fatalf("status --json database.exists=false, payload=%v", database) + } + totalFiles, _ := index["total_files"].(float64) + if totalFiles < 1 { + t.Fatalf("status --json expected indexed rows, payload=%v", index) + } + }) + + t.Run("validate --json", func(t *testing.T) { + stdout, stderr, err := runCmd("validate", "--json") + if err != nil { + t.Fatalf("validate --json failed with live WAL; stdout=%q stderr=%q err=%v", stdout, stderr, err) + } + if stderr != "" { + t.Fatalf("validate --json wrote stderr: %q", stderr) + } + if strings.Contains(strings.ToLower(stdout), "diagnostic") { + t.Fatalf("validate --json unexpectedly emitted diagnostic payload: %q", stdout) + } + var payload struct { + Valid bool `json:"valid"` + } + if err := json.Unmarshal([]byte(stdout), &payload); err != nil { + t.Fatalf("validate --json emitted invalid JSON %q: %v", stdout, err) + } + if !payload.Valid { + t.Fatalf("validate --json payload = %+v, want valid=true", payload) + } + }) t.Run("status text", func(t *testing.T) { stdout, stderr, err := runCmd("status") - if err == nil { - t.Fatalf("status text succeeded with live WAL; stdout=%q stderr=%q", stdout, stderr) + if err != nil { + t.Fatalf("status text failed with live WAL; stdout=%q stderr=%q err=%v", stdout, stderr, err) } - out := strings.ToLower(stdout + stderr) - for _, want := range []string{"diagnostic: " + string(compat.CodeIndexStale), "wal", "uncheckpointed frames"} { - if !strings.Contains(out, strings.ToLower(want)) { - t.Fatalf("status text live WAL diagnostic missing %q: %q", want, stderr) - } + if stderr != "" { + t.Fatalf("status text wrote stderr: %q", stderr) } - for _, forbidden := range []string{"continuation:", "migration_failed", "recover --from", "--dry-run"} { - if strings.Contains(out, forbidden) { - t.Fatalf("status text emitted false recovery/migration guidance %q in %q", forbidden, stderr) + out := strings.ToLower(stdout) + if strings.Contains(out, "diagnostic:") || strings.Contains(out, "continuation:") || strings.Contains(out, "recover --from") { + t.Fatalf("status text emitted unexpected recovery/diagnostic guidance: %q", stdout) + } + for _, want := range []string{"files indexed", "messages indexed"} { + if !strings.Contains(out, want) { + t.Fatalf("status text missing %q in %q", want, stdout) } } - assertSQLiteFilesUnchanged(t, dbPath, before) }) + + if _, err := os.Stat(dbPath); err != nil { + t.Fatalf("live WAL database path missing after startup: %v", err) + } } func TestRecoveryContinuationExecutesInConfiguredSamePathContextWithEmptyWAL(t *testing.T) { @@ -533,10 +553,26 @@ func TestRecoveryContinuationExecutesInConfiguredSamePathContextWithEmptyWAL(t * t.Fatalf("stat empty WAL before continuation: %v", err) } - diagnostic := continuationFor(compat.Diagnostic{Code: compat.CodeUnsupportedLineage, Summary: "fixture diagnostic"}, dbPath) + emptyInputs := filepath.Join(t.TempDir(), "empty-inputs") + if err := os.MkdirAll(emptyInputs, 0o755); err != nil { + t.Fatalf("mkdir empty recovery inputs: %v", err) + } + cfg := &config.Config{DatabasePath: dbPath, SessionDirs: []string{emptyInputs}} + startupDiagnostic := continuationFor(compat.Diagnostic{Code: compat.CodeUnsupportedLineage, Summary: "fixture diagnostic"}, dbPath) + startupErr := indexDiagnosticError{diagnostic: startupDiagnostic} + var stdout, stderr bytes.Buffer - if err := run(&stdout, &stderr, diagnostic.Continuation); err != nil { - t.Fatalf("execute continuation %v: %v\nstdout=%q stderr=%q", diagnostic.Continuation, err, stdout.String(), stderr.String()) + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg, Failure: &startupFailure{ + Stage: startupStageIndexPrepare, + Cause: startupErr, + Diagnostic: startupDiagnostic, + Recoverable: true, + }} + }) + root.SetArgs(startupDiagnostic.Continuation) + if err := root.Execute(); err != nil { + t.Fatalf("execute continuation %v after startup diagnostic %s: %v\nstdout=%q stderr=%q", startupDiagnostic.Continuation, startupDiagnostic.Code, err, stdout.String(), stderr.String()) } if !strings.Contains(stdout.String(), "recovery dry run") { t.Fatalf("continuation output = %q, want recovery dry run", stdout.String()) diff --git a/cmd/backscroll/config.go b/cmd/backscroll/config.go index 561b7b4..9877575 100644 --- a/cmd/backscroll/config.go +++ b/cmd/backscroll/config.go @@ -15,8 +15,9 @@ func newConfigCmd(stdout, stderr io.Writer) *cobra.Command { var jsonFormat bool cmd := &cobra.Command{ - Use: "config", - Short: "Show effective configuration and input manifests", + Use: "config", + Short: "Show effective configuration and input manifests", + SilenceUsage: true, Long: `Config displays the effective configuration, including: - Database path - Session directories @@ -25,7 +26,11 @@ func newConfigCmd(stdout, stderr io.Writer) *cobra.Command { Use --json to output as JSON.`, RunE: func(cmd *cobra.Command, args []string) error { - return runConfig(stdout, stderr, jsonFormat) + startup := startupResultFrom(cmd) + if startup.Config == nil { + return fmt.Errorf("startup configuration unavailable") + } + return runConfig(stdout, stderr, startup.Config, jsonFormat) }, } @@ -34,12 +39,7 @@ Use --json to output as JSON.`, return cmd } -func runConfig(stdout, stderr io.Writer, jsonFormat bool) error { - cfg, err := config.Load() - if err != nil { - return fmt.Errorf("load config: %w", err) - } - +func runConfig(stdout, stderr io.Writer, cfg *config.Config, jsonFormat bool) error { // Resolve active inputs defs, mode, err := input_config.ActiveInputs(cfg.SessionDirs) if err != nil { diff --git a/cmd/backscroll/diagnostics_test.go b/cmd/backscroll/diagnostics_test.go index 9b0b772..9846216 100644 --- a/cmd/backscroll/diagnostics_test.go +++ b/cmd/backscroll/diagnostics_test.go @@ -41,13 +41,13 @@ func TestSearchZeroResultHintsToStderr(t *testing.T) { defer cleanup() // Initialize the database by running validate - _, _, err := runCmd("validate", "--indexed-only") + _, _, err := runCmd("validate") if err != nil { // validate may fail if no sessions exist, which is fine } // A query that cannot match anything; --json keeps stdout a clean empty array. - out, stderr, err := runCmd("search", "zzqqxx_no_such_token_zzqqxx", "--json", "--indexed-only") + out, stderr, err := runCmd("search", "zzqqxx_no_such_token_zzqqxx", "--json") if err != nil { t.Fatalf("search error: %v\nstderr: %s", err, stderr) } @@ -88,12 +88,12 @@ func TestSearchShortToolQueryWarnsToStderr(t *testing.T) { defer cleanup() // Initialize the database by running validate - _, _, err := runCmd("validate", "--indexed-only") + _, _, err := runCmd("validate") if err != nil { // validate may fail if no sessions exist, which is fine } - out, stderr, err := runCmd("search", "go", "--content-type", "tool", "--json", "--indexed-only") + out, stderr, err := runCmd("search", "go", "--content-type", "tool", "--json") if err != nil { t.Fatalf("search error: %v\nstderr: %s", err, stderr) } diff --git a/cmd/backscroll/index_policy.go b/cmd/backscroll/index_policy.go index 82c22b5..c1ab27c 100644 --- a/cmd/backscroll/index_policy.go +++ b/cmd/backscroll/index_policy.go @@ -21,11 +21,9 @@ type indexCommandClass uint8 const ( indexDataRead indexCommandClass = iota indexMutation - indexDiagnostic - indexRemediation ) -func prepareIndex(ctx context.Context, cfg *config.Config, class indexCommandClass, autoSync bool) (*storage.Database, *compat.Diagnostic, error) { +func prepareIndex(ctx context.Context, cfg *config.Config, class indexCommandClass) (*storage.Database, *compat.Diagnostic, error) { if cfg == nil { return nil, &compat.Diagnostic{Code: compat.CodeIndexStale, Summary: "index configuration is unavailable"}, fmt.Errorf("index configuration is unavailable") } @@ -35,31 +33,14 @@ func prepareIndex(ctx context.Context, cfg *config.Config, class indexCommandCla return nil, &d, err } - if class == indexDataRead && !autoSync { - if _, statErr := os.Stat(cfg.DatabasePath); os.IsNotExist(statErr) { - return nil, nil, fmt.Errorf("backscroll database not found: %s: %w", cfg.DatabasePath, statErr) - } else if statErr != nil { - return nil, nil, fmt.Errorf("stat database: %w", statErr) - } - } - - openPrepared := func() (*storage.Database, *compat.Diagnostic, error) { - switch class { - case indexDataRead: - if !autoSync { - return openReadOnlyCurrentIndex(ctx, cfg.DatabasePath) - } - return storage.OpenCompatible(ctx, cfg.DatabasePath) - case indexMutation: - return storage.OpenCompatible(ctx, cfg.DatabasePath) - case indexDiagnostic, indexRemediation: - return openImmutableCurrentIndex(ctx, cfg.DatabasePath) - default: - return nil, nil, fmt.Errorf("unknown index command class %d", class) - } + var db *storage.Database + var diag *compat.Diagnostic + switch class { + case indexDataRead, indexMutation: + db, diag, err = storage.OpenCompatible(ctx, cfg.DatabasePath) + default: + return nil, nil, fmt.Errorf("unknown index command class %d", class) } - - db, diag, err := openPrepared() if diag != nil { d := continuationFor(*diag, activePath) return nil, &d, nil @@ -76,68 +57,6 @@ func prepareIndex(ctx context.Context, cfg *config.Config, class indexCommandCla return nil, &d, err } - if autoSync { - if closeErr := db.Close(); closeErr != nil { - d := continuationFor(compat.Diagnostic{Code: compat.CodeIndexStale, Summary: fmt.Sprintf("close prepared index before sync: %v", closeErr)}, activePath) - return nil, &d, closeErr - } - if err := maybeAutoSync(cfg); err != nil { - d := continuationFor(compat.Diagnostic{Code: compat.CodeIndexStale, Summary: fmt.Sprintf("index sync failed: %v", err)}, activePath) - return nil, &d, err - } - db, diag, err = openPrepared() - if diag != nil { - d := continuationFor(*diag, activePath) - return nil, &d, nil - } - if err != nil { - if errors.Is(err, storage.ErrImmutableReadOnlyWALUnsafe) { - d := compat.Diagnostic{ - Code: compat.CodeIndexStale, - Summary: fmt.Sprintf("current index snapshot cannot be inspected without side effects while its WAL has uncheckpointed frames; close the writer or checkpoint the database, then retry: %v", err), - } - return nil, &d, err - } - d := continuationFor(compat.Diagnostic{Code: compat.CodeMigrationFailed, Summary: fmt.Sprintf("prepare index after sync failed: %v", err)}, activePath) - return nil, &d, err - } - } - - return db, nil, nil -} - -func openReadOnlyCurrentIndex(ctx context.Context, path string) (*storage.Database, *compat.Diagnostic, error) { - return openCurrentIndexWith(ctx, path, storage.OpenReadOnly) -} - -func openImmutableCurrentIndex(ctx context.Context, path string) (*storage.Database, *compat.Diagnostic, error) { - return openCurrentIndexWith(ctx, path, storage.OpenImmutableReadOnly) -} - -func openCurrentIndexWith(ctx context.Context, path string, open func(string) (*storage.Database, error)) (*storage.Database, *compat.Diagnostic, error) { - db, err := open(path) - if err != nil { - return nil, nil, err - } - plan, diag, inspectErr := compat.InspectIndex(ctx, db.DB()) - if inspectErr != nil || diag != nil { - closeErr := closeIndexDB(db, inspectErr) - if diag != nil && closeErr != nil { - diag.Summary = strings.TrimSpace(diag.Summary) + "; " + closeErr.Error() - } - return nil, diag, closeErr - } - if len(plan.Steps) > 0 { - diag := &compat.Diagnostic{ - Code: compat.CodeIndexStale, - Summary: fmt.Sprintf("index schema %s has %d pending migration step(s)", plan.From.Signature, len(plan.Steps)), - } - if closeErr := closeIndexDB(db, nil); closeErr != nil { - diag.Summary += "; " + closeErr.Error() - return nil, diag, closeErr - } - return nil, diag, nil - } return db, nil, nil } @@ -187,21 +106,36 @@ func writeDiagnostic(stdout, stderr io.Writer, d compat.Diagnostic, jsonMode boo } func writeRobotDiagnostic(stdout io.Writer, d compat.Diagnostic) error { - if _, err := fmt.Fprintf(stdout, "diagnostic_code=%s\n", d.Code); err != nil { + if _, err := fmt.Fprintf(stdout, "diagnostic_code=%s\n", robotEscape(string(d.Code))); err != nil { return err } - if _, err := fmt.Fprintf(stdout, "diagnostic_summary=%s\n", strings.TrimSpace(d.Summary)); err != nil { + if _, err := fmt.Fprintf(stdout, "diagnostic_summary=%s\n", robotEscape(strings.TrimSpace(d.Summary))); err != nil { return err } if len(d.Continuation) > 0 { - if _, err := fmt.Fprintf(stdout, "diagnostic_continuation_argv=%s\n", strings.Join(d.Continuation, " ")); err != nil { + encoded, err := json.Marshal(d.Continuation) + if err != nil { + return fmt.Errorf("encode diagnostic continuation argv: %w", err) + } + if _, err := fmt.Fprintf(stdout, "diagnostic_continuation_argv=%s\n", robotEscape(string(encoded))); err != nil { return err } } return nil } +func robotEscape(value string) string { + value = strings.ReplaceAll(value, `\`, `\\`) + value = strings.ReplaceAll(value, "\r", `\r`) + value = strings.ReplaceAll(value, "\n", `\n`) + return value +} + func refuseIndex(stdout, stderr io.Writer, d compat.Diagnostic, jsonMode, robotMode bool) error { + return refuseIndexWithCause(stdout, stderr, d, nil, jsonMode, robotMode) +} + +func refuseIndexWithCause(stdout, stderr io.Writer, d compat.Diagnostic, cause error, jsonMode, robotMode bool) error { var err error if robotMode { err = writeRobotDiagnostic(stdout, d) @@ -211,17 +145,22 @@ func refuseIndex(stdout, stderr io.Writer, d compat.Diagnostic, jsonMode, robotM if err != nil { return err } - return indexDiagnosticError{diagnostic: d} + return indexDiagnosticError{diagnostic: d, cause: cause} } type indexDiagnosticError struct { diagnostic compat.Diagnostic + cause error } func (e indexDiagnosticError) Error() string { return fmt.Sprintf("%s: %s", e.diagnostic.Code, strings.TrimSpace(e.diagnostic.Summary)) } +func (e indexDiagnosticError) Unwrap() error { + return e.cause +} + func indexPolicyMachineArgs(args []string) bool { for _, arg := range args { if arg == "--json" || arg == "--robot" || strings.HasPrefix(arg, "--json=") || strings.HasPrefix(arg, "--robot=") { diff --git a/cmd/backscroll/index_policy_test.go b/cmd/backscroll/index_policy_test.go index 06eae0c..3e0f15a 100644 --- a/cmd/backscroll/index_policy_test.go +++ b/cmd/backscroll/index_policy_test.go @@ -8,6 +8,7 @@ import ( "fmt" "os" "path/filepath" + "regexp" "strings" "testing" @@ -23,7 +24,6 @@ func TestStaleIndexBlocksIndexBackedCommands(t *testing.T) { mutation bool }{ {"search", []string{"search", "sentinel"}, false}, - {"search-indexed-only", []string{"search", "sentinel", "--indexed-only"}, false}, {"search-json", []string{"search", "sentinel", "--json"}, false}, {"search-robot", []string{"search", "sentinel", "--robot"}, false}, {"list", []string{"list"}, false}, @@ -77,22 +77,6 @@ func TestStaleIndexBlocksIndexBackedCommands(t *testing.T) { } } -func TestIndexedOnlyDoesNotBypassStaleBlock(t *testing.T) { - dbPath := newUnsupportedIndexedConsumerDB(t) - var stdout, stderr bytes.Buffer - err := run(&stdout, &stderr, []string{"search", "sentinel", "--indexed-only", "--source-path", "/sentinel/%"}) - if err == nil { - t.Fatalf("indexed-only search succeeded; stdout=%q stderr=%q", stdout.String(), stderr.String()) - } - combined := stdout.String() + stderr.String() - if !strings.Contains(combined, string(compat.CodeUnsupportedLineage)) { - t.Fatalf("missing stale diagnostic for %s; stdout=%q stderr=%q", dbPath, stdout.String(), stderr.String()) - } - if strings.Contains(combined, "sentinel") { - t.Fatalf("indexed-only/filter path emitted cached sentinel output; stdout=%q stderr=%q", stdout.String(), stderr.String()) - } -} - func TestMachineModesCarryDiagnosticCodeAndContinuation(t *testing.T) { for _, tc := range []struct { name string @@ -129,30 +113,129 @@ func TestMachineModesCarryDiagnosticCodeAndContinuation(t *testing.T) { if !strings.Contains(out, "diagnostic_code="+string(compat.CodeUnsupportedLineage)) { t.Fatalf("robot diagnostic missing code: %q", out) } - if !strings.Contains(out, "diagnostic_continuation_argv=recover --from "+dbPath+" --dry-run") { - t.Fatalf("robot diagnostic missing exact continuation: %q", out) + wantContinuation := fmt.Sprintf(`diagnostic_continuation_argv=["recover","--from","%s","--dry-run"]`, dbPath) + if !strings.Contains(out, wantContinuation) { + t.Fatalf("robot diagnostic missing encoded continuation %q: %q", wantContinuation, out) } } }) } } -func TestIndexedOnlyRejectsPendingMigrationWithoutMutating(t *testing.T) { - dbPath := newFixtureIndexDB(t, "v12.sql") - before := readDBBytes(t, dbPath) +func TestHumanDiagnosticRenderedOnceWithoutCobraEcho(t *testing.T) { + dbPath := newUnsupportedIndexedConsumerDB(t) + var stdout, stderr bytes.Buffer + err := run(&stdout, &stderr, []string{"search", "sentinel"}) + if err == nil { + t.Fatalf("human diagnostic command succeeded; stdout=%q stderr=%q", stdout.String(), stderr.String()) + } + if strings.Contains(stdout.String(), "sentinel") { + t.Fatalf("human diagnostic emitted cached sentinel rows: %q", stdout.String()) + } + diagnosticLines := 0 + for _, line := range strings.Split(stderr.String(), "\n") { + if strings.HasPrefix(line, "diagnostic:") { + diagnosticLines++ + } + } + if diagnosticLines != 1 { + t.Fatalf("diagnostic lines=%d, want 1; stderr=%q", diagnosticLines, stderr.String()) + } + if strings.Contains(stderr.String(), "Error:") { + t.Fatalf("stderr duplicated by Cobra error echo: %q", stderr.String()) + } + if !strings.Contains(stderr.String(), "continuation: recover --from "+dbPath+" --dry-run") { + t.Fatalf("stderr missing continuation path: %q", stderr.String()) + } +} + +func TestMandatoryStartupIndexesMarkdownDocumentForSearch(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "index.db") + root := filepath.Join(t.TempDir(), "notes") setIndexPolicyEnv(t, dbPath, t.TempDir()) + writeInputManifest(t, root, "markdown_document", root, []string{"*.md"}, nil) + path := filepath.Join(root, "decision.md") + writeFile(t, path, "# Decision\n\nperennial sqlite sentinel\n") var stdout, stderr bytes.Buffer - err := run(&stdout, &stderr, []string{"search", "sentinel", "--indexed-only"}) - if err == nil { - t.Fatalf("indexed-only pending migration succeeded; stdout=%q stderr=%q", stdout.String(), stderr.String()) + err := run(&stdout, &stderr, []string{"search", "perennial sqlite sentinel", "--all-projects", "--source-path", path, "--json"}) + if err != nil { + t.Fatalf("search: %v stderr=%q", err, stderr.String()) } - combined := stdout.String() + stderr.String() - if !strings.Contains(combined, string(compat.CodeIndexStale)) || !strings.Contains(combined, "pending migration") { - t.Fatalf("missing pending-migration diagnostic; stdout=%q stderr=%q err=%v", stdout.String(), stderr.String(), err) + var rows []minimalSearchResult + if err := json.Unmarshal(stdout.Bytes(), &rows); err != nil { + t.Fatalf("invalid JSON %q: %v", stdout.String(), err) } - if !bytes.Equal(readDBBytes(t, dbPath), before) { - t.Fatal("indexed-only pending migration mutated the database") + if len(rows) != 1 || rows[0].SourcePath != path { + t.Fatalf("rows=%+v, want indexed markdown path %s", rows, path) + } +} + +func TestMandatoryStartupMarkdownSearchSurvivesMissingSourceFile(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "index.db") + root := filepath.Join(t.TempDir(), "notes") + setIndexPolicyEnv(t, dbPath, t.TempDir()) + writeInputManifest(t, root, "markdown_document", root, []string{"*.md"}, nil) + path := filepath.Join(root, "decision.md") + writeFile(t, path, "# Decision\n\nperennial sqlite sentinel\n") + + // First run must trigger mandatory startup ingestion from markdown. + var firstStdout, firstStderr bytes.Buffer + if err := run(&firstStdout, &firstStderr, []string{"search", "perennial sqlite sentinel", "--all-projects", "--source-path", path, "--json"}); err != nil { + t.Fatalf("initial search: %v stdout=%q stderr=%q", err, firstStdout.String(), firstStderr.String()) + } + var firstRows []minimalSearchResult + if err := json.Unmarshal(firstStdout.Bytes(), &firstRows); err != nil { + t.Fatalf("initial invalid JSON %q: %v", firstStdout.String(), err) + } + if len(firstRows) != 1 || firstRows[0].SourcePath != path { + t.Fatalf("initial rows=%+v, want indexed markdown path %s", firstRows, path) + } + + if err := os.Remove(path); err != nil { + t.Fatalf("remove markdown source: %v", err) + } + if _, err := os.Stat(path); !os.IsNotExist(err) { + t.Fatalf("source path still available after removal: stat err=%v", err) + } + + // Second run proves retrieval from the perennial SQLite index via public search API, + // even when the original markdown file is unavailable. + var secondStdout, secondStderr bytes.Buffer + if err := run(&secondStdout, &secondStderr, []string{"search", "perennial sqlite sentinel", "--all-projects", "--source-path", path, "--json"}); err != nil { + t.Fatalf("search after source removal: %v stdout=%q stderr=%q", err, secondStdout.String(), secondStderr.String()) + } + var secondRows []minimalSearchResult + if err := json.Unmarshal(secondStdout.Bytes(), &secondRows); err != nil { + t.Fatalf("post-removal invalid JSON %q: %v", secondStdout.String(), err) + } + if len(secondRows) != 1 || secondRows[0].SourcePath != path { + t.Fatalf("post-removal rows=%+v, want indexed markdown path %s", secondRows, path) + } + if !strings.Contains(secondRows[0].Snippet, "sqlite sentinel") { + t.Fatalf("post-removal snippet missing sentinel: rows=%+v", secondRows) + } +} + +func TestMandatoryStartupIndexesMarkdownSectionsForSearch(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "index.db") + root := filepath.Join(t.TempDir(), "notes") + setIndexPolicyEnv(t, dbPath, t.TempDir()) + writeInputManifest(t, root, "markdown_sections", root, []string{"*.md"}, nil) + path := filepath.Join(root, "decisions.md") + writeFile(t, path, "# Decisions\n\n## First\nalpha\n\n## Second\nsection sentinel omega\n") + + var stdout, stderr bytes.Buffer + err := run(&stdout, &stderr, []string{"search", "section sentinel omega", "--all-projects", "--source-path", path, "--json"}) + if err != nil { + t.Fatalf("search: %v stderr=%q", err, stderr.String()) + } + var rows []minimalSearchResult + if err := json.Unmarshal(stdout.Bytes(), &rows); err != nil { + t.Fatalf("invalid JSON %q: %v", stdout.String(), err) + } + if len(rows) != 1 || rows[0].SourcePath != path || !strings.Contains(rows[0].Snippet, "sentinel") { + t.Fatalf("rows=%+v, want second indexed section from %s", rows, path) } } @@ -209,25 +292,71 @@ func TestAutoSyncFailuresBlockCachedConsumers(t *testing.T) { for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - dbPath := newSupportedIndexedConsumerDB(t) - root := filepath.Join(t.TempDir(), "inputs-root") - if err := os.MkdirAll(root, 0o755); err != nil { - t.Fatalf("mkdir input root: %v", err) + modes := []struct { + name string + argv []string + machine bool + json bool + }{ + {name: "text", argv: []string{"search", "sentinel"}}, + {name: "robot", argv: []string{"search", "sentinel", "--robot"}, machine: true}, + {name: "json", argv: []string{"search", "sentinel", "--json"}, machine: true, json: true}, } - setIndexPolicyEnv(t, dbPath, t.TempDir()) - tc.setup(t, root) + for _, mode := range modes { + t.Run(mode.name, func(t *testing.T) { + dbPath := newSupportedIndexedConsumerDB(t) + root := filepath.Join(t.TempDir(), "inputs-root") + if err := os.MkdirAll(root, 0o755); err != nil { + t.Fatalf("mkdir input root: %v", err) + } + setIndexPolicyEnv(t, dbPath, t.TempDir()) + tc.setup(t, root) - var stdout, stderr bytes.Buffer - err := run(&stdout, &stderr, []string{"search", "sentinel"}) - if err == nil { - t.Fatalf("auto-sync %s failure succeeded; stdout=%q stderr=%q", tc.name, stdout.String(), stderr.String()) - } - combined := stdout.String() + stderr.String() - if !strings.Contains(combined, string(compat.CodeIndexStale)) || !strings.Contains(combined, tc.wantError) { - t.Fatalf("missing %s diagnostic; stdout=%q stderr=%q err=%v", tc.wantError, stdout.String(), stderr.String(), err) - } - if strings.Contains(combined, "sentinel") { - t.Fatalf("auto-sync %s failure emitted cached sentinel: stdout=%q stderr=%q", tc.name, stdout.String(), stderr.String()) + var stdout, stderr bytes.Buffer + err := run(&stdout, &stderr, mode.argv) + if err == nil { + t.Fatalf("auto-sync %s failure succeeded; stdout=%q stderr=%q", tc.name, stdout.String(), stderr.String()) + } + combined := stdout.String() + stderr.String() + if strings.Contains(combined, "sentinel cached") { + t.Fatalf("auto-sync %s failure emitted cached sentinel row: stdout=%q stderr=%q", tc.name, stdout.String(), stderr.String()) + } + if strings.Contains(stdout.String(), "result_") { + t.Fatalf("auto-sync %s failure emitted cached result rows: stdout=%q stderr=%q", tc.name, stdout.String(), stderr.String()) + } + + if mode.machine && stderr.Len() != 0 { + t.Fatalf("auto-sync %s %s diagnostic wrote stderr: %q", tc.name, mode.name, stderr.String()) + } + + if mode.json { + var diag struct { + Code string `json:"code"` + Summary string `json:"summary"` + Continuation []string `json:"continuation_argv"` + } + if err := json.Unmarshal(stdout.Bytes(), &diag); err != nil { + t.Fatalf("auto-sync %s JSON diagnostic is invalid: %v stdout=%q", tc.name, err, stdout.String()) + } + if diag.Code != string(compat.CodeIndexStale) { + t.Fatalf("auto-sync %s JSON code=%q, want %q", tc.name, diag.Code, compat.CodeIndexStale) + } + if !strings.Contains(diag.Summary, tc.wantError) { + t.Fatalf("auto-sync %s JSON summary=%q missing %q", tc.name, diag.Summary, tc.wantError) + } + if len(diag.Continuation) == 0 { + t.Fatalf("auto-sync %s JSON continuation is empty: stdout=%q", tc.name, stdout.String()) + } + if strings.Contains(stdout.String(), `"source_path"`) { + t.Fatalf("auto-sync %s JSON diagnostic leaked result payload: %q", tc.name, stdout.String()) + } + return + } + + if !strings.Contains(combined, string(compat.CodeIndexStale)) || !strings.Contains(combined, tc.wantError) { + t.Fatalf("missing %s diagnostic; stdout=%q stderr=%q err=%v", tc.wantError, stdout.String(), stderr.String(), err) + } + }) } }) } @@ -268,28 +397,70 @@ func TestMigrationFailureBlocksCachedConsumer(t *testing.T) { } func TestMachineModesSuppressAutoSyncProgressStderr(t *testing.T) { - dbPath := newSupportedIndexedConsumerDB(t) - root := filepath.Join(t.TempDir(), "inputs-root") - if err := os.MkdirAll(root, 0o755); err != nil { - t.Fatalf("mkdir input root: %v", err) - } - setIndexPolicyEnv(t, dbPath, t.TempDir()) - writeInputManifest(t, root, "claude", root, []string{"*.jsonl"}, nil) - writeFile(t, filepath.Join(root, "session.jsonl"), `{"type":"message","message":{"role":"user","content":"fresh machine"}}`+"\n") + for _, tc := range []struct { + name string + flag string + query string + content string + validate func(t *testing.T, stdout string) + }{ + { + name: "json", + flag: "--json", + query: "fresh machine json sentinel", + content: "fresh machine json sentinel", + validate: func(t *testing.T, stdout string) { + t.Helper() + var rows []minimalSearchResult + if err := json.Unmarshal([]byte(stdout), &rows); err != nil { + t.Fatalf("startup contaminated JSON stdout %q: %v", stdout, err) + } + if len(rows) == 0 { + t.Fatalf("JSON machine search returned no rows: stdout=%q", stdout) + } + }, + }, + { + name: "robot", + flag: "--robot", + query: "fresh machine robot sentinel", + content: "fresh machine robot sentinel", + validate: func(t *testing.T, stdout string) { + t.Helper() + assertRobotResultLines(t, stdout) + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + dbPath := newSupportedIndexedConsumerDB(t) + root := filepath.Join(t.TempDir(), "inputs-root") + if err := os.MkdirAll(root, 0o755); err != nil { + t.Fatalf("mkdir input root: %v", err) + } + setIndexPolicyEnv(t, dbPath, t.TempDir()) + writeInputManifest(t, root, "claude", root, []string{"*.jsonl"}, nil) + writeFile(t, filepath.Join(root, "session.jsonl"), fmt.Sprintf(`{"type":"message","message":{"role":"user","content":%q}}`+"\n", tc.content)) - for _, argv := range [][]string{{"search", "fresh", "--json", "--all-projects"}, {"search", "fresh", "--robot", "--all-projects"}} { - t.Run(strings.Join(argv, " "), func(t *testing.T) { var stdout, stderr bytes.Buffer + argv := []string{"search", tc.query, tc.flag, "--all-projects"} if err := run(&stdout, &stderr, argv); err != nil { t.Fatalf("machine search failed: %v stdout=%q stderr=%q", err, stdout.String(), stderr.String()) } if stderr.Len() != 0 { t.Fatalf("machine mode wrote progress to stderr: %q", stderr.String()) } + tc.validate(t, stdout.String()) }) } } +func TestValidateRobotResultLinesRejectsNonResultLines(t *testing.T) { + err := validateRobotResultLines("result_0_source=session\nthis is not a result line") + if err == nil { + t.Fatal("validateRobotResultLines accepted non-result nonempty output") + } +} + func TestRebuildFailsOnDerivedMaintenanceError(t *testing.T) { dbPath := newSupportedIndexedConsumerDB(t) setIndexPolicyEnv(t, dbPath, t.TempDir()) @@ -318,7 +489,7 @@ func TestResolveActiveIndexPathPropagatesBrokenSymlink(t *testing.T) { if err := os.Symlink(filepath.Join(dir, "missing-target.db"), broken); err != nil { t.Skipf("symlink unavailable: %v", err) } - _, diag, err := prepareIndex(context.Background(), &config.Config{DatabasePath: broken}, indexDataRead, true) + _, diag, err := prepareIndex(context.Background(), &config.Config{DatabasePath: broken}, indexDataRead) if err == nil { t.Fatalf("broken symlink resolved successfully; diagnostic=%+v", diag) } @@ -327,6 +498,38 @@ func TestResolveActiveIndexPathPropagatesBrokenSymlink(t *testing.T) { } } +func assertRobotResultLines(t *testing.T, robotStdout string) { + t.Helper() + if err := validateRobotResultLines(robotStdout); err != nil { + t.Fatal(err) + } +} + +func validateRobotResultLines(robotStdout string) error { + if strings.TrimSpace(robotStdout) == "" { + return fmt.Errorf("robot machine search returned empty stdout") + } + resultLine := regexp.MustCompile(`^result_[0-9]+_[a-z_]+=.*$`) + resultLines := 0 + for _, raw := range strings.Split(strings.TrimSpace(robotStdout), "\n") { + line := strings.TrimSpace(raw) + if line == "" { + continue + } + if !resultLine.MatchString(line) { + return fmt.Errorf("invalid robot output line %q; expected result_N_field=value", raw) + } + if strings.Contains(line, "\r") { + return fmt.Errorf("robot output contains unescaped carriage return: %q", raw) + } + resultLines++ + } + if resultLines == 0 { + return fmt.Errorf("robot machine search returned no result lines: %q", robotStdout) + } + return nil +} + func assertDiagnosticFields(t *testing.T, code string, continuation []string, dbPath string) { t.Helper() if code != string(compat.CodeUnsupportedLineage) { diff --git a/cmd/backscroll/legacy_sources_test.go b/cmd/backscroll/legacy_sources_test.go index c03dd78..830e8e4 100644 --- a/cmd/backscroll/legacy_sources_test.go +++ b/cmd/backscroll/legacy_sources_test.go @@ -17,13 +17,12 @@ func TestLegacySourcesBlockEveryExecutableCommand(t *testing.T) { name string args []string }{ - {name: "search", args: []string{"search"}}, - {name: "read", args: []string{"read", "--path", missingPath}}, + {name: "search", args: []string{"search", "needle"}}, {name: "list", args: []string{"list"}}, {name: "patterns", args: []string{"patterns", "--kind", "commands"}}, {name: "rebuild", args: []string{"rebuild"}}, {name: "purge", args: []string{"purge", "--before", "2026-01-01"}}, - {name: "validate", args: []string{"validate", "--indexed-only"}}, + {name: "validate", args: []string{"validate"}}, {name: "status", args: []string{"status"}}, {name: "config", args: []string{"config"}}, {name: "annotate", args: []string{"annotate", "--uuid", "test-uuid", "--kind", "correction", "--label", "false-positive"}}, @@ -52,13 +51,12 @@ func TestLegacySourcesPreflightHasNoSideEffects(t *testing.T) { name string args []string }{ - {name: "search", args: []string{"search"}}, - {name: "read", args: []string{"read", "--path", missingPath}}, + {name: "search", args: []string{"search", "needle"}}, {name: "list", args: []string{"list"}}, {name: "patterns", args: []string{"patterns", "--kind", "commands"}}, {name: "rebuild", args: []string{"rebuild"}}, {name: "purge", args: []string{"purge", "--before", "2026-01-01"}}, - {name: "validate", args: []string{"validate", "--indexed-only"}}, + {name: "validate", args: []string{"validate"}}, {name: "status", args: []string{"status"}}, {name: "config", args: []string{"config"}}, {name: "annotate", args: []string{"annotate", "--uuid", "test-uuid", "--kind", "correction", "--label", "false-positive"}}, diff --git a/cmd/backscroll/list.go b/cmd/backscroll/list.go index 212cd43..9e6256e 100644 --- a/cmd/backscroll/list.go +++ b/cmd/backscroll/list.go @@ -5,7 +5,6 @@ import ( "encoding/json" "fmt" "io" - "os" "github.com/spf13/cobra" @@ -20,7 +19,6 @@ func newListCmd(stdout, stderr io.Writer) *cobra.Command { recent int jsonFormat bool robotFormat bool - indexedOnly bool order string limit int offset int @@ -38,13 +36,21 @@ Use --order to sort results (e.g., timestamp:desc). Use --limit to restrict result count. Use --offset to skip results. Use --recent N to show N most recent sessions (legacy flag; prefer --order timestamp:desc --limit N). -Use --indexed-only to skip auto-sync (read existing index only). Use --json to output as JSON.`, + Args: func(cmd *cobra.Command, args []string) error { + return validateCommandBeforeStartup(cmd, args, func(_ *cobra.Command, args []string) error { + if len(args) > 0 { + return fmt.Errorf("unexpected positional argument %q; use --text for text search", args[0]) + } + return nil + }, nil) + }, RunE: func(cmd *cobra.Command, args []string) error { - if len(args) > 0 { - return fmt.Errorf("unexpected positional argument %q; use --text for text search", args[0]) + startup := startupResultFrom(cmd) + if startup.Config == nil { + return fmt.Errorf("startup configuration unavailable") } - return runList(stdout, stderr, project, allProjects, recent, jsonFormat, robotFormat, indexedOnly, + return runList(cmd.Context(), stdout, stderr, startup.Config, project, allProjects, recent, jsonFormat, robotFormat, order, limit, offset) }, } @@ -52,7 +58,6 @@ Use --json to output as JSON.`, cmd.Flags().StringVar(&project, "project", "", "Filter to project") cmd.Flags().BoolVar(&allProjects, "all-projects", false, "List all projects") cmd.Flags().IntVar(&recent, "recent", 20, "Show N most recent sessions (0 = all)") - cmd.Flags().BoolVar(&indexedOnly, "indexed-only", false, "Read existing index without auto-sync") cmd.Flags().BoolVar(&jsonFormat, "json", false, "Output as JSON") cmd.Flags().BoolVar(&robotFormat, "robot", false, "Output in robot format") cmd.Flags().StringVar(&order, "order", "", "Sort results (e.g., timestamp:desc)") @@ -62,27 +67,11 @@ Use --json to output as JSON.`, return cmd } -func runList(stdout, stderr io.Writer, - project string, allProjects bool, recent int, jsonFormat, robotFormat, indexedOnly bool, +func runList(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, + project string, allProjects bool, recent int, jsonFormat, robotFormat bool, order string, limit, offset int) (retErr error) { - cfg, err := config.Load() - if err != nil { - return fmt.Errorf("load config: %w", err) - } - - if indexedOnly { - if _, statErr := os.Stat(cfg.DatabasePath); os.IsNotExist(statErr) { - if jsonFormat { - _, _ = fmt.Fprintf(stdout, "{\"count\":0,\"sessions\":[]}\n") - } else { - _, _ = fmt.Fprintf(stdout, "No sessions found\n") - } - return nil - } - } - - db, diag, err := prepareIndex(context.Background(), cfg, indexDataRead, !indexedOnly) + db, diag, err := prepareIndex(ctx, cfg, indexDataRead) if diag != nil { return refuseIndex(stdout, stderr, *diag, jsonFormat, robotFormat) } diff --git a/cmd/backscroll/list_coverage_test.go b/cmd/backscroll/list_coverage_test.go index b07e41a..561ddc5 100644 --- a/cmd/backscroll/list_coverage_test.go +++ b/cmd/backscroll/list_coverage_test.go @@ -1,7 +1,7 @@ package main import ( - "strings" + "encoding/json" "testing" ) @@ -9,12 +9,33 @@ func TestListJSONSeeded(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("list", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("list", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } - if !strings.Contains(stdout, "cov/s.jsonl") && !strings.Contains(stdout, "[]") { - t.Errorf("unexpected list json: %q", stdout) + + var payload struct { + Count int `json:"count"` + Sessions []struct { + Path string `json:"Path"` + Project string `json:"Project"` + } `json:"sessions"` + } + if err := json.Unmarshal([]byte(stdout), &payload); err != nil { + t.Fatalf("list --json emitted invalid JSON %q: %v", stdout, err) + } + if payload.Count == 0 || len(payload.Sessions) == 0 { + t.Fatalf("list --json returned empty sessions after seeded startup: %+v", payload) + } + foundSeed := false + for _, s := range payload.Sessions { + if s.Path == "/cov/s.jsonl" && s.Project == "covproj" { + foundSeed = true + break + } + } + if !foundSeed { + t.Fatalf("list --json missing seeded /cov/s.jsonl@covproj session: %+v", payload.Sessions) } } @@ -22,7 +43,7 @@ func TestListRobotSeeded(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - if _, _, err := runCmd("list", "--robot", "--all-projects", "--indexed-only"); err != nil { + if _, _, err := runCmd("list", "--robot", "--all-projects"); err != nil { t.Fatalf("run: %v", err) } } @@ -31,7 +52,7 @@ func TestListOrderAscWithLimitOffset(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - if _, _, err := runCmd("list", "--order", "timestamp:asc", "--limit", "1", "--offset", "1", "--all-projects", "--indexed-only"); err != nil { + if _, _, err := runCmd("list", "--order", "timestamp:asc", "--limit", "1", "--offset", "1", "--all-projects"); err != nil { t.Fatalf("run: %v", err) } } @@ -40,7 +61,7 @@ func TestListInvalidOrderRejected(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - if _, _, err := runCmd("list", "--order", "nonsense:updown", "--all-projects", "--indexed-only"); err == nil { + if _, _, err := runCmd("list", "--order", "nonsense:updown", "--all-projects"); err == nil { t.Log("invalid order accepted silently (documenting current behavior)") } } @@ -49,7 +70,7 @@ func TestListRecentZeroAll(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - if _, _, err := runCmd("list", "--recent", "0", "--all-projects", "--indexed-only"); err != nil { + if _, _, err := runCmd("list", "--recent", "0", "--all-projects"); err != nil { t.Fatalf("run: %v", err) } } @@ -58,7 +79,7 @@ func TestListProjectFilterSeeded(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("list", "--project", "covproj", "--indexed-only") + stdout, _, err := runCmd("list", "--project", "covproj") if err != nil { t.Fatalf("run: %v", err) } diff --git a/cmd/backscroll/main.go b/cmd/backscroll/main.go index c0824dd..05ad50c 100644 --- a/cmd/backscroll/main.go +++ b/cmd/backscroll/main.go @@ -1,31 +1,30 @@ package main import ( - "errors" "fmt" "io" "os" "time" "github.com/pablontiv/picokit/autoupdate" - "github.com/spf13/cobra" - - "github.com/pablontiv/backscroll/internal/config" - "github.com/pablontiv/backscroll/internal/input_config" ) var version = "dev" func main() { if err := run(os.Stdout, os.Stderr, os.Args[1:]); err != nil { - var indexErr indexDiagnosticError - if !errors.As(err, &indexErr) { + if !diagnosticAlreadyRendered(err) { _, _ = fmt.Fprintln(os.Stderr, err) } os.Exit(1) } } +func diagnosticAlreadyRendered(err error) bool { + _, ok := err.(indexDiagnosticError) + return ok +} + // newUpdater is the single wiring point for autoupdate. It is called with no // envDisable argument, so no environment variable can disable a released binary // — the only exemption is version=="dev", which picokit applies intrinsically. @@ -48,7 +47,6 @@ func run(stdout, stderr io.Writer, args []string) error { rootCmd := buildRootCmd(stdout, stderr) if indexPolicyMachineArgs(args) { - rootCmd.SilenceErrors = true rootCmd.SilenceUsage = true } rootCmd.SetArgs(args) @@ -66,43 +64,3 @@ func run(stdout, stderr io.Writer, args []string) error { return err } - -func buildRootCmd(stdout, stderr io.Writer) *cobra.Command { - root := &cobra.Command{ - Use: "backscroll", - Short: "A permanent, searchable record of your coding-agent sessions", - Long: `Backscroll turns your coding-agent sessions into a permanent, searchable -record of what happened. It indexes Claude Code, Pi and OpenCode sessions into -SQLite and keeps them after the session files expire. - -Prose and tool activity are indexed separately — a Porter-stemmed FTS5 index for -conversation, a trigram index for commands, paths and errors — and an unfiltered -query merges both by rank position (RRF).`, - Version: version, - PersistentPreRunE: func(cmd *cobra.Command, args []string) error { - inputsDir, err := input_config.InputsDir() - if err != nil { - return fmt.Errorf("resolve inputs directory: %w", err) - } - return config.ValidateNoLegacySources(inputsDir) - }, - } - root.SetOut(stdout) - root.SetErr(stderr) - - root.AddCommand( - newSearchCmd(stdout, stderr), - newReadCmd(stdout, stderr), - newListCmd(stdout, stderr), - newPatternsCmd(stdout, stderr), - newRebuildCmd(stdout, stderr), - newPurgeCmd(stdout, stderr), - newValidateCmd(stdout, stderr), - newStatusCmd(stdout, stderr), - newConfigCmd(stdout, stderr), - newAnnotateCmd(stdout, stderr), - newRecoverCmd(stdout, stderr), - ) - - return root -} diff --git a/cmd/backscroll/main_test.go b/cmd/backscroll/main_test.go index 58ab724..0828408 100644 --- a/cmd/backscroll/main_test.go +++ b/cmd/backscroll/main_test.go @@ -2,8 +2,10 @@ package main import ( "bytes" + "context" "encoding/json" "fmt" + "io" "os" "path/filepath" "strings" @@ -31,10 +33,15 @@ func testEnv(t *testing.T) (dbPath string, cleanup func()) { t.Fatalf("mkdir isolated test env path %s: %v", path, err) } } + emptyInputs := filepath.Join(dir, "empty-inputs") + if err := os.MkdirAll(emptyInputs, 0o755); err != nil { + t.Fatalf("mkdir isolated empty input path %s: %v", emptyInputs, err) + } dbPath = filepath.Join(dir, "test.db") t.Setenv("HOME", homeDir) t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) + t.Setenv("BACKSCROLL_SESSION_DIRS", emptyInputs) db, err := storage.Open(dbPath) if err != nil { t.Fatalf("create isolated test database: %v", err) @@ -70,7 +77,7 @@ func syncForTest(t *testing.T, args ...string) (string, string, error) { if err != nil { return "", "", err } - return "", "", maybeAutoSync(cfg) + return "", "", maybeAutoSync(cfg, io.Discard) } } return "", "", nil @@ -103,7 +110,7 @@ func TestHelp(t *testing.T) { commandsSection := parts[1] // v2 approved root commands that SHOULD be present - approvedV2 := []string{"list", "search", "read", "status", "validate", "rebuild", "purge", "config"} + approvedV2 := []string{"list", "search", "status", "validate", "rebuild", "purge", "config"} for _, cmd := range approvedV2 { if !strings.Contains(commandsSection, "\n "+cmd+" ") && !strings.Contains(commandsSection, "\n "+cmd+"\n") { t.Errorf("--help missing approved v2 command %q", cmd) @@ -120,6 +127,19 @@ func TestHelp(t *testing.T) { } } +func TestValidateHelpDescribesStartupPreparedIndex(t *testing.T) { + out, _, err := runCmd("validate", "--help") + if err != nil { + t.Fatalf("validate --help error: %v", err) + } + if strings.Contains(out, "never auto-syncs") { + t.Fatalf("validate help contains stale no-sync claim: %s", out) + } + if !strings.Contains(out, "Command startup may synchronize active inputs") || !strings.Contains(out, "second sync") { + t.Fatalf("validate help does not describe startup-prepared validation: %s", out) + } +} + func TestVersion(t *testing.T) { out, _, err := runCmd("--version") if err != nil { @@ -198,211 +218,6 @@ func TestStatusJSON(t *testing.T) { } } -func TestRead(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - piFixture := filepath.Join(fixturesDir(), "pi-session.jsonl") - out, _, err := runCmd("read", piFixture) - if err != nil { - t.Fatalf("read error: %v", err) - } - if len(out) == 0 { - t.Error("read returned empty output") - } -} - -func TestReadLargeJSONLLine(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - path := writeLargeJSONLFixture(t) - out, _, err := runCmd("read", path) - if err != nil { - t.Fatalf("read large JSONL error: %v", err) - } - if !strings.Contains(out, "Total messages: 47") { - t.Fatalf("read output missing message count; output prefix: %.200q", out) - } -} - -func TestReadPathTailSemanticHandlesLargeJSONLLine(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - path := writeLargeJSONLFixture(t) - out, _, err := runCmd("read", "--path", path, "--tail", "45", "--semantic") - if err != nil { - t.Fatalf("read --path --tail --semantic error: %v", err) - } - if strings.Contains(out, "bufio.Scanner: token too long") { - t.Fatalf("semantic tail hit scanner token limit: %s", out) - } - if strings.Contains(out, "oversized-") { - t.Fatalf("semantic tail should not include the oversized first row: %.200q", out) - } - for _, want := range []string{"path=", "line=4", "line=48", "role=\"assistant\"", "content=\"final answer\""} { - if !strings.Contains(out, want) { - t.Fatalf("semantic tail output missing %q: %s", want, out) - } - } -} - -func writeLargeJSONLFixture(t *testing.T) string { - t.Helper() - - path := filepath.Join(t.TempDir(), "large.jsonl") - large := "oversized-" + strings.Repeat("x", 70*1024) - lines := []string{ - fmt.Sprintf(`{"type":"message","timestamp":"2024-01-01T00:00:00Z","message":{"role":"user","content":%q}}`, large), - `{"type":"message","timestamp":"2024-01-01T00:00:01Z","message":{"role":"assistant","content":"middle answer"}}`, - `{"type":"message","timestamp":"2024-01-01T00:00:02Z","message":{"role":"user","content":[{"type":"tool_use","id":"tool-1","name":"bash","input":{"command":"echo hi"}}]}}`, - } - for i := 4; i < 48; i++ { - lines = append(lines, fmt.Sprintf(`{"type":"message","timestamp":"2024-01-01T00:00:%02dZ","message":{"role":"assistant","content":"tail item %d"}}`, i, i)) - } - lines = append(lines, `{"type":"message","timestamp":"2024-01-01T00:00:48Z","message":{"role":"assistant","content":"final answer"}}`) - if err := os.WriteFile(path, []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { - t.Fatalf("write fixture: %v", err) - } - return path -} - -func TestReadNonExistent(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - _, _, err := runCmd("read", "/nonexistent/path.jsonl") - if err == nil { - t.Error("expected error for nonexistent file, got nil") - } -} - -func TestReadSemanticDefaultAgentReadable(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - path := writeLargeJSONLFixture(t) - out, stderr, err := runCmd("read", "--path", path, "--tail", "5", "--semantic") - if err != nil { - t.Fatalf("read --semantic error: %v", err) - } - - // Default output should be agent-readable (key=value format) - // and should NOT go to stderr - if stderr != "" { - t.Errorf("stderr should be empty for data output, got: %s", stderr) - } - - // Verify agent-readable format: tab-separated key=value rows - lines := strings.Split(strings.TrimSpace(out), "\n") - if len(lines) == 0 { - t.Error("no output generated") - } - - // Each line should have key=value format (except the final total= line) - for i, line := range lines { - if line == "" { - continue - } - if !strings.Contains(line, "=") { - t.Errorf("line %d not in key=value format: %q", i, line) - } - } -} - -func TestReadSemanticWithPretty(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - path := writeLargeJSONLFixture(t) - out, _, err := runCmd("read", "--path", path, "--tail", "3", "--semantic", "--pretty") - if err != nil { - t.Fatalf("read --semantic --pretty error: %v", err) - } - - // Pretty output should be human-readable (different from default key=value) - // For now, we just verify it runs without error - // Real implementation may add table headers or aligned columns - if len(out) == 0 { - t.Error("--pretty output should not be empty") - } - - // Pretty output should include human-readable elements - // For now just verify it exists and is different from raw key=value - if strings.Count(out, "=") > 3 { - t.Logf("pretty output still has key=value pairs: %s", out[:200]) - } -} - -func TestReadSemanticNoRobotFlag(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - path := writeLargeJSONLFixture(t) - - // Verify that --robot flag is not recognized (removed from v2 UX) - // Expected: error about unknown flag - _, _, err := runCmd("read", "--path", path, "--semantic", "--robot") - if err == nil { - t.Error("--robot flag should not be recognized in v2 CLI (expected error)") - } -} - -func TestReadSemanticPrettyIncludesHeaders(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - path := writeLargeJSONLFixture(t) - out, _, err := runCmd("read", "--path", path, "--tail", "2", "--semantic", "--pretty") - if err != nil { - t.Fatalf("read --semantic --pretty error: %v", err) - } - - // Pretty output should include table headers - if !strings.Contains(out, "Path") || !strings.Contains(out, "Line") { - t.Errorf("pretty output should have headers; got: %s", out[:200]) - } - - // Pretty output should have content aligned in columns - if !strings.Contains(out, "Total rows:") { - t.Errorf("pretty output should have total rows summary; got: %s", out) - } -} - -func TestReadSemanticAgentFormatIsTabSeparated(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - path := writeLargeJSONLFixture(t) - out, _, err := runCmd("read", "--path", path, "--tail", "1", "--semantic") - if err != nil { - t.Fatalf("read --semantic error: %v", err) - } - - // Default agent format should use key=value pairs - lines := strings.Split(strings.TrimSpace(out), "\n") - if len(lines) < 2 { - t.Fatalf("expected at least 2 lines (data + total), got %d", len(lines)) - } - - // Each data line should have key=value pairs separated by spaces - dataLine := lines[0] - - // Verify presence of expected keys in agent format - expectedKeys := []string{"path=", "line=", "ordinal=", "timestamp=", "role=", "kind=", "content="} - for _, key := range expectedKeys { - if !strings.Contains(dataLine, key) { - t.Errorf("agent format missing key %q in line: %q", key, dataLine) - } - } - - // Verify it's NOT pretty format - if strings.Contains(dataLine, "---") || strings.Contains(dataLine, "Total rows:") { - t.Errorf("agent format should not include pretty formatting") - } -} - func TestList(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() @@ -867,7 +682,7 @@ func TestHelpListsAllCommands(t *testing.T) { // v2 approved root commands for _, cmd := range []string{ - "search", "read", "list", "purge", "validate", "status", + "search", "list", "purge", "validate", "status", "rebuild", "config", } { if !strings.Contains(commandsSection, "\n "+cmd+" ") && !strings.Contains(commandsSection, "\n "+cmd+"\n") { @@ -939,30 +754,6 @@ func TestListRecentN(t *testing.T) { _ = out } -func TestListIndexedOnly(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - out, _, err := runCmd("list", "--indexed-only") - if err != nil { - t.Fatalf("list --indexed-only error: %v", err) - } - _ = out -} - -func TestStatusIndexedOnly(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - out, _, err := runCmd("status", "--indexed-only") - if err != nil { - t.Fatalf("status --indexed-only error: %v", err) - } - if !strings.Contains(out, "Backscroll Status") { - t.Errorf("status --indexed-only missing header: %s", out) - } -} - func TestStatusWithDeclarativeInputs(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() @@ -1164,24 +955,28 @@ func TestStatusJSONIndexUsable(t *testing.T) { defer cleanup() t.Setenv("HOME", t.TempDir()) - // No index yet: --indexed-only must report usable=false without creating the DB - out, _, err := runCmd("status", "--json", "--indexed-only") + // Mandatory startup prepares an empty usable index before status runs. + out, _, err := runCmd("status", "--json") if err != nil { - t.Fatalf("status --json --indexed-only error: %v", err) + t.Fatalf("status --json error: %v", err) } var doc map[string]any if err := json.Unmarshal([]byte(out), &doc); err != nil { t.Fatalf("status JSON invalid: %v\noutput: %s", err, out) } + database, ok := doc["database"].(map[string]any) + if !ok { + t.Fatalf("status JSON missing database object: %s", out) + } + if exists, _ := database["exists"].(bool); !exists { + t.Error("expected database.exists=true after mandatory startup prepares an empty index") + } index, ok := doc["index"].(map[string]any) if !ok { t.Fatalf("status JSON missing index object: %s", out) } - if usable, _ := index["usable"].(bool); usable { - t.Error("expected index.usable=false with no index") - } - // After syncing a session, usable must flip to true + // After syncing a session, usable must flip to true and include rows. piDir := filepath.Dir(filepath.Join(fixturesDir(), "claude-tool-events.jsonl")) _, _, _ = syncForTest(t, "sync", "--path", piDir) // Status is read-only; the explicit sync above populated the index. @@ -1281,8 +1076,8 @@ func TestListStructuredFlagsRemoved(t *testing.T) { if err == nil { t.Fatal("expected error: --type flag should be removed from list") } - if !strings.Contains(stderr, "unknown flag") { - t.Errorf("expected 'unknown flag', got: %q", stderr) + if !strings.Contains(err.Error(), "unknown flag") { + t.Errorf("expected unknown flag error, got err=%q stderr=%q", err.Error(), stderr) } } @@ -1294,8 +1089,8 @@ func TestStatsCommandRemoved(t *testing.T) { if err == nil { t.Fatal("expected error: stats command should no longer exist") } - if !strings.Contains(stderr, "unknown command") { - t.Errorf("expected 'unknown command' on stderr, got: %q", stderr) + if !strings.Contains(err.Error(), "unknown command") { + t.Errorf("expected unknown command error, got err=%q stderr=%q", err.Error(), stderr) } } @@ -1342,28 +1137,6 @@ func TestRebuildCommand(t *testing.T) { } } -// TestValidateWithIndexedOnly verifies that validate respects --indexed-only flag. -func TestValidateWithIndexedOnly(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() - - // validate --indexed-only should work on empty DB without erroring - out, _, err := runCmd("validate", "--indexed-only") - if err != nil { - // validate may fail on empty DB (expected), but the flag should be recognized - if strings.Contains(err.Error(), "unknown flag: --indexed-only") { - t.Fatalf("validate does not support --indexed-only flag") - } - // Otherwise, error is acceptable for empty DB - t.Logf("validate --indexed-only error (may be expected on empty DB): %v", err) - return - } - // Success: validate --indexed-only worked - if len(strings.TrimSpace(out)) == 0 { - t.Logf("validate --indexed-only produced empty output") - } -} - // TestConfigCommand verifies that config command exists and shows input manifest info. func TestConfigCommand(t *testing.T) { _, cleanup := testEnv(t) @@ -1396,19 +1169,19 @@ func TestStatusAndValidateAreMaintenanceV2(t *testing.T) { t.Fatalf("sync error: %v", err) } - // status --indexed-only should succeed and show agent-readable output by default - out, _, err := runCmd("status", "--indexed-only") + // status should succeed and show agent-readable output by default + out, _, err := runCmd("status") if err != nil { - t.Fatalf("status --indexed-only error: %v", err) + t.Fatalf("status error: %v", err) } if len(strings.TrimSpace(out)) == 0 { t.Errorf("status produced empty output") } // validate should succeed - out, _, err = runCmd("validate", "--indexed-only") + out, _, err = runCmd("validate") if err != nil { - t.Fatalf("validate --indexed-only error: %v", err) + t.Fatalf("validate error: %v", err) } if !strings.Contains(out, "passed") && !strings.Contains(out, "✓") { t.Logf("validate output may not clearly indicate success: %s", out) @@ -1564,7 +1337,7 @@ roots = ["/home/shared/myproject"] t.Setenv("BACKSCROLL_CONFIG_DIR", cfgDir) t.Setenv("HOME", home) - // Status is read-only; search below is responsible for auto-syncing content. + // Status participates in mandatory startup sync and should prepare indexed content. out, stderr, err := runCmd("status") if err != nil { t.Fatalf("status failed: %v; stderr: %s", err, stderr) @@ -2165,7 +1938,7 @@ func TestAutoSyncReParsesStalePaths(t *testing.T) { DatabasePath: dbPath, SessionDirs: []string{sessionDir}, } - if err := maybeAutoSync(cfg); err != nil { + if err := maybeAutoSync(cfg, io.Discard); err != nil { t.Logf("sync warning (acceptable): %v", err) } @@ -2218,7 +1991,7 @@ func TestAutoSyncCapsStaleParsesPerRun(t *testing.T) { DatabasePath: dbPath, SessionDirs: []string{sessionDir}, } - _ = maybeAutoSync(cfg) + _ = maybeAutoSync(cfg, io.Discard) // Count how many were re-parsed (uuid not NULL) db, _ = storage.Open(dbPath) @@ -2262,7 +2035,7 @@ func TestRebuildProjectResolution(t *testing.T) { // Run rebuild (this should re-resolve projects) var stdout, stderr bytes.Buffer - err = runRebuild(&stdout, &stderr) + err = runRebuild(context.Background(), &stdout, &stderr, &config.Config{DatabasePath: dbPath}) if err != nil { t.Fatalf("rebuild: %v", err) } @@ -2319,7 +2092,7 @@ func TestRebuildMultipleProjects(t *testing.T) { // Run rebuild var stdout, stderr bytes.Buffer - err = runRebuild(&stdout, &stderr) + err = runRebuild(context.Background(), &stdout, &stderr, &config.Config{DatabasePath: dbPath}) if err != nil { t.Fatalf("rebuild: %v", err) } @@ -2449,7 +2222,7 @@ func TestQ3AutoSyncUpgradesStaleTemplates(t *testing.T) { // Auto-sync alone must re-mine the stale path — no rebuild. cfg := &config.Config{DatabasePath: dbPath, SessionDirs: []string{sessionDir}} - if err := maybeAutoSync(cfg); err != nil { + if err := maybeAutoSync(cfg, io.Discard); err != nil { t.Fatalf("maybeAutoSync: %v", err) } @@ -2480,7 +2253,7 @@ func TestQ3AutoSyncUpgradesStaleTemplates(t *testing.T) { } // Idempotence: another sync must not double-count or duplicate matches. - if err := maybeAutoSync(cfg); err != nil { + if err := maybeAutoSync(cfg, io.Discard); err != nil { t.Fatalf("second maybeAutoSync: %v", err) } var occAfter, matchesAfter int @@ -2584,7 +2357,7 @@ func TestQ5AutoSyncClearsSupersededSignalsWithoutRebuild(t *testing.T) { _ = db.Close() cfg := &config.Config{DatabasePath: dbPath, SessionDirs: []string{sessionDir}} - if err := maybeAutoSync(cfg); err != nil { + if err := maybeAutoSync(cfg, io.Discard); err != nil { t.Fatalf("maybeAutoSync: %v", err) } diff --git a/cmd/backscroll/patterns.go b/cmd/backscroll/patterns.go index 6328874..6cba6da 100644 --- a/cmd/backscroll/patterns.go +++ b/cmd/backscroll/patterns.go @@ -24,7 +24,6 @@ func newPatternsCmd(stdout, stderr io.Writer) *cobra.Command { offset int jsonFormat bool robotFormat bool - indexedOnly bool minSupport int minConfidence float64 pending bool @@ -52,13 +51,25 @@ Use --min-support for template/sequence filtering (default 3; minimum occurrence Use --min-length, --max-length for sequence pattern length bounds (default 2, 6). Use --min-confidence for correction filtering (default 0.6; detector confidence threshold). Use --limit, --offset for pagination. -Use --json, --robot for output formats. -Use --indexed-only to skip auto-sync (read existing index only).`, +Use --json, --robot for output formats.`, + Args: func(cmd *cobra.Command, args []string) error { + return validateCommandBeforeStartup(cmd, args, func(_ *cobra.Command, args []string) error { + if len(args) > 0 { + return fmt.Errorf("unexpected positional argument %q", args[0]) + } + return nil + }, func() error { + return validatePatternsRequest(kind, project, allProjects, limit, offset, trend) + }) + }, RunE: func(cmd *cobra.Command, args []string) error { - if len(args) > 0 { - return fmt.Errorf("unexpected positional argument %q", args[0]) + startup := startupResultFrom(cmd) + if startup.Config == nil { + return fmt.Errorf("startup configuration unavailable") } - return runPatterns(stdout, stderr, kind, project, allProjects, tag, limit, offset, jsonFormat, robotFormat, indexedOnly, minSupport, minConfidence, pending, batch, minLength, maxLength, after, before, trend) + return runPatterns(cmd.Context(), stdout, stderr, startup.Config, kind, project, allProjects, tag, limit, offset, + jsonFormat, robotFormat, minSupport, minConfidence, pending, batch, + minLength, maxLength, after, before, trend) }, } @@ -76,7 +87,6 @@ Use --indexed-only to skip auto-sync (read existing index only).`, cmd.Flags().StringVar(&before, "before", "", "Filter before date (ISO 8601, for --kind sequences)") cmd.Flags().BoolVar(&jsonFormat, "json", false, "Output as JSON") cmd.Flags().BoolVar(&robotFormat, "robot", false, "Output in robot format") - cmd.Flags().BoolVar(&indexedOnly, "indexed-only", false, "Read existing index without auto-sync") cmd.Flags().BoolVar(&pending, "pending", false, "Only corrections without a 'correction' annotation (checkpoint resume)") cmd.Flags().IntVar(&batch, "batch", 0, "Alias for --limit (batch size for loop)") cmd.Flags().BoolVar(&trend, "trend", false, "Week-over-week bucketing (--kind commands|failures only)") @@ -86,12 +96,7 @@ Use --indexed-only to skip auto-sync (read existing index only).`, return cmd } -func runPatterns(stdout, stderr io.Writer, - kind string, project string, allProjects bool, tag string, - limit, offset int, jsonFormat, robotFormat, indexedOnly bool, minSupport int, minConfidence float64, pending bool, batch int, - minLength, maxLength int, after, before string, trend bool) (retErr error) { - - // Early flag validation before DB open +func validatePatternsRequest(kind, project string, allProjects bool, limit, offset int, trend bool) error { validKinds := map[string]bool{ "commands": true, "failures": true, @@ -102,25 +107,28 @@ func runPatterns(stdout, stderr io.Writer, if !validKinds[kind] { return fmt.Errorf("unsupported --kind %q (supported: commands, failures, templates, sequences, corrections)", kind) } - if trend && kind != "commands" && kind != "failures" { return fmt.Errorf("--trend only supported for --kind commands|failures, got %q", kind) } - if project != "" && allProjects { return fmt.Errorf("--project and --all-projects are mutually exclusive") } - if limit < 0 || offset < 0 { return fmt.Errorf("--limit and --offset must be >= 0") } + return nil +} - cfg, err := config.Load() - if err != nil { - return fmt.Errorf("load config: %w", err) +func runPatterns(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, + kind string, project string, allProjects bool, tag string, + limit, offset int, jsonFormat, robotFormat bool, minSupport int, minConfidence float64, pending bool, batch int, + minLength, maxLength int, after, before string, trend bool) (retErr error) { + + if err := validatePatternsRequest(kind, project, allProjects, limit, offset, trend); err != nil { + return err } - db, diag, err := prepareIndex(context.Background(), cfg, indexDataRead, !indexedOnly) + db, diag, err := prepareIndex(ctx, cfg, indexDataRead) if diag != nil { return refuseIndex(stdout, stderr, *diag, jsonFormat, robotFormat) } diff --git a/cmd/backscroll/patterns_coverage_test.go b/cmd/backscroll/patterns_coverage_test.go index e687fd2..2cdf37c 100644 --- a/cmd/backscroll/patterns_coverage_test.go +++ b/cmd/backscroll/patterns_coverage_test.go @@ -35,7 +35,7 @@ func TestPatternsCommandsJSONWithData(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "commands", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -48,7 +48,7 @@ func TestPatternsFailuresRobotWithData(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "failures", "--robot", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "failures", "--robot", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -61,7 +61,7 @@ func TestPatternsSequencesRobotSeeded(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - if _, _, err := runCmd("patterns", "--kind", "sequences", "--robot", "--min-support", "1", "--min-length", "2", "--indexed-only"); err != nil { + if _, _, err := runCmd("patterns", "--kind", "sequences", "--robot", "--min-support", "1", "--min-length", "2"); err != nil { t.Fatalf("run: %v", err) } } @@ -70,7 +70,7 @@ func TestPatternsCorrectionsJSONEmpty(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) // creates the DB; corpus has no correction signals - if _, _, err := runCmd("patterns", "--kind", "corrections", "--json", "--all-projects", "--indexed-only"); err != nil { + if _, _, err := runCmd("patterns", "--kind", "corrections", "--json", "--all-projects"); err != nil { t.Fatalf("run: %v", err) } } @@ -95,7 +95,7 @@ func TestPatternsCommandsRobotSeeded(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "commands", "--robot", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--robot", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -108,7 +108,7 @@ func TestPatternsFailuresJSONSeeded(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "failures", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "failures", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -121,7 +121,7 @@ func TestPatternsFailuresTextWithTag(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - if _, _, err := runCmd("patterns", "--kind", "failures", "--tag", "debugging", "--all-projects", "--indexed-only"); err != nil { + if _, _, err := runCmd("patterns", "--kind", "failures", "--tag", "debugging", "--all-projects"); err != nil { t.Fatalf("run: %v", err) } } @@ -130,7 +130,7 @@ func TestPatternsTemplatesMinSupportSeeded(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - if _, _, err := runCmd("patterns", "--kind", "templates", "--min-support", "1", "--all-projects", "--indexed-only"); err != nil { + if _, _, err := runCmd("patterns", "--kind", "templates", "--min-support", "1", "--all-projects"); err != nil { t.Fatalf("run: %v", err) } } @@ -140,7 +140,7 @@ func TestPatternsFailuresTextFormatNullExitCode(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "failures", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "failures", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -159,7 +159,7 @@ func TestPatternsFailuresRobotFormatNullExitCode(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "failures", "--robot", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "failures", "--robot", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -243,7 +243,7 @@ func TestPatternsTrendCommandsText(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedTrendData(t) - stdout, _, err := runCmd("patterns", "--kind", "commands", "--trend", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--trend", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -263,7 +263,7 @@ func TestPatternsTrendCommandsJSON(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedTrendData(t) - stdout, _, err := runCmd("patterns", "--kind", "commands", "--trend", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--trend", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -284,7 +284,7 @@ func TestPatternsTrendCommandsRobot(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedTrendData(t) - stdout, _, err := runCmd("patterns", "--kind", "commands", "--trend", "--robot", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--trend", "--robot", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -301,7 +301,7 @@ func TestPatternsTrendFailuresText(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedTrendData(t) - stdout, _, err := runCmd("patterns", "--kind", "failures", "--trend", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "failures", "--trend", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -318,7 +318,7 @@ func TestPatternsTrendFailuresJSON(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedTrendData(t) - stdout, _, err := runCmd("patterns", "--kind", "failures", "--trend", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "failures", "--trend", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -339,7 +339,7 @@ func TestPatternsTrendFailuresRobot(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedTrendData(t) - stdout, _, err := runCmd("patterns", "--kind", "failures", "--trend", "--robot", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "failures", "--trend", "--robot", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -356,7 +356,7 @@ func TestPatternsCommandsText(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "commands", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -373,7 +373,7 @@ func TestPatternsFailuresText(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "failures", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "failures", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -390,7 +390,7 @@ func TestPatternsTemplatesText(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "templates", "--min-support", "1", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "templates", "--min-support", "1", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -404,7 +404,7 @@ func TestPatternsCorrectionsText(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "corrections", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "corrections", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -419,7 +419,7 @@ func TestPatternsSequencesText(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "sequences", "--min-support", "1", "--min-length", "2", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "sequences", "--min-support", "1", "--min-length", "2", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -440,7 +440,7 @@ func TestPatternsCommandsZeroResultText(t *testing.T) { } _ = db.Close() - stdout, _, err := runCmd("patterns", "--kind", "commands", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -459,7 +459,7 @@ func TestPatternsFailuresZeroResultText(t *testing.T) { } _ = db.Close() - stdout, _, err := runCmd("patterns", "--kind", "failures", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "failures", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -478,7 +478,7 @@ func TestPatternsTemplatesZeroResultText(t *testing.T) { } _ = db.Close() - stdout, _, err := runCmd("patterns", "--kind", "templates", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "templates", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -497,7 +497,7 @@ func TestPatternsCorrectionsZeroResultText(t *testing.T) { } _ = db.Close() - stdout, _, err := runCmd("patterns", "--kind", "corrections", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "corrections", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -516,7 +516,7 @@ func TestPatternsSequencesZeroResultText(t *testing.T) { } _ = db.Close() - stdout, _, err := runCmd("patterns", "--kind", "sequences", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "sequences", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -530,7 +530,7 @@ func TestPatternsCommandsLimitZero(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "commands", "--limit", "0", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--limit", "0", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -543,7 +543,7 @@ func TestPatternsCommandsOffsetBeyond(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "commands", "--offset", "999", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--offset", "999", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -557,7 +557,7 @@ func TestPatternsBatchAlias(t *testing.T) { defer cleanup() seedToolEvents(t) // --batch should alias to --limit for corrections - stdout, _, err := runCmd("patterns", "--kind", "corrections", "--batch", "10", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "corrections", "--batch", "10", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -577,7 +577,7 @@ func TestPatternsCommandsZeroResultJSON(t *testing.T) { } _ = db.Close() - stdout, _, err := runCmd("patterns", "--kind", "commands", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -596,7 +596,7 @@ func TestPatternsFailuresZeroResultJSON(t *testing.T) { } _ = db.Close() - stdout, _, err := runCmd("patterns", "--kind", "failures", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "failures", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -610,7 +610,7 @@ func TestPatternsCommandsJSONOutput(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "commands", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "commands", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -657,7 +657,7 @@ func TestPatternsTemplatesRobotOutput(t *testing.T) { // error result so there is something to template, otherwise this test asserts on an // empty census and would pass or fail for the wrong reason. seedErrorOutput(t) - stdout, _, err := runCmd("patterns", "--kind", "templates", "--robot", "--min-support", "1", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "templates", "--robot", "--min-support", "1", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -679,7 +679,7 @@ func TestPatternsSequencesRobotOutput(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - stdout, _, err := runCmd("patterns", "--kind", "sequences", "--robot", "--min-support", "1", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "sequences", "--robot", "--min-support", "1", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -693,7 +693,7 @@ func TestPatternsCorrectionsRobotOutput(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() seedToolEvents(t) - _, _, err := runCmd("patterns", "--kind", "corrections", "--robot", "--all-projects", "--indexed-only") + _, _, err := runCmd("patterns", "--kind", "corrections", "--robot", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -711,7 +711,7 @@ func TestPatternsTemplatesZeroResultJSON(t *testing.T) { } _ = db.Close() - stdout, _, err := runCmd("patterns", "--kind", "templates", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "templates", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -730,7 +730,7 @@ func TestPatternsSequencesZeroResultJSON(t *testing.T) { } _ = db.Close() - stdout, _, err := runCmd("patterns", "--kind", "sequences", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "sequences", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } @@ -749,7 +749,7 @@ func TestPatternsCorrectionsZeroResultJSON(t *testing.T) { } _ = db.Close() - stdout, _, err := runCmd("patterns", "--kind", "corrections", "--json", "--all-projects", "--indexed-only") + stdout, _, err := runCmd("patterns", "--kind", "corrections", "--json", "--all-projects") if err != nil { t.Fatalf("run: %v", err) } diff --git a/cmd/backscroll/patterns_sequences_test.go b/cmd/backscroll/patterns_sequences_test.go index 0cadfd5..393a11b 100644 --- a/cmd/backscroll/patterns_sequences_test.go +++ b/cmd/backscroll/patterns_sequences_test.go @@ -10,157 +10,97 @@ import ( "testing" ) -func TestPatternsSequencesCommandBasic(t *testing.T) { - t.Setenv("HOME", t.TempDir()) - - // Use in-memory temp dir for config - configDir := filepath.Join(t.TempDir(), ".config/backscroll") - os.MkdirAll(configDir, 0o755) - t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) - - dbPath := filepath.Join(configDir, "backscroll.db") - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) +func setupSequencesHermeticEmptyInputEnv(t *testing.T) (dbPath, cfgDir string) { + t.Helper() + root := t.TempDir() + dbPath = filepath.Join(root, "backscroll.db") + cfgDir = filepath.Join(root, "config") + setIndexPolicyEnv(t, dbPath, cfgDir) + return dbPath, cfgDir +} - // Run patterns command with sequences kind - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - err := run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--indexed-only", - "--json", - }) +func runPatternsSequences(t *testing.T, extraArgs ...string) (string, string, error) { + t.Helper() + var stdout, stderr bytes.Buffer + args := append([]string{"patterns", "--kind", "sequences"}, extraArgs...) + err := run(&stdout, &stderr, args) + return stdout.String(), stderr.String(), err +} - // Command should succeed or gracefully handle no patterns +func TestPatternsSequencesCommandBasic(t *testing.T) { + setupSequencesHermeticEmptyInputEnv(t) + stdout, stderr, err := runPatternsSequences(t, "--json") if err != nil { - t.Logf("command returned: %v (may be expected for empty DB)", err) + t.Fatalf("patterns sequences --json failed on hermetic empty input: %v stdout=%q stderr=%q", err, stdout, stderr) + } + if !strings.Contains(stdout, `"kind":"sequences"`) { + t.Fatalf("patterns sequences --json missing kind marker, stdout=%q", stdout) } } func TestPatternsSequencesCommandText(t *testing.T) { - t.Setenv("HOME", t.TempDir()) - configDir := filepath.Join(t.TempDir(), ".config/backscroll") - os.MkdirAll(configDir, 0o755) - t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) - - dbPath := filepath.Join(configDir, "backscroll.db") - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) - - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - err := run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--indexed-only", - }) - - // Should succeed without crash + setupSequencesHermeticEmptyInputEnv(t) + stdout, stderr, err := runPatternsSequences(t) if err != nil { - t.Logf("command returned: %v (may be expected for empty DB)", err) + t.Fatalf("patterns sequences text failed on hermetic empty input: %v stdout=%q stderr=%q", err, stdout, stderr) + } + if !strings.Contains(stdout, "No patterns found") { + t.Fatalf("expected no-pattern guidance in stdout, got %q", stdout) } - t.Logf("text output: %s", stdout.String()) } func TestPatternsSequencesJSON(t *testing.T) { - t.Setenv("HOME", t.TempDir()) - configDir := filepath.Join(t.TempDir(), ".config/backscroll") - os.MkdirAll(configDir, 0o755) - t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) - - dbPath := filepath.Join(configDir, "backscroll.db") - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) - - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - err := run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--indexed-only", - "--json", - }) - + setupSequencesHermeticEmptyInputEnv(t) + stdout, stderr, err := runPatternsSequences(t, "--json") if err != nil { - t.Logf("command returned: %v (may be expected for empty DB)", err) + t.Fatalf("patterns sequences --json failed: %v stdout=%q stderr=%q", err, stdout, stderr) } - - // Parse JSON to verify structure - var result map[string]interface{} - if len(stdout.Bytes()) > 0 { - if err := json.Unmarshal(stdout.Bytes(), &result); err == nil { - if kind, ok := result["kind"].(string); ok && kind == "sequences" { - t.Logf("JSON output valid: %v", result) - } - } + var result map[string]any + if err := json.Unmarshal([]byte(stdout), &result); err != nil { + t.Fatalf("patterns sequences --json emitted invalid JSON: %v stdout=%q", err, stdout) + } + if kind, _ := result["kind"].(string); kind != "sequences" { + t.Fatalf("patterns sequences --json kind=%q, want sequences payload=%v", kind, result) } } func TestPatternsSequencesRobotFormat(t *testing.T) { - t.Setenv("HOME", t.TempDir()) - configDir := filepath.Join(t.TempDir(), ".config/backscroll") - os.MkdirAll(configDir, 0o755) - t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) - - dbPath := filepath.Join(configDir, "backscroll.db") - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) - - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - err := run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--indexed-only", - "--robot", - }) - + setupSequencesHermeticEmptyInputEnv(t) + stdout, stderr, err := runPatternsSequences(t, "--robot") if err != nil { - t.Logf("command returned: %v (may be expected for empty DB)", err) + t.Fatalf("patterns sequences --robot failed: %v stdout=%q stderr=%q", err, stdout, stderr) + } + if !strings.Contains(stderr, "no results") { + t.Fatalf("patterns sequences --robot missing empty-index hint in stderr=%q", stderr) } - t.Logf("robot output: %s", stdout.String()) } func TestPatternsSequencesWithFlags(t *testing.T) { - t.Setenv("HOME", t.TempDir()) - configDir := filepath.Join(t.TempDir(), ".config/backscroll") - os.MkdirAll(configDir, 0o755) - t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) - - dbPath := filepath.Join(configDir, "backscroll.db") - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) - - // Test with various flags - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - err := run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--indexed-only", + setupSequencesHermeticEmptyInputEnv(t) + stdout, stderr, err := runPatternsSequences(t, "--min-support", "5", "--min-length", "3", "--max-length", "8", "--limit", "10", "--offset", "0", "--json", - }) - + ) if err != nil { - t.Logf("command returned: %v (may be expected for empty DB)", err) + t.Fatalf("patterns sequences with tuning flags failed: %v stdout=%q stderr=%q", err, stdout, stderr) } } func TestPatternsSequencesRobotWithData(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() + _, cfgDir := setupSequencesHermeticEmptyInputEnv(t) - // Seed test data sessionDir := t.TempDir() jsonl := `{"uuid":"u1","message":{"role":"assistant","content":{"type":"tool","name":"Read"}},"type":"message","timestamp":"2026-01-01T00:00:00Z"} {"uuid":"u2","message":{"role":"assistant","content":{"type":"tool","name":"Write"}},"type":"message","timestamp":"2026-01-02T00:00:00Z"} ` if err := os.WriteFile(filepath.Join(sessionDir, "test.jsonl"), []byte(jsonl), 0o644); err != nil { - t.Fatalf("write session: %v", err) + t.Fatalf("write session fixture: %v", err) } - cfgDir := t.TempDir() toml := fmt.Sprintf(`version = 1 [[inputs]] id = "claude-test" @@ -174,137 +114,70 @@ format = "claude" `, sessionDir) inputsDir := filepath.Join(cfgDir, "backscroll", "inputs") if err := os.MkdirAll(inputsDir, 0o755); err != nil { - t.Fatal(err) + t.Fatalf("mkdir inputs dir: %v", err) } if err := os.WriteFile(filepath.Join(inputsDir, "claude-test.inputs.toml"), []byte(toml), 0o644); err != nil { - t.Fatal(err) + t.Fatalf("write input manifest: %v", err) } - t.Setenv("BACKSCROLL_CONFIG_DIR", cfgDir) - - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - err := run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--robot", - }) + stdout, stderr, err := runPatternsSequences(t, "--robot", "--min-support", "1", "--min-length", "2") if err != nil { - t.Logf("command result: %v", err) + t.Fatalf("patterns sequences --robot custom-manifest fixture failed: %v stdout=%q stderr=%q", err, stdout, stderr) } - - output := stdout.String() - if strings.Contains(output, "Sequences") || len(output) > 0 { - t.Logf("robot output: %s", output) + if stderr != "" && !strings.Contains(stderr, "no results") { + t.Fatalf("patterns sequences --robot custom-manifest unexpected stderr=%q stdout=%q", stderr, stdout) } } func TestPatternsSequencesProjectFilter(t *testing.T) { - t.Setenv("HOME", t.TempDir()) - configDir := filepath.Join(t.TempDir(), ".config/backscroll") - os.MkdirAll(configDir, 0o755) - t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) - - dbPath := filepath.Join(configDir, "backscroll.db") - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) - - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - err := run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--indexed-only", - "--project", "myproject", - "--json", - }) - + setupSequencesHermeticEmptyInputEnv(t) + stdout, stderr, err := runPatternsSequences(t, "--project", "myproject", "--json") if err != nil { - t.Logf("command returned: %v (expected for empty DB)", err) + t.Fatalf("patterns sequences --project failed: %v stdout=%q stderr=%q", err, stdout, stderr) } } func TestPatternsSequencesAllProjects(t *testing.T) { - t.Setenv("HOME", t.TempDir()) - configDir := filepath.Join(t.TempDir(), ".config/backscroll") - os.MkdirAll(configDir, 0o755) - t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) - - dbPath := filepath.Join(configDir, "backscroll.db") - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) - - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - err := run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--indexed-only", - "--all-projects", - "--json", - }) - + setupSequencesHermeticEmptyInputEnv(t) + stdout, stderr, err := runPatternsSequences(t, "--all-projects", "--json") if err != nil { - t.Logf("command returned: %v (expected for empty DB)", err) + t.Fatalf("patterns sequences --all-projects failed: %v stdout=%q stderr=%q", err, stdout, stderr) } } func TestPatternsSequencesInvalidMinSupport(t *testing.T) { - t.Setenv("HOME", t.TempDir()) - configDir := filepath.Join(t.TempDir(), ".config/backscroll") - os.MkdirAll(configDir, 0o755) - t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) - - dbPath := filepath.Join(configDir, "backscroll.db") - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) - - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - err := run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--indexed-only", - "--min-support", "-1", - "--json", - }) - - t.Logf("command with invalid --min-support: %v", err) + setupSequencesHermeticEmptyInputEnv(t) + stdout, stderr, err := runPatternsSequences(t, "--min-support", "-1", "--json") + if err != nil { + t.Fatalf("patterns sequences --min-support -1 unexpectedly failed: %v stdout=%q stderr=%q", err, stdout, stderr) + } } func TestPatternsSequencesEmptyDBGuidance(t *testing.T) { - t.Setenv("HOME", t.TempDir()) - configDir := filepath.Join(t.TempDir(), ".config/backscroll") - os.MkdirAll(configDir, 0o755) - t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) - - dbPath := filepath.Join(configDir, "backscroll.db") - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) - - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - _ = run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--indexed-only", - }) - - stderrStr := stderr.String() - t.Logf("stderr guidance: %s", stderrStr) + setupSequencesHermeticEmptyInputEnv(t) + stdout, stderr, err := runPatternsSequences(t) + if err != nil { + t.Fatalf("patterns sequences text failed on empty DB: %v stdout=%q stderr=%q", err, stdout, stderr) + } + for _, want := range []string{"no results", "--all-projects", "backscroll status"} { + if !strings.Contains(stderr, want) { + t.Fatalf("missing guidance %q in stderr=%q", want, stderr) + } + } } func TestPatternsSequencesFullEndToEnd(t *testing.T) { - _, cleanup := testEnv(t) - defer cleanup() + _, cfgDir := setupSequencesHermeticEmptyInputEnv(t) - // Seed multiple tool events that will create a pattern sessionDir := t.TempDir() jsonl := `{"uuid":"u1","message":{"role":"assistant","content":"test output"},"type":"message","timestamp":"2026-01-01T00:00:00Z"} {"uuid":"u2","message":{"role":"assistant","content":{"type":"tool","name":"Read"}},"type":"message","timestamp":"2026-01-01T00:00:01Z"} {"uuid":"u3","message":{"role":"assistant","content":{"type":"tool","name":"Write"}},"type":"message","timestamp":"2026-01-01T00:00:02Z"} ` if err := os.WriteFile(filepath.Join(sessionDir, "test.jsonl"), []byte(jsonl), 0o644); err != nil { - t.Fatalf("write session: %v", err) + t.Fatalf("write session fixture: %v", err) } - cfgDir := t.TempDir() toml := fmt.Sprintf(`version = 1 [[inputs]] id = "claude-test" @@ -318,67 +191,72 @@ format = "claude" `, sessionDir) inputsDir := filepath.Join(cfgDir, "backscroll", "inputs") if err := os.MkdirAll(inputsDir, 0o755); err != nil { - t.Fatal(err) + t.Fatalf("mkdir inputs dir: %v", err) } if err := os.WriteFile(filepath.Join(inputsDir, "claude-test.inputs.toml"), []byte(toml), 0o644); err != nil { - t.Fatal(err) + t.Fatalf("write input manifest: %v", err) } - t.Setenv("BACKSCROLL_CONFIG_DIR", cfgDir) - // Run patterns with different output formats - for _, format := range []string{"", "--json", "--robot"} { - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - args := []string{ - "patterns", - "--kind", "sequences", - "--min-support", "1", - } - if format != "" { - args = append(args, format) - } - err := run(stdout, stderr, args) - t.Logf("patterns sequences %s: err=%v, stdout=%d bytes, stderr=%d bytes", format, err, len(stdout.String()), len(stderr.String())) + for _, tc := range []struct { + name string + args []string + }{ + {name: "text", args: nil}, + {name: "json", args: []string{"--json"}}, + {name: "robot", args: []string{"--robot"}}, + } { + t.Run(tc.name, func(t *testing.T) { + args := append([]string{"--min-support", "1"}, tc.args...) + stdout, stderr, err := runPatternsSequences(t, args...) + if err != nil { + t.Fatalf("patterns sequences %s failed: %v stdout=%q stderr=%q", tc.name, err, stdout, stderr) + } + if tc.name == "json" && !strings.Contains(stdout, `"kind":"sequences"`) { + t.Fatalf("patterns sequences json output missing kind marker: %q", stdout) + } + }) } } func TestPatternsSequencesDefaultMinSupport(t *testing.T) { - t.Setenv("HOME", t.TempDir()) - configDir := filepath.Join(t.TempDir(), ".config/backscroll") - os.MkdirAll(configDir, 0o755) - t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) - - dbPath := filepath.Join(configDir, "backscroll.db") - t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) - - stdout := &bytes.Buffer{} - stderr := &bytes.Buffer{} - _ = run(stdout, stderr, []string{ - "patterns", - "--kind", "sequences", - "--indexed-only", - "--min-support", "0", - "--json", - }) + setupSequencesHermeticEmptyInputEnv(t) + stdout, stderr, err := runPatternsSequences(t, "--min-support", "0", "--json") + if err != nil { + t.Fatalf("patterns sequences --min-support 0 failed: %v stdout=%q stderr=%q", err, stdout, stderr) + } + if !strings.Contains(stdout, `"kind":"sequences"`) { + t.Fatalf("patterns sequences --min-support 0 missing JSON payload: %q", stdout) + } } -// TestPatternsSequencesMalformedCategoriesFails asserts a categories config -// load failure fails the command (non-nil error) instead of masquerading as -// an empty result — scripts rely on the exit code to distinguish the two. -func TestPatternsSequencesMalformedCategoriesFails(t *testing.T) { - tempDir := t.TempDir() - t.Setenv("HOME", tempDir) - t.Setenv("BACKSCROLL_CONFIG_DIR", tempDir) - t.Setenv("BACKSCROLL_DATABASE_PATH", tempDir+"/t.db") - if err := os.MkdirAll(tempDir+"/backscroll", 0o755); err != nil { - t.Fatal(err) +// TestPatternsSequencesMalformedCategoriesReturnsLoadError asserts malformed +// categories with a valid version field fail the command through the genuine +// load chain (getVersion succeeds, parseMapper regexp compile fails). +func TestPatternsSequencesMalformedCategoriesReturnsLoadError(t *testing.T) { + dbPath, cfgDir := setupSequencesHermeticEmptyInputEnv(t) + + categoriesDir := filepath.Join(cfgDir, "backscroll", "inputs") + if err := os.MkdirAll(categoriesDir, 0o755); err != nil { + t.Fatalf("mkdir categories dir: %v", err) } - if err := os.WriteFile(tempDir+"/backscroll/categories.toml", []byte("version = [broken"), 0o644); err != nil { - t.Fatal(err) + badCategories := `version = 2 +[[rule]] +tool = "Bash" +pattern = "[" +category = "BROKEN" +` + if err := os.WriteFile(filepath.Join(categoriesDir, "categories.toml"), []byte(badCategories), 0o644); err != nil { + t.Fatalf("write categories fixture: %v", err) } - var stdout, stderr bytes.Buffer - err := run(&stdout, &stderr, []string{"patterns", "--kind", "sequences", "--indexed-only"}) + + stdout, stderr, err := runPatternsSequences(t) if err == nil { - t.Fatal("malformed categories config must fail the command, got nil error") + t.Fatalf("patterns sequences succeeded with invalid categories regex; stdout=%q stderr=%q db=%s", stdout, stderr, dbPath) + } + errText := err.Error() + for _, want := range []string{"load sequences", "load categories", "compile pattern"} { + if !strings.Contains(errText, want) { + t.Fatalf("patterns sequences error missing %q: %v", want, err) + } } } diff --git a/cmd/backscroll/purge.go b/cmd/backscroll/purge.go index a492da2..91b1519 100644 --- a/cmd/backscroll/purge.go +++ b/cmd/backscroll/purge.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "io" + "time" "github.com/spf13/cobra" @@ -22,8 +23,17 @@ func newPurgeCmd(stdout, stderr io.Writer) *cobra.Command { The date should be in YYYY-MM-DD format (e.g., 2024-01-15). Example: backscroll purge --before 2024-01-01`, + Args: func(cmd *cobra.Command, args []string) error { + return validateCommandBeforeStartup(cmd, args, cobra.NoArgs, func() error { + return validatePurgeBefore(before) + }) + }, RunE: func(cmd *cobra.Command, args []string) error { - return runPurge(stdout, stderr, before) + startup := startupResultFrom(cmd) + if startup.Config == nil { + return fmt.Errorf("startup configuration unavailable") + } + return runPurge(cmd.Context(), stdout, stderr, startup.Config, before) }, } @@ -33,17 +43,25 @@ Example: backscroll purge --before 2024-01-01`, return cmd } -func runPurge(stdout, stderr io.Writer, before string) (retErr error) { +func validatePurgeBefore(before string) error { if before == "" { return fmt.Errorf("--before date is required") } + if _, err := time.Parse(time.RFC3339, before); err == nil { + return nil + } + if _, err := time.Parse("2006-01-02", before); err != nil { + return fmt.Errorf("invalid --before date %q: expected YYYY-MM-DD or RFC3339", before) + } + return nil +} - cfg, err := config.Load() - if err != nil { - return fmt.Errorf("load config: %w", err) +func runPurge(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, before string) (retErr error) { + if err := validatePurgeBefore(before); err != nil { + return err } - db, diag, err := prepareIndex(context.Background(), cfg, indexMutation, false) + db, diag, err := prepareIndex(ctx, cfg, indexMutation) if diag != nil { return refuseIndex(stdout, stderr, *diag, false, false) } diff --git a/cmd/backscroll/read.go b/cmd/backscroll/read.go deleted file mode 100644 index d74ecbc..0000000 --- a/cmd/backscroll/read.go +++ /dev/null @@ -1,128 +0,0 @@ -package main - -import ( - "fmt" - "io" - - "github.com/spf13/cobra" - - "github.com/pablontiv/backscroll/internal/reader" -) - -func newReadCmd(stdout, stderr io.Writer) *cobra.Command { - var path string - var tail int - var semantic bool - var pretty bool - cmd := &cobra.Command{ - Use: "read [path]", - Short: "Read a specific session or plan file", - Long: `Read displays the contents of a session file or plan. -Default output format is agent-readable structured rows. -Use --pretty for human-readable formatting.`, - Args: cobra.MaximumNArgs(1), - RunE: func(cmd *cobra.Command, args []string) error { - locator := path - if len(args) > 0 { - if locator != "" { - return fmt.Errorf("use either --path or positional path, not both") - } - locator = args[0] - } - if locator == "" { - return fmt.Errorf("read requires --path ") - } - if semantic { - return runReadSemantic(stdout, locator, tail, pretty) - } - return runRead(stdout, stderr, locator) - }, - } - cmd.Flags().StringVar(&path, "path", "", "path to the JSONL input file to read") - cmd.Flags().IntVar(&tail, "tail", 0, "return only the last N semantic rows") - cmd.Flags().BoolVar(&semantic, "semantic", false, "output concise semantic text/tool rows") - cmd.Flags().BoolVar(&pretty, "pretty", false, "human-readable formatting") - - return cmd -} - -func runRead(stdout, stderr io.Writer, path string) error { - // Read the session file - messages, err := reader.ReadFile(path) - if err != nil { - return fmt.Errorf("read file: %w", err) - } - - // Format and print - for i, msg := range messages { - _, _ = fmt.Fprintf(stdout, "=== Message %d ===\n", i+1) - _, _ = fmt.Fprintf(stdout, "Role: %s\n", msg.Role) - _, _ = fmt.Fprintf(stdout, "ContentType: %s\n", msg.ContentType) - _, _ = fmt.Fprintf(stdout, "Timestamp: %s\n", msg.Timestamp.Format("2006-01-02 15:04:05 MST")) - _, _ = fmt.Fprintf(stdout, "\n%s\n\n", msg.Content) - } - - _, _ = fmt.Fprintf(stdout, "Total messages: %d\n", len(messages)) - - return nil -} - -func runReadSemantic(stdout io.Writer, path string, tail int, pretty bool) error { - rows, err := reader.ReadSemanticTail(path, tail) - if err != nil { - return fmt.Errorf("read semantic file: %w", err) - } - - if pretty { - return formatSemanticRowsPretty(stdout, rows) - } - return formatSemanticRowsAgent(stdout, rows) -} - -func formatSemanticRowsAgent(stdout io.Writer, rows []reader.SemanticRow) error { - // Agent-readable format: tab-separated key=value pairs (default, no --pretty) - for _, row := range rows { - _, _ = fmt.Fprintf( - stdout, - "path=%q line=%d ordinal=%d timestamp=%q role=%q kind=%q content=%q\n", - row.Path, - row.Line, - row.Ordinal, - row.Timestamp, - row.Role, - row.Kind, - row.Content, - ) - } - _, _ = fmt.Fprintf(stdout, "total=%d\n", len(rows)) - return nil -} - -func formatSemanticRowsPretty(stdout io.Writer, rows []reader.SemanticRow) error { - // Human-readable format with headers and aligned columns - _, _ = fmt.Fprintf(stdout, "Path Line Timestamp Role Kind Content\n") - _, _ = fmt.Fprintf(stdout, "---- ---- --------- ---- ---- -------\n") - for _, row := range rows { - // Truncate long fields for readability - path := row.Path - if len(path) > 40 { - path = "..." + path[len(path)-37:] - } - content := row.Content - if len(content) > 30 { - content = content[:27] + "..." - } - _, _ = fmt.Fprintf( - stdout, - "%-40s %4d %-23s %-11s %-10s %s\n", - path, - row.Line, - row.Timestamp, - row.Role, - row.Kind, - content, - ) - } - _, _ = fmt.Fprintf(stdout, "\nTotal rows: %d\n", len(rows)) - return nil -} diff --git a/cmd/backscroll/rebuild.go b/cmd/backscroll/rebuild.go index 2328afc..60bb6b4 100644 --- a/cmd/backscroll/rebuild.go +++ b/cmd/backscroll/rebuild.go @@ -18,11 +18,15 @@ func newRebuildCmd(stdout, stderr io.Writer) *cobra.Command { Short: "Rebuild the FTS search indexes from the database", SilenceUsage: true, Long: `Rebuild re-derives the FTS search indexes from the database itself and -runs an incremental sync. It never deletes indexed content: sessions whose -source files have expired from disk are preserved (the database is the -perennial event store). Use 'purge' to delete data explicitly.`, +operates on the index synchronized at command startup. It never deletes indexed +content: sessions whose source files have expired from disk are preserved (the +database is the perennial event store). Use 'purge' to delete data explicitly.`, RunE: func(cmd *cobra.Command, args []string) error { - return runRebuild(stdout, stderr) + startup := startupResultFrom(cmd) + if startup.Config == nil { + return fmt.Errorf("startup configuration unavailable") + } + return runRebuild(cmd.Context(), stdout, stderr, startup.Config) }, } @@ -41,13 +45,8 @@ var ( } ) -func runRebuild(stdout, stderr io.Writer) (retErr error) { - cfg, err := config.Load() - if err != nil { - return fmt.Errorf("load config: %w", err) - } - - db, diag, err := prepareIndex(context.Background(), cfg, indexMutation, true) +func runRebuild(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config) (retErr error) { + db, diag, err := prepareIndex(ctx, cfg, indexMutation) if diag != nil { return refuseIndex(stdout, stderr, *diag, false, false) } @@ -93,7 +92,7 @@ func runRebuild(stdout, stderr io.Writer) (retErr error) { } return "" } - resolved, err := rebuildReresolveProjects(db, context.Background(), resolver) + resolved, err := rebuildReresolveProjects(db, ctx, resolver) if err != nil { return fmt.Errorf("project re-resolution: %w", err) } else if resolved > 0 { @@ -103,7 +102,7 @@ func runRebuild(stdout, stderr io.Writer) (retErr error) { // Registry-aware re-resolution: correct historical fallback labels _, _ = fmt.Fprintf(stdout, "Checking registry for project label corrections...\n") registry := projects.LoadGlobalRegistry() - registryMatched, err := rebuildReresolveProjectsWithRegistry(db, context.Background(), registry) + registryMatched, err := rebuildReresolveProjectsWithRegistry(db, ctx, registry) if err != nil { return fmt.Errorf("registry re-resolution: %w", err) } else if registryMatched > 0 { diff --git a/cmd/backscroll/recover.go b/cmd/backscroll/recover.go index 1899080..70950ec 100644 --- a/cmd/backscroll/recover.go +++ b/cmd/backscroll/recover.go @@ -1,7 +1,7 @@ package main import ( - "context" + "errors" "fmt" "io" "strings" @@ -11,6 +11,9 @@ import ( "github.com/spf13/cobra" ) +var recoverExecute = recovery.Execute +var recoverPostInstallSync = maybeAutoSync + func newRecoverCmd(stdout, stderr io.Writer) *cobra.Command { var from string var dryRun bool @@ -20,13 +23,26 @@ func newRecoverCmd(stdout, stderr io.Writer) *cobra.Command { Use: "recover", Short: "Recover stranded database rows into the configured database", SilenceUsage: true, - Args: cobra.NoArgs, + Args: func(cmd *cobra.Command, args []string) error { + return validateCommandBeforeStartup(cmd, args, cobra.NoArgs, func() error { + if from == "" { + return fmt.Errorf("--from path must not be empty") + } + return nil + }) + }, RunE: func(cmd *cobra.Command, args []string) error { - cfg, err := config.Load() - if err != nil { - return err + startup := startupResultFrom(cmd) + startupFailure := optionalStartupFailureError(startup.startupFailure()) + cfg := startup.Config + if cfg == nil { + loaded, err := config.Load() + if err != nil { + return errors.Join(startupFailure, fmt.Errorf("load config for recovery: %w", err)) + } + cfg = loaded } - report, err := recovery.Execute(context.Background(), recovery.Options{ + report, err := recoverExecute(cmd.Context(), recovery.Options{ ActivePath: cfg.DatabasePath, FromPath: from, DryRun: dryRun, @@ -35,7 +51,20 @@ func newRecoverCmd(stdout, stderr io.Writer) *cobra.Command { if backupPath, ok := recovery.RestorableBackupPath(err); ok { _, _ = fmt.Fprintf(stderr, "manual recovery backup path: %s\n", backupPath) } - return err + return errors.Join(startupFailure, fmt.Errorf("recovery failed: %w", err)) + } + if !dryRun { + if err := recoverPostInstallSync(cfg, stderr); err != nil { + installedPath := report.ActivePath + if installedPath == "" { + installedPath = cfg.DatabasePath + } + _, _ = fmt.Fprintf(stderr, "recovery replacement installed at: %s\n", installedPath) + if report.BackupPath != "" { + _, _ = fmt.Fprintf(stderr, "manual recovery backup path: %s\n", report.BackupPath) + } + return errors.Join(startupFailure, fmt.Errorf("post-recovery sync: %w", err)) + } } printRecoveryReport(stdout, report, dryRun) return nil diff --git a/cmd/backscroll/recover_test.go b/cmd/backscroll/recover_test.go index bf09beb..5922f56 100644 --- a/cmd/backscroll/recover_test.go +++ b/cmd/backscroll/recover_test.go @@ -6,6 +6,7 @@ import ( "database/sql" "errors" "fmt" + "io" "os" "path/filepath" "reflect" @@ -15,8 +16,10 @@ import ( "time" "github.com/pablontiv/backscroll/internal/compat" + "github.com/pablontiv/backscroll/internal/config" "github.com/pablontiv/backscroll/internal/recovery" "github.com/pablontiv/backscroll/internal/storage" + "github.com/spf13/cobra" ) type fileSnapshot struct { @@ -25,6 +28,223 @@ type fileSnapshot struct { MTime time.Time } +type firstWriteMarker struct { + bytes.Buffer + events *[]string + marked bool +} + +func (w *firstWriteMarker) Write(p []byte) (int, error) { + if !w.marked { + *w.events = append(*w.events, "report") + w.marked = true + } + return w.Buffer.Write(p) +} + +func buildRecoverRootWithConfig(t *testing.T, stdout, stderr io.Writer, activePath string) *cobra.Command { + t.Helper() + emptyInputs := filepath.Join(t.TempDir(), "empty-inputs") + if err := os.MkdirAll(emptyInputs, 0o755); err != nil { + t.Fatalf("mkdir empty recovery inputs: %v", err) + } + cfg := &config.Config{DatabasePath: activePath, SessionDirs: []string{emptyInputs}} + return buildRootCmdWithStartup(stdout, stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg} + }) +} + +func TestRecoverExecuteReceivesCommandContext(t *testing.T) { + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + type contextKey string + const markerKey contextKey = "recover-context-marker" + baseCtx := context.WithValue(context.Background(), markerKey, "present") + ctx, cancel := context.WithCancel(baseCtx) + cancel() + + called := false + originalExecute := recoverExecute + recoverExecute = func(execCtx context.Context, opts recovery.Options) (recovery.Report, error) { + called = true + if got := execCtx.Value(markerKey); got != "present" { + t.Fatalf("recovery context marker = %v, want present", got) + } + if !errors.Is(execCtx.Err(), context.Canceled) { + t.Fatalf("recovery context err = %v, want context canceled", execCtx.Err()) + } + if opts.ActivePath != cfg.DatabasePath || opts.FromPath != "stranded.db" || !opts.DryRun { + t.Fatalf("recovery options = %+v, want active=%q from=stranded.db dryRun=true", opts, cfg.DatabasePath) + } + return recovery.Report{ActivePath: opts.ActivePath}, nil + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + var stdout, stderr bytes.Buffer + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg} + }) + root.SetContext(ctx) + root.SetArgs([]string{"recover", "--from", "stranded.db", "--dry-run"}) + if err := root.Execute(); err != nil { + t.Fatalf("recover returned error: %v", err) + } + if !called { + t.Fatal("recover execute seam was not called") + } +} + +func TestRecoverPostInstallSyncBeforeReport(t *testing.T) { + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + events := []string{} + stdout := &firstWriteMarker{events: &events} + var stderr bytes.Buffer + + originalExecute := recoverExecute + recoverExecute = func(_ context.Context, opts recovery.Options) (recovery.Report, error) { + events = append(events, "recover") + if opts.ActivePath != cfg.DatabasePath || opts.FromPath != "stranded.db" || opts.DryRun { + t.Fatalf("recovery options = %+v, want active=%q from=stranded.db dryRun=false", opts, cfg.DatabasePath) + } + return recovery.Report{ActivePath: opts.ActivePath}, nil + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + originalPostInstallSync := recoverPostInstallSync + recoverPostInstallSync = func(got *config.Config, progress io.Writer) error { + events = append(events, "sync") + if got != cfg { + t.Fatalf("post-install sync config pointer = %p, want %p", got, cfg) + } + if progress != &stderr { + t.Fatalf("post-install sync progress writer = %T, want stderr buffer", progress) + } + return nil + } + t.Cleanup(func() { recoverPostInstallSync = originalPostInstallSync }) + + root := buildRootCmdWithStartup(stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg} + }) + root.SetArgs([]string{"recover", "--from", "stranded.db"}) + if err := root.Execute(); err != nil { + t.Fatalf("recover returned error: %v", err) + } + want := []string{"recover", "sync", "report"} + if !reflect.DeepEqual(events, want) { + t.Fatalf("events=%v, want %v", events, want) + } +} + +func TestRecoverDryRunSkipsPostInstallSync(t *testing.T) { + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + originalExecute := recoverExecute + recoverExecute = func(_ context.Context, opts recovery.Options) (recovery.Report, error) { + if !opts.DryRun { + t.Fatalf("DryRun = false, want true") + } + return recovery.Report{ActivePath: opts.ActivePath}, nil + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + syncCalled := false + originalPostInstallSync := recoverPostInstallSync + recoverPostInstallSync = func(*config.Config, io.Writer) error { + syncCalled = true + return nil + } + t.Cleanup(func() { recoverPostInstallSync = originalPostInstallSync }) + + var stdout, stderr bytes.Buffer + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg} + }) + root.SetArgs([]string{"recover", "--from", "stranded.db", "--dry-run"}) + if err := root.Execute(); err != nil { + t.Fatalf("recover dry-run returned error: %v", err) + } + if syncCalled { + t.Fatal("post-install sync ran during dry-run") + } +} + +func TestRecoverPostInstallSyncFailurePreservesStartupCause(t *testing.T) { + startupErr := errors.New("injected startup failure") + syncErr := errors.New("injected post-sync failure") + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + installedPath := cfg.DatabasePath + ".installed" + backupPath := cfg.DatabasePath + ".backup-test" + + originalExecute := recoverExecute + recoverExecute = func(context.Context, recovery.Options) (recovery.Report, error) { + return recovery.Report{ActivePath: installedPath, BackupPath: backupPath}, nil + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + originalPostInstallSync := recoverPostInstallSync + recoverPostInstallSync = func(*config.Config, io.Writer) error { + return syncErr + } + t.Cleanup(func() { recoverPostInstallSync = originalPostInstallSync }) + + var stdout, stderr bytes.Buffer + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg, Failure: &startupFailure{ + Stage: startupStageStartupSync, + Cause: startupErr, + Diagnostic: continuationFor(compat.Diagnostic{Code: compat.CodeIndexStale, Summary: startupErr.Error()}, cfg.DatabasePath), + Recoverable: true, + }} + }) + root.SetArgs([]string{"recover", "--from", "stranded.db"}) + err := root.Execute() + if !errors.Is(err, startupErr) { + t.Fatalf("error=%v does not preserve startup failure", err) + } + if !errors.Is(err, syncErr) { + t.Fatalf("error=%v does not preserve post-sync failure", err) + } + if stdout.Len() != 0 { + t.Fatalf("report printed before failed post-sync: %q", stdout.String()) + } + for _, want := range []string{ + "recovery replacement installed at: " + installedPath, + "manual recovery backup path: " + backupPath, + } { + if !strings.Contains(stderr.String(), want) { + t.Fatalf("stderr=%q, want failure diagnostic %q", stderr.String(), want) + } + } +} + +func TestRecoverSuccessfulContinuationRemediatesStartupFailure(t *testing.T) { + startupErr := errors.New("injected startup failure") + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + + originalExecute := recoverExecute + recoverExecute = func(context.Context, recovery.Options) (recovery.Report, error) { + return recovery.Report{ActivePath: cfg.DatabasePath}, nil + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + originalPostInstallSync := recoverPostInstallSync + recoverPostInstallSync = func(*config.Config, io.Writer) error { return nil } + t.Cleanup(func() { recoverPostInstallSync = originalPostInstallSync }) + + var stdout, stderr bytes.Buffer + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg, Failure: &startupFailure{ + Stage: startupStageStartupSync, + Cause: startupErr, + Diagnostic: continuationFor(compat.Diagnostic{Code: compat.CodeIndexStale, Summary: startupErr.Error()}, cfg.DatabasePath), + Recoverable: true, + }} + }) + root.SetArgs([]string{"recover", "--from", "stranded.db"}) + if err := root.Execute(); err != nil { + t.Fatalf("recover returned startup failure after successful remediation: %v", err) + } +} + func TestRecoverDryRunMatchesUnionApplyPlanWithoutWrites(t *testing.T) { dir := t.TempDir() home := filepath.Join(dir, "home") @@ -72,7 +292,7 @@ func TestRecoverDryRunMatchesUnionApplyPlanWithoutWrites(t *testing.T) { t.Chdir(dir) var stdout, stderr bytes.Buffer - root := buildRootCmd(&stdout, &stderr) + root := buildRecoverRootWithConfig(t, &stdout, &stderr, activePath) root.SetArgs([]string{"recover", "--from", fromPath, "--dry-run"}) err := root.Execute() if err != nil { @@ -126,7 +346,7 @@ func TestRecoverCommandPreservesApplyFailureAs(t *testing.T) { t.Chdir(dir) var stdout, stderr bytes.Buffer - root := buildRootCmd(&stdout, &stderr) + root := buildRecoverRootWithConfig(t, &stdout, &stderr, activePath) root.SetArgs([]string{"recover", "--from", fromPath}) err := root.Execute() if err == nil { @@ -176,7 +396,7 @@ func TestRecoverApplyReportsActualBackupAndCounts(t *testing.T) { t.Chdir(dir) var stdout, stderr bytes.Buffer - root := buildRootCmd(&stdout, &stderr) + root := buildRecoverRootWithConfig(t, &stdout, &stderr, activePath) root.SetArgs([]string{"recover", "--from", fromPath}) if err := root.Execute(); err != nil { t.Fatalf("recover apply returned error: %v\nstderr=%s", err, stderr.String()) @@ -237,7 +457,7 @@ func TestRecoverInstalledDestinationPassesIndexedValidation(t *testing.T) { t.Chdir(dir) var recoverOut, recoverErr bytes.Buffer - recoverCmd := buildRootCmd(&recoverOut, &recoverErr) + recoverCmd := buildRecoverRootWithConfig(t, &recoverOut, &recoverErr, activePath) recoverCmd.SetArgs([]string{"recover", "--from", strandedPath}) if err := recoverCmd.Execute(); err != nil { t.Fatalf("recover: %v\nstderr=%s", err, recoverErr.String()) @@ -256,7 +476,7 @@ func TestRecoverInstalledDestinationPassesIndexedValidation(t *testing.T) { var validateOut, validateErr bytes.Buffer validateCmd := buildRootCmd(&validateOut, &validateErr) - validateCmd.SetArgs([]string{"validate", "--indexed-only"}) + validateCmd.SetArgs([]string{"validate"}) if err := validateCmd.Execute(); err != nil { t.Fatalf("validate recovered destination: %v\nstdout=%s\nstderr=%s", err, validateOut.String(), validateErr.String()) } @@ -306,7 +526,7 @@ func TestRecoverCommandReturnsStructuredApplyFailure(t *testing.T) { t.Chdir(dir) var stdout, stderr bytes.Buffer - root := buildRootCmd(&stdout, &stderr) + root := buildRecoverRootWithConfig(t, &stdout, &stderr, missingActivePath) root.SetArgs([]string{"recover", "--from", fromPath}) err := root.Execute() if err == nil { @@ -321,7 +541,7 @@ func TestRecoverCommandReturnsStructuredApplyFailure(t *testing.T) { } } -func TestRecoverCommandEmptyFromReturnsApplyFailure(t *testing.T) { +func TestRecoverCommandRejectsEmptyFromBeforeApply(t *testing.T) { dir := t.TempDir() home := filepath.Join(dir, "home") if err := os.MkdirAll(home, 0o755); err != nil { @@ -335,18 +555,18 @@ func TestRecoverCommandEmptyFromReturnsApplyFailure(t *testing.T) { t.Chdir(dir) var stdout, stderr bytes.Buffer - root := buildRootCmd(&stdout, &stderr) + root := buildRecoverRootWithConfig(t, &stdout, &stderr, activePath) root.SetArgs([]string{"recover", "--from", ""}) err := root.Execute() if err == nil { - t.Fatal("recover command succeeded; want structured missing --from failure") + t.Fatal("recover command succeeded; want empty --from validation failure") } - var failure *recovery.ApplyFailure - if !errors.As(err, &failure) { - t.Fatalf("command error %T %[1]v, want *recovery.ApplyFailure", err) + if !strings.Contains(err.Error(), "--from path must not be empty") { + t.Fatalf("command error = %v, want empty --from validation failure", err) } - if failure.Phase != recovery.ApplyFailurePhase("source-read") || failure.ActivePath == "" || failure.FromPath != "" { - t.Fatalf("ApplyFailure = %+v, want source-read with active and missing from", failure) + var failure *recovery.ApplyFailure + if errors.As(err, &failure) { + t.Fatalf("command reached recovery apply: %+v", failure) } } @@ -369,7 +589,7 @@ func TestRecoverCommandPathCanonicalizationFailurePreservesApplyFailureAs(t *tes t.Chdir(dir) var stdout, stderr bytes.Buffer - root := buildRootCmd(&stdout, &stderr) + root := buildRecoverRootWithConfig(t, &stdout, &stderr, activePath) root.SetArgs([]string{"recover", "--from", fromPath}) execErr := root.Execute() if execErr == nil { @@ -393,7 +613,7 @@ func TestRecoverCommandPathCanonicalizationFailurePreservesApplyFailureAs(t *tes func TestRecoverRejectsMissingFrom(t *testing.T) { var stdout, stderr bytes.Buffer - cmd := buildRootCmd(&stdout, &stderr) + cmd := buildRecoverRootWithConfig(t, &stdout, &stderr, filepath.Join(t.TempDir(), "active.db")) cmd.SetArgs([]string{"recover", "--dry-run"}) err := cmd.Execute() if err == nil { @@ -439,7 +659,7 @@ func TestRecoverHasNoGeneralMergeFlags(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { var stdout, stderr bytes.Buffer - cmd := buildRootCmd(&stdout, &stderr) + cmd := buildRecoverRootWithConfig(t, &stdout, &stderr, filepath.Join(t.TempDir(), "active.db")) cmd.SetArgs(tt.args) err := cmd.Execute() if err == nil { diff --git a/cmd/backscroll/search.go b/cmd/backscroll/search.go index d9e4c1e..ddff4e2 100644 --- a/cmd/backscroll/search.go +++ b/cmd/backscroll/search.go @@ -35,7 +35,6 @@ func newSearchCmd(stdout, stderr io.Writer) *cobra.Command { lexicalOnly bool similarityThreshold float64 text string - indexedOnly bool ) cmd := &cobra.Command{ @@ -56,21 +55,22 @@ Use --tag to filter sessions by auto-detected tags. Use --source-path to filter by indexed source path (exact, SQL LIKE pattern, or * glob). Use --json to output as JSON. Use --fields to choose JSON detail: minimal (default) or full. -Use --max-tokens to limit output size (approximate token count). -Use --indexed-only to skip auto-sync (read existing index only).`, - Args: cobra.MaximumNArgs(1), +Use --max-tokens to limit output size (approximate token count).`, + Args: func(cmd *cobra.Command, args []string) error { + return validateCommandBeforeStartup(cmd, args, cobra.MaximumNArgs(1), func() error { + _, _, err := validateAndParseSearchRequest(searchQuery(text, args), fields, contentType, after, before) + return err + }) + }, RunE: func(cmd *cobra.Command, args []string) error { - query := text - if query == "" && len(args) > 0 { - query = args[0] - } - if query == "" { - return fmt.Errorf("search query required (use --text or positional argument)") + query := searchQuery(text, args) + startup := startupResultFrom(cmd) + if startup.Config == nil { + return fmt.Errorf("startup configuration unavailable") } - return runSearch(stdout, stderr, query, - project, allProjects, jsonFormat, robotFormat, + return runSearch(cmd.Context(), stdout, stderr, startup.Config, query, project, allProjects, jsonFormat, robotFormat, source, sourcePath, after, before, role, limit, offset, contentType, tag, - fields, maxTokens, lexicalOnly, similarityThreshold, indexedOnly) + fields, maxTokens, lexicalOnly, similarityThreshold) }, } @@ -92,22 +92,23 @@ Use --indexed-only to skip auto-sync (read existing index only).`, cmd.Flags().BoolVar(&lexicalOnly, "lexical-only", false, "Use BM25 only, skip vector search") cmd.Flags().Float64Var(&similarityThreshold, "similarity-threshold", 0.3, "Minimum cosine similarity for vector results (0=no threshold)") cmd.Flags().StringVar(&text, "text", "", "Search text (v2 preferred grammar)") - cmd.Flags().BoolVar(&indexedOnly, "indexed-only", false, "Read existing index without auto-sync") return cmd } -func runSearch(stdout, stderr io.Writer, - query string, - project string, allProjects bool, jsonFormat, robotFormat bool, - source, sourcePath, after, before, role string, - limit, offset int, contentType, tag string, - fields string, maxTokens int, - lexicalOnly bool, similarityThreshold float64, indexedOnly bool) (retErr error) { +func searchQuery(text string, args []string) string { + if text == "" && len(args) > 0 { + return args[0] + } + return text +} - // Validate flag values before opening the database +func validateAndParseSearchRequest(query, fields, contentType, after, before string) (*time.Time, *time.Time, error) { + if query == "" { + return nil, nil, fmt.Errorf("search query required (use --text or positional argument)") + } if fields != "minimal" && fields != "full" { - return fmt.Errorf("invalid --fields value %q: must be minimal or full", fields) + return nil, nil, fmt.Errorf("invalid --fields value %q: must be minimal or full", fields) } validContentTypes := map[string]bool{ @@ -116,17 +117,42 @@ func runSearch(stdout, stderr io.Writer, "tool": true, "reasoning": true, } - if contentType != "" && !validContentTypes[contentType] { - return fmt.Errorf("invalid --content-type %q; must be one of: text, code, tool, reasoning", contentType) + return nil, nil, fmt.Errorf("invalid --content-type %q; must be one of: text, code, tool, reasoning", contentType) + } + + var afterTime, beforeTime *time.Time + if after != "" { + parsed, err := time.Parse("2006-01-02", after) + if err != nil { + return nil, nil, fmt.Errorf("parse --after date: %w", err) + } + afterTime = &parsed + } + if before != "" { + parsed, err := time.Parse("2006-01-02", before) + if err != nil { + return nil, nil, fmt.Errorf("parse --before date: %w", err) + } + beforeTime = &parsed } + return afterTime, beforeTime, nil +} + +func runSearch(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, + query string, + project string, allProjects bool, jsonFormat, robotFormat bool, + source, sourcePath, after, before, role string, + limit, offset int, contentType, tag string, + fields string, maxTokens int, + lexicalOnly bool, similarityThreshold float64) (retErr error) { - cfg, err := config.Load() + afterTime, beforeTime, err := validateAndParseSearchRequest(query, fields, contentType, after, before) if err != nil { - return fmt.Errorf("load config: %w", err) + return err } - db, diag, err := prepareIndex(context.Background(), cfg, indexDataRead, !indexedOnly) + db, diag, err := prepareIndex(ctx, cfg, indexDataRead) if diag != nil { return refuseIndex(stdout, stderr, *diag, jsonFormat, robotFormat) } @@ -140,23 +166,6 @@ func runSearch(stdout, stderr io.Writer, // Derive effective project from cwd if not explicitly set project = effectiveProject(project, allProjects) - // Parse dates - var afterTime, beforeTime *time.Time - if after != "" { - t, err := time.Parse("2006-01-02", after) - if err != nil { - return fmt.Errorf("parse --after date: %w", err) - } - afterTime = &t - } - if before != "" { - t, err := time.Parse("2006-01-02", before) - if err != nil { - return fmt.Errorf("parse --before date: %w", err) - } - beforeTime = &t - } - // Build search options opts := models.SearchOptions{ Project: project, @@ -267,21 +276,22 @@ func resultsToLines(results []models.SearchResult, format picokitoutput.Format) for i, result := range results { if format == picokitoutput.FormatRobot { // Robot format: result_N_field=value + // String values are escaped to keep each field on a single deterministic line. lines = append(lines, - fmt.Sprintf("result_%d_source=%s", i, result.Source), - fmt.Sprintf("result_%d_role=%s", i, result.Role), - fmt.Sprintf("result_%d_filepath=%s", i, result.FilePath), - fmt.Sprintf("result_%d_content=%s", i, result.Content), + fmt.Sprintf("result_%d_source=%s", i, escapeRobotValue(result.Source)), + fmt.Sprintf("result_%d_role=%s", i, escapeRobotValue(result.Role)), + fmt.Sprintf("result_%d_filepath=%s", i, escapeRobotValue(result.FilePath)), + fmt.Sprintf("result_%d_content=%s", i, escapeRobotValue(result.Content)), ) if result.SessionID != "" { - lines = append(lines, fmt.Sprintf("result_%d_session_id=%s", i, result.SessionID)) + lines = append(lines, fmt.Sprintf("result_%d_session_id=%s", i, escapeRobotValue(result.SessionID))) } if result.ProjectPath != "" { - lines = append(lines, fmt.Sprintf("result_%d_project=%s", i, result.ProjectPath)) + lines = append(lines, fmt.Sprintf("result_%d_project=%s", i, escapeRobotValue(result.ProjectPath))) } lines = append(lines, fmt.Sprintf("result_%d_score=%.2f", i, result.Score)) if len(result.Tags) > 0 { - lines = append(lines, fmt.Sprintf("result_%d_tags=%s", i, strings.Join(result.Tags, ","))) + lines = append(lines, fmt.Sprintf("result_%d_tags=%s", i, escapeRobotValue(strings.Join(result.Tags, ",")))) } lines = append(lines, fmt.Sprintf("result_%d_rank=%d", i, result.Rank)) } else { @@ -307,3 +317,10 @@ func resultsToLines(results []models.SearchResult, format picokitoutput.Format) return lines } + +func escapeRobotValue(value string) string { + escaped := strings.ReplaceAll(value, `\`, `\\`) + escaped = strings.ReplaceAll(escaped, "\r", `\r`) + escaped = strings.ReplaceAll(escaped, "\n", `\n`) + return escaped +} diff --git a/cmd/backscroll/search_robot_test.go b/cmd/backscroll/search_robot_test.go index d0164a3..042925d 100644 --- a/cmd/backscroll/search_robot_test.go +++ b/cmd/backscroll/search_robot_test.go @@ -31,22 +31,18 @@ func TestSearchRobotFormatUnwrapped(t *testing.T) { lines := strings.Split(strings.TrimSpace(out), "\n") - // Check each robot format line for double-wrapping + // Check each robot format line for double-wrapping and strict line contract. + correctPattern := regexp.MustCompile(`^result_\d+_\w+=.*$`) + bugPattern := regexp.MustCompile(`^result_\d+=result_\d+_`) for _, line := range lines { - if strings.HasPrefix(line, "result_") { - // Correct pattern: result_0_source=value or result_0_rank=0 - correctPattern := regexp.MustCompile(`^result_\d+_\w+=.+$`) - // Bug pattern: result_0=result_0_source=value (double-wrapped) - bugPattern := regexp.MustCompile(`^result_\d+=result_\d+_`) - - if !correctPattern.MatchString(line) && !bugPattern.MatchString(line) { - // Line might be in a different format (e.g., from text output) - continue - } - - if bugPattern.MatchString(line) { - t.Errorf("detected double-wrapped robot line (bug): %s", line) - } + if strings.TrimSpace(line) == "" { + continue + } + if !correctPattern.MatchString(line) { + t.Fatalf("non-result or malformed robot line: %q", line) + } + if bugPattern.MatchString(line) { + t.Errorf("detected double-wrapped robot line (bug): %s", line) } } } diff --git a/cmd/backscroll/search_test.go b/cmd/backscroll/search_test.go index bd6c551..b794506 100644 --- a/cmd/backscroll/search_test.go +++ b/cmd/backscroll/search_test.go @@ -209,6 +209,46 @@ func TestSearchRobotFormatStructure(t *testing.T) { } } +func TestSearchRobotFormatEscapesStringValuesToSingleLine(t *testing.T) { + results := []models.SearchResult{ + { + Source: "sess\\ion", + Role: "assist\rant", + Content: "line 1\nline 2\r\npath\\tail", + FilePath: "/tmp/file\nname.md", + Rank: 2, + Score: 0.42, + SessionID: "session\n789", + ProjectPath: "project\\root", + Tags: []string{"tag\\one", "tag\ntwo"}, + }, + } + + lines := resultsToLines(results, picokitoutput.FormatRobot) + allText := strings.Join(lines, "\n") + + for _, line := range lines { + if strings.Contains(line, "\n") || strings.Contains(line, "\r") { + t.Fatalf("robot line must be one-line escaped output, got %q", line) + } + } + + expected := []string{ + `result_0_source=sess\\ion`, + `result_0_role=assist\rant`, + `result_0_filepath=/tmp/file\nname.md`, + `result_0_content=line 1\nline 2\r\npath\\tail`, + `result_0_session_id=session\n789`, + `result_0_project=project\\root`, + `result_0_tags=tag\\one,tag\ntwo`, + } + for _, want := range expected { + if !strings.Contains(allText, want) { + t.Fatalf("robot escaped output missing %q in %q", want, allText) + } + } +} + func TestSearchWithJSONAndMaxTokens(t *testing.T) { _, cleanup := testEnv(t) defer cleanup() diff --git a/cmd/backscroll/skill_contract_test.go b/cmd/backscroll/skill_contract_test.go index c63f1c7..6951dfd 100644 --- a/cmd/backscroll/skill_contract_test.go +++ b/cmd/backscroll/skill_contract_test.go @@ -38,7 +38,7 @@ func TestBackscrollSkillContractAcceptsCurrentCLIForms(t *testing.T) { "backscroll --help", "backscroll search --help", "command -v backscroll >/dev/null", - "backscroll search \"needle\" --all-projects --source-path \"*uuid*\" --indexed-only --robot --fields full --max-tokens 4000", + "backscroll search \"needle\" --all-projects --source-path \"*uuid*\" --robot --fields full --max-tokens 4000", "backscroll list --all-projects --limit 10 --json", "backscroll patterns --kind corrections --pending --batch 50 --robot", "backscroll annotate --uuid u --kind correction --label false-positive", @@ -76,6 +76,192 @@ func TestBackscrollSkillContractRejectsUnknownFlags(t *testing.T) { ) } +func TestBackscrollSkillContractRequiresSearchQueryText(t *testing.T) { + root := buildRootCmd(io.Discard, io.Discard) + content := strings.Join([]string{ + `backscroll search --source-path "*/session.jsonl" --all-projects --json`, + `backscroll search --source-path "*/session.jsonl" --text --all-projects --json`, + `backscroll search --all-projects --limit 5`, + }, "\n") + + violations := validateSkillMarkdown(root, "synthetic-queryless-search.md", content) + assertSkillContractViolations(t, violations, + "synthetic-queryless-search.md:1: search invocation lacks query text (use --text or positional query)", + "synthetic-queryless-search.md:2: search invocation lacks query text (use --text or positional query)", + "synthetic-queryless-search.md:3: search invocation lacks query text (use --text or positional query)", + ) +} + +func TestBackscrollSkillContractAcceptsSearchQueryText(t *testing.T) { + root := buildRootCmd(io.Discard, io.Discard) + content := strings.Join([]string{ + `backscroll search "artifact literal" --source-path "*/session.jsonl" --all-projects --json`, + `backscroll search --text "$QUERY" --source-path "$SOURCE_PATH" --robot --fields full`, + `backscroll search --source-path "*/session.jsonl" --text "artifact literal" --limit 5`, + `backscroll search --text=2>out.txt --all-projects`, + `backscroll search "artifact literal">out.txt --all-projects`, + `backscroll search --help`, + }, "\n") + + violations := validateSkillMarkdown(root, "synthetic-search-with-query.md", content) + if len(violations) > 0 { + t.Fatalf("expected search invocations with query text to pass, got violations:\n%s", formatSkillContractViolations(violations)) + } +} + +func TestBackscrollSkillContractAcceptsInlineValueFlagsBeforePositionalQuery(t *testing.T) { + root := buildRootCmd(io.Discard, io.Discard) + content := strings.Join([]string{ + `backscroll search --source-path="*/session.jsonl" --all-projects "artifact literal"`, + `backscroll search --project=backscroll --limit=5 "migration plan" --json`, + }, "\n") + + violations := validateSkillMarkdown(root, "synthetic-inline-flags-with-query.md", content) + if len(violations) > 0 { + t.Fatalf("expected inline-value flags with positional query to pass, got violations:\n%s", formatSkillContractViolations(violations)) + } +} + +func TestBackscrollSkillContractAcceptsQuotedMetacharSearchQueryText(t *testing.T) { + root := buildRootCmd(io.Discard, io.Discard) + content := strings.Join([]string{ + `backscroll search ";" --all-projects`, + `backscroll search ">" --all-projects`, + `backscroll search "2>err" --all-projects`, + `backscroll search "\\" --all-projects`, + }, "\n") + + violations := validateSkillMarkdown(root, "synthetic-search-quoted-metachar-query.md", content) + if len(violations) > 0 { + t.Fatalf("expected quoted shell metachar query tokens to pass, got violations:\n%s", formatSkillContractViolations(violations)) + } +} + +func TestBackscrollSkillContractRejectsSearchNonQueryTokens(t *testing.T) { + root := buildRootCmd(io.Discard, io.Discard) + content := strings.Join([]string{ + `backscroll search --source-path "*/session.jsonl" ; echo no-query`, + `backscroll search --source-path "*/session.jsonl" > out.txt`, + `backscroll search --source-path "*/session.jsonl" >> out.txt`, + `backscroll search --source-path "*/session.jsonl" < in.txt`, + `backscroll search --source-path "*/session.jsonl" 2>err.txt`, + `backscroll search --source-path "*/session.jsonl" 2>&1`, + `backscroll search --source-path "*/session.jsonl" &`, + `backscroll search --source-path "*/session.jsonl" <<< marker`, + `backscroll search --source-path "*/session.jsonl" \`, + `backscroll search --source-path "*/session.jsonl" ''>out.txt`, + `backscroll search "" --all-projects`, + `backscroll search --text= --all-projects`, + `backscroll search --all-projects --limit 5`, + `backscroll search --source-path="*/session.jsonl" --all-projects`, + }, "\n") + + violations := validateSkillMarkdown(root, "synthetic-search-non-query-tokens.md", content) + assertSkillContractViolations(t, violations, + "synthetic-search-non-query-tokens.md:1: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:2: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:3: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:4: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:5: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:6: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:7: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:8: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:9: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:10: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:11: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:12: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:13: search invocation lacks query text (use --text or positional query)", + "synthetic-search-non-query-tokens.md:14: search invocation lacks query text (use --text or positional query)", + ) +} + +func TestBackscrollSkillContractReadNarrativeLiteralPassesButInvocationsFail(t *testing.T) { + root := buildRootCmd(io.Discard, io.Discard) + + narrative := "The literal `backscroll read` names the removed command in narrative prose." + violations := validateSkillMarkdown(root, "synthetic-read-narrative.md", narrative) + if len(violations) > 0 { + t.Fatalf("expected narrative inline literal to pass full validator, got violations:\n%s", formatSkillContractViolations(violations)) + } + + imperativeInline := strings.Join([]string{ + "Run `backscroll read` now.", + "Use `backscroll read` for this file.", + }, "\n") + violations = validateSkillMarkdown(root, "synthetic-read-inline-imperative.md", imperativeInline) + assertSkillContractViolations(t, violations, + "synthetic-read-inline-imperative.md:1: unknown backscroll command \"read\"", + "synthetic-read-inline-imperative.md:2: unknown backscroll command \"read\"", + ) + + descriptorImperative := strings.Join([]string{ + "Please run the literal `backscroll read` removed command.", + "You must use the removed command `backscroll read`.", + "Operators should execute the deprecated command `backscroll read`.", + "Try to invoke the literal `backscroll read` in the historical narrative.", + "Reference `backscroll read` when checking an old transcript.", + "You may mention `backscroll read` in migration notes.", + "Operators can cite `backscroll read` in examples.", + "The guide will include `backscroll read` as history.", + "Reviewers need `backscroll read` for this case.", + }, "\n") + violations = validateSkillMarkdown(root, "synthetic-read-descriptor-imperative.md", descriptorImperative) + assertSkillContractViolations(t, violations, + "synthetic-read-descriptor-imperative.md:1: unknown backscroll command \"read\"", + "synthetic-read-descriptor-imperative.md:2: unknown backscroll command \"read\"", + "synthetic-read-descriptor-imperative.md:3: unknown backscroll command \"read\"", + "synthetic-read-descriptor-imperative.md:4: unknown backscroll command \"read\"", + "synthetic-read-descriptor-imperative.md:5: unknown backscroll command \"read\"", + "synthetic-read-descriptor-imperative.md:6: unknown backscroll command \"read\"", + "synthetic-read-descriptor-imperative.md:7: unknown backscroll command \"read\"", + "synthetic-read-descriptor-imperative.md:8: unknown backscroll command \"read\"", + "synthetic-read-descriptor-imperative.md:9: unknown backscroll command \"read\"", + ) + + changedClause := strings.Join([]string{ + "The literal `backscroll read` names the removed command in narrative prose, but keep it in mind.", + "The literal `backscroll read` names the removed command in historical prose.", + "The literal `backscroll read` names a removed command in narrative prose.", + }, "\n") + violations = validateSkillMarkdown(root, "synthetic-read-changed-clause.md", changedClause) + assertSkillContractViolations(t, violations, + "synthetic-read-changed-clause.md:1: unknown backscroll command \"read\"", + "synthetic-read-changed-clause.md:2: unknown backscroll command \"read\"", + "synthetic-read-changed-clause.md:3: unknown backscroll command \"read\"", + ) + + multipleSpans := "The literal `backscroll read` names the removed command in narrative prose, while `backscroll read` remains removed." + violations = validateSkillMarkdown(root, "synthetic-read-multiple-spans.md", multipleSpans) + assertSkillContractViolations(t, violations, + "synthetic-read-multiple-spans.md:1: unknown backscroll command \"read\"", + ) + + content := strings.Join([]string{ + "backscroll read", + "backscroll read /tmp/session.jsonl", + "Run `backscroll read /tmp/session.jsonl` now.", + }, "\n") + violations = validateSkillMarkdown(root, "synthetic-read-invocations.md", content) + assertSkillContractViolations(t, violations, + "synthetic-read-invocations.md:1: unknown backscroll command \"read\"", + "synthetic-read-invocations.md:2: unknown backscroll command \"read\"", + "synthetic-read-invocations.md:3: unknown backscroll command \"read\"", + ) + + if !containsBackscrollReadInvocation(content) { + t.Fatal("expected actual backscroll read invocation with argument to be detected") + } + if containsBackscrollReadInvocation(narrative) { + t.Fatal("narrative literal backscroll read mention must not be treated as an invocation") + } + if !containsBackscrollReadInvocation(imperativeInline) { + t.Fatal("imperative inline backscroll read mention must be treated as a removed-command invocation") + } + if !containsBackscrollReadInvocation(descriptorImperative) { + t.Fatal("descriptor plus imperative backscroll read mention must be treated as a removed-command invocation") + } +} + func TestBackscrollSkillContractRejectsBareSubcommands(t *testing.T) { root := buildRootCmd(io.Discard, io.Discard) content := strings.Join([]string{ @@ -150,8 +336,8 @@ func TestBackscrollSkillContainsSearchDiscipline(t *testing.T) { "Two empty searches prove nothing", "Raw-file boundary", "--source-path", - "--indexed-only", - "backscroll validate --indexed-only", + "mandatory startup sync", + "backscroll validate", } for _, anchor := range anchors { if !strings.Contains(content, anchor) { @@ -159,11 +345,18 @@ func TestBackscrollSkillContainsSearchDiscipline(t *testing.T) { } } + if strings.Contains(content, "--indexed-only") { + t.Error("shipped skill must not document removed --indexed-only flag") + } + if containsBackscrollReadInvocation(content) { + t.Error("shipped skill must not invoke removed backscroll read command") + } + rawBoundaryAnchors := []string{ "cat", "jq", "Python", - "direct `backscroll read`", + "filesystem session hunting", "not a normal retrieval fallback", } for _, anchor := range rawBoundaryAnchors { @@ -192,6 +385,32 @@ func TestBackscrollContextModeCommandsMatchCLI(t *testing.T) { } } +func containsBackscrollReadInvocation(content string) bool { + for _, line := range strings.Split(strings.ReplaceAll(content, "\r\n", "\n"), "\n") { + if startsWithBackscrollRead(shellishFields(withoutInlineCodeSpans(line))) { + return true + } + for _, span := range inlineCodeSpans(line) { + if isNarrativeInlineLiteralBackscrollRead(line, span) { + continue + } + if startsWithBackscrollRead(shellishFields(span)) { + return true + } + } + } + return false +} + +func startsWithBackscrollRead(tokens []string) bool { + for i, token := range tokens { + if token == "backscroll" && i+1 < len(tokens) && tokens[i+1] == "read" && isInvocationStart(tokens, i) { + return true + } + } + return false +} + func assertSourceSessionOnlyOnSearch(t *testing.T, content string) { t.Helper() found := false @@ -282,6 +501,9 @@ func validateSkillMarkdown(root *cobra.Command, path, content string) []skillCon violations = append(violations, validateBackscrollInvocations(root, path, lineNumber, prose)...) violations = append(violations, validateBareSubcommand(path, lineNumber, prose, registeredSubcommands)...) for _, span := range inlineCodeSpans(line) { + if isNarrativeInlineLiteralBackscrollRead(line, span) { + continue + } violations = append(violations, validateBackscrollInvocations(root, path, lineNumber, span)...) violations = append(violations, validateBareSubcommand(path, lineNumber, span, registeredSubcommands)...) } @@ -299,18 +521,33 @@ func validateSkillMarkdown(root *cobra.Command, path, content string) []skillCon return uniqueSkillContractViolations(violations) } +const normalizedHistoricalBackscrollReadNarrative = "the literal `backscroll read` names the removed command in narrative prose." + +func isNarrativeInlineLiteralBackscrollRead(line, span string) bool { + if strings.TrimSpace(span) != "backscroll read" { + return false + } + return normalizeHistoricalNarrativeLine(line) == normalizedHistoricalBackscrollReadNarrative +} + +func normalizeHistoricalNarrativeLine(line string) string { + return strings.Join(strings.Fields(strings.ToLower(strings.TrimSpace(line))), " ") +} + func validateBackscrollInvocations(root *cobra.Command, path string, lineNumber int, text string) []skillContractViolation { - tokens := shellishFields(text) + lexedTokens := shellishTokens(text) + tokens := shellishTokenValues(lexedTokens) var violations []skillContractViolation for i, token := range tokens { if token != "backscroll" || isURLToken(token) || isCommandVBackscroll(tokens, i) || !isInvocationStart(tokens, i) { continue } - args := invocationArgs(tokens[i+1:]) + argsTokens := invocationArgsTokens(lexedTokens[i+1:]) + args := shellishTokenValues(argsTokens) cmd := root cmdName := root.Name() - flagArgs := args + flagArgsTokens := argsTokens if len(args) > 0 && !strings.HasPrefix(args[0], "-") { child := findSubcommand(root, args[0]) if child == nil { @@ -328,9 +565,10 @@ func validateBackscrollInvocations(root *cobra.Command, path string, lineNumber } cmd = child cmdName = root.Name() + " " + child.Name() - flagArgs = args[1:] + flagArgsTokens = argsTokens[1:] } + flagArgs := shellishTokenValues(flagArgsTokens) for _, flagName := range longFlags(flagArgs) { if isSupportedFlag(cmd, flagName) { continue @@ -341,10 +579,122 @@ func validateBackscrollInvocations(root *cobra.Command, path string, lineNumber message: fmt.Sprintf("unknown flag --%s for %s", flagName, cmdName), }) } + if cmd.Name() == "search" && !searchInvocationHasQuery(cmd, flagArgsTokens) { + violations = append(violations, skillContractViolation{ + path: path, + line: lineNumber, + message: "search invocation lacks query text (use --text or positional query)", + }) + } } return violations } +func searchInvocationHasQuery(cmd *cobra.Command, tokens []shellishToken) bool { + if hasUnsupportedLineContinuation(tokens) { + return false + } + + for i := 0; i < len(tokens); i++ { + token := tokens[i] + tokenValue := token.value + if token.isOperator() && (isShellTerminator(tokenValue) || isRedirectOperator(tokenValue)) { + return false + } + if tokenValue == "--help" || tokenValue == "-h" { + return true + } + if tokenValue == "--" { + return firstQueryToken(tokens[i+1:]) != "" + } + if strings.HasPrefix(tokenValue, "--") { + name, value, hasValue := splitLongFlag(tokenValue) + if name == "text" && hasValue { + return isValidQueryToken(shellishToken{value: value, quoted: token.quoted}) + } + if hasValue { + continue + } + if flagConsumesNext(cmd, name) { + if i+1 >= len(tokens) { + if name == "text" { + return false + } + continue + } + next := tokens[i+1] + if name == "text" { + if strings.HasPrefix(next.value, "--") && !next.quoted { + return false + } + return isValidQueryToken(next) + } + if next.isOperator() && (isShellTerminator(next.value) || isRedirectOperator(next.value)) { + return false + } + i++ + } + continue + } + if strings.HasPrefix(tokenValue, "-") { + continue + } + return isValidQueryToken(token) + } + return false +} + +func hasUnsupportedLineContinuation(tokens []shellishToken) bool { + for _, token := range tokens { + if token.isOperator() && token.value == `\` { + return true + } + } + return false +} + +func firstQueryToken(tokens []shellishToken) string { + for _, token := range tokens { + if token.isOperator() && (isShellTerminator(token.value) || isRedirectOperator(token.value) || token.value == `\`) { + return "" + } + if isValidQueryToken(token) { + return token.value + } + } + return "" +} + +func isValidQueryToken(token shellishToken) bool { + trimmed := strings.TrimSpace(token.value) + if trimmed == "" { + return false + } + if token.isOperator() && (isShellTerminator(trimmed) || isRedirectOperator(trimmed) || trimmed == `\`) { + return false + } + return true +} + +func splitLongFlag(token string) (name, value string, hasValue bool) { + name = strings.TrimPrefix(token, "--") + if before, after, ok := strings.Cut(name, "="); ok { + return before, after, true + } + return name, "", false +} + +func flagConsumesNext(cmd *cobra.Command, name string) bool { + flag := cmd.Flags().Lookup(name) + if flag == nil { + flag = cmd.InheritedFlags().Lookup(name) + } + if flag == nil { + flag = cmd.PersistentFlags().Lookup(name) + } + return flag != nil && flag.NoOptDefVal == "" +} + func validateBareSubcommand(path string, lineNumber int, text string, registeredSubcommands map[string]struct{}) []skillContractViolation { trimmed := strings.TrimSpace(text) if trimmed == "" || strings.HasPrefix(trimmed, "backscroll ") || strings.HasPrefix(trimmed, "command -v backscroll") { @@ -384,24 +734,78 @@ func findSubcommand(root *cobra.Command, name string) *cobra.Command { return nil } -func invocationArgs(tokens []string) []string { +type shellishTokenKind int + +const ( + shellishWordToken shellishTokenKind = iota + shellishOperatorToken +) + +type shellishToken struct { + value string + quoted bool + kind shellishTokenKind +} + +func (token shellishToken) isOperator() bool { + return token.kind == shellishOperatorToken +} + +func invocationArgsTokens(tokens []shellishToken) []shellishToken { for i, token := range tokens { - if isShellTerminator(token) { + if token.isOperator() && isShellTerminator(token.value) { return tokens[:i] } } return tokens } +func shellishTokenValues(tokens []shellishToken) []string { + values := make([]string, 0, len(tokens)) + for _, token := range tokens { + values = append(values, token.value) + } + return values +} + func isShellTerminator(token string) bool { switch token { - case "|", "||", "&&", ";", "#": + case "|", "||", "&", "&&", ";", "#": return true default: return strings.HasPrefix(token, "#") } } +func isRedirectOperator(token string) bool { + trimmed := strings.TrimSpace(token) + if trimmed == "" { + return false + } + if strings.HasPrefix(trimmed, "<") && strings.HasSuffix(trimmed, ">") && len(trimmed) > 2 { + return false + } + + rest := trimmed + for len(rest) > 0 && rest[0] >= '0' && rest[0] <= '9' { + rest = rest[1:] + } + if rest == "" { + return false + } + + if strings.HasPrefix(rest, "&>>") || strings.HasPrefix(rest, "&>") { + return true + } + if strings.HasPrefix(rest, ">>") || strings.HasPrefix(rest, ">") || strings.HasPrefix(rest, "<<") || strings.HasPrefix(rest, "<<<") || strings.HasPrefix(rest, "<") || strings.HasPrefix(rest, "<>") { + return true + } + if strings.HasPrefix(rest, ">&") || strings.HasPrefix(rest, "<&") { + return true + } + return false +} + func isInvocationStart(tokens []string, index int) bool { if index == 0 { return true @@ -514,35 +918,157 @@ func inlineCodeSpans(line string) []string { } func shellishFields(text string) []string { - var fields []string + tokens := shellishTokens(text) + fields := make([]string, 0, len(tokens)) + for _, token := range tokens { + fields = append(fields, token.value) + } + return fields +} + +func shellishTokens(text string) []shellishToken { + var tokens []shellishToken var current strings.Builder var quote rune - for _, r := range text { + wordStarted := false + tokenQuoted := false + + flushWord := func() { + if !wordStarted { + return + } + value := current.String() + if !tokenQuoted { + value = cleanUnquotedMarkdownToken(value) + } + tokens = append(tokens, shellishToken{ + value: value, + quoted: tokenQuoted, + kind: shellishWordToken, + }) + current.Reset() + wordStarted = false + tokenQuoted = false + } + appendOperator := func(value string) { + flushWord() + tokens = append(tokens, shellishToken{ + value: value, + kind: shellishOperatorToken, + }) + } + + runes := []rune(text) + for i := 0; i < len(runes); i++ { + r := runes[i] switch { case quote != 0: if r == quote { quote = 0 continue } + wordStarted = true current.WriteRune(r) case r == '\'' || r == '"': quote = r + wordStarted = true + tokenQuoted = true case r == '\t' || r == ' ': - if current.Len() > 0 { - fields = append(fields, cleanShellToken(current.String())) - current.Reset() + flushWord() + case r == '\\': + appendOperator(`\`) + case isDigitRune(r) && wordStarted: + current.WriteRune(r) + case r == ';' || r == '|' || r == '&' || r == '#' || r == '<' || r == '>' || isDigitRune(r): + if operator, width := scanShellOperator(runes, i); width > 0 { + appendOperator(operator) + i += width - 1 + continue } + wordStarted = true + current.WriteRune(r) default: + wordStarted = true current.WriteRune(r) } } - if current.Len() > 0 { - fields = append(fields, cleanShellToken(current.String())) + flushWord() + return tokens +} + +func scanShellOperator(runes []rune, index int) (string, int) { + if index >= len(runes) { + return "", 0 } - return fields + if width := scanControlOperator(runes, index); width > 0 { + return string(runes[index : index+width]), width + } + + start := index + for index < len(runes) && isDigitRune(runes[index]) { + index++ + } + redirect, width := scanRedirectOperator(runes, index) + if width == 0 { + return "", 0 + } + operatorEnd := index + width + if redirect == ">&" || redirect == "<&" { + for operatorEnd < len(runes) && isDigitRune(runes[operatorEnd]) { + operatorEnd++ + } + } + return string(runes[start:operatorEnd]), operatorEnd - start +} + +func scanControlOperator(runes []rune, index int) int { + switch runes[index] { + case ';', '#': + return 1 + case '|': + if hasRunePrefix(runes, index, "||") { + return 2 + } + return 1 + case '&': + if hasRunePrefix(runes, index, "&&") { + return 2 + } + if hasRunePrefix(runes, index, "&>") || hasRunePrefix(runes, index, "&>>") { + return 0 + } + return 1 + } + return 0 +} + +func scanRedirectOperator(runes []rune, index int) (string, int) { + for _, operator := range []string{"<<<", "&>>", ">>", "<<", "<>", ">&", "<&", "&>", ">", "<"} { + if hasRunePrefix(runes, index, operator) { + return operator, len([]rune(operator)) + } + } + return "", 0 +} + +func hasRunePrefix(runes []rune, index int, prefix string) bool { + prefixRunes := []rune(prefix) + if index+len(prefixRunes) > len(runes) { + return false + } + for i, r := range prefixRunes { + if runes[index+i] != r { + return false + } + } + return true +} + +func isDigitRune(r rune) bool { + return r >= '0' && r <= '9' } -func cleanShellToken(token string) string { +func cleanUnquotedMarkdownToken(token string) string { return strings.Trim(token, "`.,;:()[]{}") } diff --git a/cmd/backscroll/startup_policy.go b/cmd/backscroll/startup_policy.go new file mode 100644 index 0000000..c393658 --- /dev/null +++ b/cmd/backscroll/startup_policy.go @@ -0,0 +1,235 @@ +package main + +import ( + "context" + "errors" + "fmt" + "io" + "strings" + + "github.com/pablontiv/backscroll/internal/compat" + "github.com/pablontiv/backscroll/internal/config" + "github.com/pablontiv/backscroll/internal/input_config" + "github.com/spf13/cobra" +) + +type startupPolicyFunc func(context.Context, io.Writer) startupResult + +type startupStage string + +const ( + startupStageUnknown startupStage = "unknown" + startupStageInputDir startupStage = "input_dir" + startupStageLegacySource startupStage = "legacy_source" + startupStageConfigLoad startupStage = "config_load" + startupStageActiveManifest startupStage = "active_manifest" + startupStageIndexPrepare startupStage = "index_prepare" + startupStageStartupSync startupStage = "startup_sync" +) + +type startupFailure struct { + Stage startupStage + Cause error + Diagnostic compat.Diagnostic + Recoverable bool +} + +func (f *startupFailure) Error() string { + if f == nil { + return "startup failure" + } + stage := f.Stage + if stage == "" { + stage = startupStageUnknown + } + var rendered []string + if f.Diagnostic.Code != "" || strings.TrimSpace(f.Diagnostic.Summary) != "" { + rendered = append(rendered, fmt.Sprintf("diagnostic %s: %s", f.Diagnostic.Code, strings.TrimSpace(f.Diagnostic.Summary))) + } + if len(f.Diagnostic.Continuation) > 0 { + rendered = append(rendered, fmt.Sprintf("continuation: %s", strings.Join(f.Diagnostic.Continuation, " "))) + } + if f.Cause != nil { + rendered = append(rendered, f.Cause.Error()) + } + if len(rendered) == 0 { + rendered = append(rendered, "startup failed") + } + return fmt.Sprintf("startup %s failed: %s", stage, strings.Join(rendered, ": ")) +} + +func (f *startupFailure) Unwrap() error { + if f == nil { + return nil + } + return f.Cause +} + +func (f *startupFailure) renderedDiagnostic() compat.Diagnostic { + d := f.Diagnostic + if !f.Recoverable { + d.Continuation = nil + } + return d +} + +type startupResult struct { + Config *config.Config + Failure *startupFailure +} + +func (r startupResult) startupFailure() *startupFailure { + return r.Failure +} + +func optionalStartupFailureError(f *startupFailure) error { + if f == nil { + return nil + } + return f +} + +type startupContextKey struct{} + +var startupSync = maybeAutoSync + +func startupResultFrom(cmd *cobra.Command) startupResult { + result, _ := cmd.Context().Value(startupContextKey{}).(startupResult) + return result +} + +func defaultStartupPolicy(ctx context.Context, progress io.Writer) startupResult { + inputsDir, err := input_config.InputsDir() + if err != nil { + cause := fmt.Errorf("resolve inputs directory: %w", err) + return startupResult{Failure: &startupFailure{Stage: startupStageInputDir, Cause: cause, Diagnostic: compat.Diagnostic{Code: compat.CodeMigrationFailed, Summary: cause.Error()}}} + } + if err := config.ValidateNoLegacySources(inputsDir); err != nil { + stage := startupStageConfigLoad + var legacyErr *config.LegacySourcesError + if errors.As(err, &legacyErr) { + stage = startupStageLegacySource + } + return startupResult{Failure: &startupFailure{Stage: stage, Cause: err, Diagnostic: compat.Diagnostic{Code: compat.CodeMigrationFailed, Summary: err.Error()}}} + } + cfg, err := config.Load() + if err != nil { + cause := fmt.Errorf("load config: %w", err) + return startupResult{Failure: &startupFailure{Stage: startupStageConfigLoad, Cause: cause, Diagnostic: compat.Diagnostic{Code: compat.CodeMigrationFailed, Summary: cause.Error()}}} + } + if _, _, err := input_config.ActiveInputs(cfg.SessionDirs); err != nil { + cause := fmt.Errorf("validate active inputs: %w", err) + return startupResult{Config: cfg, Failure: &startupFailure{Stage: startupStageActiveManifest, Cause: cause, Diagnostic: compat.Diagnostic{Code: compat.CodeMigrationFailed, Summary: cause.Error()}}} + } + db, diag, err := prepareIndex(ctx, cfg, indexMutation) + if db != nil { + err = closeIndexDB(db, err) + } + if diag != nil || err != nil { + d := compat.Diagnostic{Code: compat.CodeMigrationFailed, Summary: "prepare index failed"} + if diag != nil { + d = *diag + } else if err != nil { + d.Summary = fmt.Sprintf("prepare index failed: %v", err) + } + return startupResult{Config: cfg, Failure: &startupFailure{Stage: startupStageIndexPrepare, Cause: err, Diagnostic: d, Recoverable: true}} + } + if err := startupSync(cfg, progress); err != nil { + activePath, _ := resolveActiveIndexPath(cfg.DatabasePath) + d := continuationFor(compat.Diagnostic{Code: compat.CodeIndexStale, Summary: fmt.Sprintf("index sync failed: %v", err)}, activePath) + return startupResult{Config: cfg, Failure: &startupFailure{Stage: startupStageStartupSync, Cause: err, Diagnostic: d, Recoverable: true}} + } + return startupResult{Config: cfg} +} + +func validateCommandBeforeStartup(cmd *cobra.Command, args []string, positional cobra.PositionalArgs, semantic func() error) error { + if positional != nil { + if err := positional(cmd, args); err != nil { + return err + } + } + if err := validateRequiredFlagsAndGroups(cmd); err != nil { + return err + } + if semantic != nil { + return semantic() + } + return nil +} + +func validateRequiredFlagsAndGroups(cmd *cobra.Command) error { + if err := cmd.ValidateRequiredFlags(); err != nil { + return err + } + return cmd.ValidateFlagGroups() +} + +func buildRootCmd(stdout, stderr io.Writer) *cobra.Command { + return buildRootCmdWithStartup(stdout, stderr, defaultStartupPolicy) +} + +func buildRootCmdWithStartup(stdout, stderr io.Writer, policy startupPolicyFunc) *cobra.Command { + root := &cobra.Command{ + Use: "backscroll", + Short: "A permanent, searchable record of your coding-agent sessions", + SilenceErrors: true, + Long: `Backscroll turns your coding-agent sessions into a permanent, searchable +record of what happened. It indexes Claude Code, Pi and OpenCode sessions into +SQLite and keeps them after the session files expire. + +Prose and tool activity are indexed separately — a Porter-stemmed FTS5 index for +conversation, a trigram index for commands, paths and errors — and an unfiltered +query merges both by rank position (RRF).`, + Version: version, + PersistentPreRunE: func(cmd *cobra.Command, args []string) error { + if err := validateRequiredFlagsAndGroups(cmd); err != nil { + return err + } + result := policy(cmd.Context(), startupProgressWriter(cmd, stderr)) + cmd.SetContext(context.WithValue(cmd.Context(), startupContextKey{}, result)) + failure := result.startupFailure() + if failure == nil { + return nil + } + if cmd.Name() == "recover" && failure.Recoverable { + return nil + } + if failure.Diagnostic.Code != "" || strings.TrimSpace(failure.Diagnostic.Summary) != "" { + return refuseIndexWithCause(stdout, stderr, failure.renderedDiagnostic(), failure, commandBoolFlag(cmd, "json"), commandBoolFlag(cmd, "robot")) + } + return failure + }, + } + root.SetOut(stdout) + root.SetErr(stderr) + + root.AddCommand( + newSearchCmd(stdout, stderr), + newListCmd(stdout, stderr), + newPatternsCmd(stdout, stderr), + newRebuildCmd(stdout, stderr), + newPurgeCmd(stdout, stderr), + newValidateCmd(stdout, stderr), + newStatusCmd(stdout, stderr), + newConfigCmd(stdout, stderr), + newAnnotateCmd(stdout, stderr), + newRecoverCmd(stdout, stderr), + ) + + return root +} + +func startupProgressWriter(cmd *cobra.Command, stderr io.Writer) io.Writer { + if commandBoolFlag(cmd, "json") || commandBoolFlag(cmd, "robot") { + return io.Discard + } + return stderr +} + +func commandBoolFlag(cmd *cobra.Command, name string) bool { + flag := cmd.Flag(name) + if flag == nil { + return false + } + return flag.Value.String() == "true" +} diff --git a/cmd/backscroll/startup_policy_test.go b/cmd/backscroll/startup_policy_test.go new file mode 100644 index 0000000..247b484 --- /dev/null +++ b/cmd/backscroll/startup_policy_test.go @@ -0,0 +1,827 @@ +package main + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "io" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "github.com/pablontiv/backscroll/internal/compat" + "github.com/pablontiv/backscroll/internal/config" + "github.com/pablontiv/backscroll/internal/recovery" + "github.com/spf13/cobra" +) + +func TestInvalidOperationalCommandsSkipStartup(t *testing.T) { + testCases := []struct { + name string + argv []string + }{ + {name: "search missing query", argv: []string{"search"}}, + {name: "search invalid fields", argv: []string{"search", "needle", "--fields", "invalid"}}, + {name: "search invalid content type", argv: []string{"search", "needle", "--content-type", "invalid"}}, + {name: "search invalid after", argv: []string{"search", "needle", "--after", "not-a-date"}}, + {name: "search invalid before", argv: []string{"search", "needle", "--before", "not-a-date"}}, + {name: "list positional argument", argv: []string{"list", "unexpected"}}, + {name: "patterns positional argument", argv: []string{"patterns", "unexpected", "--kind", "commands"}}, + {name: "patterns invalid kind", argv: []string{"patterns", "--kind", "invalid"}}, + {name: "patterns invalid trend", argv: []string{"patterns", "--kind", "templates", "--trend"}}, + {name: "patterns conflicting project scope", argv: []string{"patterns", "--kind", "commands", "--project", "p", "--all-projects"}}, + {name: "patterns negative limit", argv: []string{"patterns", "--kind", "commands", "--limit", "-1"}}, + {name: "annotate positional argument", argv: []string{"annotate", "unexpected", "--uuid", "u", "--kind", "correction", "--label", "x"}}, + {name: "annotate missing label", argv: []string{"annotate", "--uuid", "u", "--kind", "correction"}}, + {name: "annotate missing identity", argv: []string{"annotate", "--kind", "correction", "--label", "x"}}, + {name: "purge positional argument", argv: []string{"purge", "unexpected", "--before", "2030-01-01"}}, + {name: "purge missing before", argv: []string{"purge"}}, + {name: "purge invalid before", argv: []string{"purge", "--before", "not-a-date"}}, + {name: "recover missing from", argv: []string{"recover"}}, + {name: "recover empty from", argv: []string{"recover", "--from", ""}}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + startupCalls := 0 + root := buildRootCmdWithStartup(io.Discard, io.Discard, func(context.Context, io.Writer) startupResult { + startupCalls++ + return startupResult{Config: &config.Config{DatabasePath: filepath.Join(t.TempDir(), "index.db")}} + }) + root.SetArgs(tc.argv) + + if err := root.Execute(); err == nil { + t.Fatalf("execute %v succeeded, want validation error", tc.argv) + } + if startupCalls != 0 { + t.Fatalf("startup calls=%d, want 0 for invalid invocation %v", startupCalls, tc.argv) + } + }) + } +} + +func TestInvalidSearchDoesNotCreateDatabase(t *testing.T) { + dir := t.TempDir() + homeDir := filepath.Join(dir, "home") + configDir := filepath.Join(dir, "config") + sessionDir := filepath.Join(dir, "sessions") + for _, path := range []string{homeDir, configDir, sessionDir} { + if err := os.MkdirAll(path, 0o755); err != nil { + t.Fatalf("mkdir %s: %v", path, err) + } + } + dbPath := filepath.Join(dir, "index.db") + t.Setenv("HOME", homeDir) + t.Setenv("BACKSCROLL_CONFIG_DIR", configDir) + t.Setenv("BACKSCROLL_DATABASE_PATH", dbPath) + t.Setenv("BACKSCROLL_SESSION_DIRS", sessionDir) + + root := buildRootCmd(io.Discard, io.Discard) + root.SetArgs([]string{"search"}) + if err := root.Execute(); err == nil { + t.Fatal("search without a query succeeded, want validation error") + } + if _, err := os.Stat(dbPath); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("database stat error=%v, want not-exist after invalid invocation", err) + } +} + +func TestEveryOperationalCommandRunsStartupBeforeHandler(t *testing.T) { + commands := [][]string{ + {"search", "needle"}, {"list"}, {"patterns", "--kind", "commands"}, + {"annotate", "--uuid", "u", "--kind", "correction", "--label", "x"}, + {"purge", "--before", "2030-01-01"}, + {"purge", "--before", "2030-01-01T12:30:00Z"}, + {"rebuild"}, {"status"}, + {"validate"}, {"config"}, {"recover", "--from", "missing.db", "--dry-run"}, + } + for _, argv := range commands { + t.Run(strings.Join(argv, "_"), func(t *testing.T) { + calls := 0 + markerCalls := 0 + events := []string{} + policy := func(context.Context, io.Writer) startupResult { + calls++ + events = append(events, "startup") + return startupResult{Config: &config.Config{DatabasePath: filepath.Join(t.TempDir(), "index.db")}} + } + root := buildRootCmdWithStartup(io.Discard, io.Discard, policy) + replaceRootCommandRunE(t, root, argv[0], func(cmd *cobra.Command, args []string) error { + markerCalls++ + events = append(events, "handler") + return nil + }) + root.SetArgs(argv) + if err := root.Execute(); err != nil { + t.Fatalf("execute %v: %v", argv, err) + } + if calls != 1 { + t.Fatalf("startup calls=%d, want 1", calls) + } + if markerCalls != 1 { + t.Fatalf("handler calls=%d, want 1", markerCalls) + } + if want := []string{"startup", "handler"}; !reflect.DeepEqual(events, want) { + t.Fatalf("events=%v, want %v", events, want) + } + }) + } +} + +func TestFailedStartupAllowsOnlyRecoverWithInjectedPolicy(t *testing.T) { + testCases := []struct { + name string + result startupResult + }{ + { + name: "recoverable_sync_error", + result: startupResult{ + Config: &config.Config{DatabasePath: "/tmp/error-only.db"}, + Failure: &startupFailure{ + Stage: startupStageStartupSync, + Cause: errors.New("synthetic startup error"), + Diagnostic: compat.Diagnostic{Code: compat.CodeIndexStale, Summary: "synthetic startup error", Continuation: []string{"recover", "--from", "/tmp/error-only.db", "--dry-run"}}, + Recoverable: true, + }, + }, + }, + { + name: "recoverable_diagnostic_and_error", + result: startupResult{ + Config: &config.Config{DatabasePath: "/tmp/diagnostic.db"}, + Failure: &startupFailure{ + Stage: startupStageIndexPrepare, + Diagnostic: compat.Diagnostic{ + Code: compat.CodeIndexStale, + Summary: "synthetic startup diagnostic", + Continuation: []string{"recover", "--from", "/tmp/diagnostic.db", "--dry-run"}, + }, + Cause: errors.New("synthetic startup diagnostic error"), + Recoverable: true, + }, + }, + }, + } + + blockedCommands := []struct { + name string + argv []string + }{ + {name: "search", argv: []string{"search", "needle"}}, + {name: "list", argv: []string{"list"}}, + {name: "config", argv: []string{"config"}}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + policyCalls := 0 + policy := func(context.Context, io.Writer) startupResult { + policyCalls++ + return tc.result + } + + var recoverOut, recoverErr bytes.Buffer + recoverRoot := buildRootCmdWithStartup(&recoverOut, &recoverErr, policy) + recoverReached := false + replaceRootCommandRunE(t, recoverRoot, "recover", func(cmd *cobra.Command, args []string) error { + recoverReached = true + assertStartupResultInContext(t, startupResultFrom(cmd), tc.result) + _, _ = io.WriteString(cmd.OutOrStdout(), "recover-marker\n") + return nil + }) + recoverRoot.SetArgs([]string{"recover", "--from", "missing.db", "--dry-run"}) + if err := recoverRoot.Execute(); err != nil { + t.Fatalf("recover should proceed on startup failure: %v", err) + } + if !recoverReached { + t.Fatal("recover marker was not reached") + } + if !strings.Contains(recoverOut.String(), "recover-marker") { + t.Fatalf("recover marker output missing: stdout=%q stderr=%q", recoverOut.String(), recoverErr.String()) + } + if policyCalls != 1 { + t.Fatalf("recover startup calls=%d, want 1", policyCalls) + } + + for _, blocked := range blockedCommands { + t.Run("blocks_"+blocked.name, func(t *testing.T) { + var blockedOut, blockedErr bytes.Buffer + blockedRoot := buildRootCmdWithStartup(&blockedOut, &blockedErr, policy) + blockedReached := false + replaceRootCommandRunE(t, blockedRoot, blocked.name, func(cmd *cobra.Command, args []string) error { + blockedReached = true + _, _ = io.WriteString(cmd.OutOrStdout(), "blocked-marker\n") + return nil + }) + blockedRoot.SetArgs(blocked.argv) + err := blockedRoot.Execute() + if err == nil { + t.Fatalf("%s unexpectedly succeeded on startup failure; stdout=%q stderr=%q", blocked.name, blockedOut.String(), blockedErr.String()) + } + if blockedReached { + t.Fatalf("%s marker should not run; stdout=%q stderr=%q", blocked.name, blockedOut.String(), blockedErr.String()) + } + combined := blockedOut.String() + blockedErr.String() + if strings.Contains(combined, "blocked-marker") { + t.Fatalf("blocked command emitted marker output: stdout=%q stderr=%q", blockedOut.String(), blockedErr.String()) + } + if failure := tc.result.startupFailure(); failure != nil && failure.Diagnostic.Code != "" && !strings.Contains(blockedErr.String(), "diagnostic:") { + t.Fatalf("expected diagnostic output for blocked command; stdout=%q stderr=%q", blockedOut.String(), blockedErr.String()) + } + }) + } + if policyCalls != 1+len(blockedCommands) { + t.Fatalf("total startup calls=%d, want %d", policyCalls, 1+len(blockedCommands)) + } + }) + } +} + +func TestRecoverAloneContinuesAfterStartupFailure(t *testing.T) { + startupErr := errors.New("injected startup failure") + recoveryErr := errors.New("injected recovery failure") + called := false + root := buildRootCmdWithStartup(io.Discard, io.Discard, func(context.Context, io.Writer) startupResult { + dbPath := filepath.Join(t.TempDir(), "active.db") + return startupResult{Config: &config.Config{DatabasePath: dbPath}, Failure: &startupFailure{ + Stage: startupStageStartupSync, + Cause: startupErr, + Diagnostic: continuationFor(compat.Diagnostic{Code: compat.CodeIndexStale, Summary: startupErr.Error()}, dbPath), + Recoverable: true, + }} + }) + originalExecute := recoverExecute + recoverExecute = func(context.Context, recovery.Options) (recovery.Report, error) { + called = true + return recovery.Report{}, recoveryErr + } + t.Cleanup(func() { recoverExecute = originalExecute }) + root.SetArgs([]string{"recover", "--from", "stranded.db"}) + err := root.Execute() + if !called { + t.Fatal("recover handler did not continue after startup failure") + } + if !errors.Is(err, startupErr) { + t.Fatalf("error=%v does not preserve startup failure", err) + } + if !errors.Is(err, recoveryErr) { + t.Fatalf("error=%v does not preserve recovery failure", err) + } +} + +func TestStartupFailurePreventsHandlerOutput(t *testing.T) { + var stdout, stderr bytes.Buffer + policyErr := errors.New("injected startup failure") + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Failure: &startupFailure{Stage: startupStageConfigLoad, Cause: policyErr, Diagnostic: compat.Diagnostic{Code: compat.CodeMigrationFailed, Summary: "injected startup failure"}}} + }) + root.SetArgs([]string{"config", "--json"}) + err := root.Execute() + if !errors.Is(err, policyErr) { + t.Fatalf("error=%v, want injected startup failure", err) + } + if stdout.Len() == 0 { + t.Fatalf("startup failure did not emit machine diagnostic") + } + if strings.Contains(stdout.String(), "database") || strings.Contains(stdout.String(), "session_dirs") { + t.Fatalf("handler emitted config payload after startup failure: %q", stdout.String()) + } +} + +func TestRecoverBlocksNonrecoverableStartupFailures(t *testing.T) { + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + for _, tc := range []struct { + name string + stage startupStage + }{ + {name: "input_dir", stage: startupStageInputDir}, + {name: "legacy_source", stage: startupStageLegacySource}, + {name: "config_load", stage: startupStageConfigLoad}, + {name: "active_manifest", stage: startupStageActiveManifest}, + } { + t.Run(tc.name, func(t *testing.T) { + cause := errors.New("nonrecoverable " + tc.name) + called := false + originalExecute := recoverExecute + recoverExecute = func(context.Context, recovery.Options) (recovery.Report, error) { + called = true + return recovery.Report{}, nil + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + var stdout, stderr bytes.Buffer + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg, Failure: &startupFailure{ + Stage: tc.stage, + Cause: cause, + Diagnostic: compat.Diagnostic{Code: compat.CodeMigrationFailed, Summary: "blocked " + tc.name}, + Recoverable: false, + }} + }) + root.SetArgs([]string{"recover", "--from", "stranded.db", "--dry-run"}) + err := root.Execute() + if err == nil { + t.Fatalf("recover unexpectedly succeeded for %s; stdout=%q stderr=%q", tc.stage, stdout.String(), stderr.String()) + } + if !errors.Is(err, cause) { + t.Fatalf("error=%v does not preserve startup cause %v", err, cause) + } + if called { + t.Fatal("recoverExecute was called for nonrecoverable startup failure") + } + if stdout.Len() != 0 { + t.Fatalf("nonrecoverable recover emitted stdout: %q", stdout.String()) + } + }) + } +} + +func TestRecoverableStartupFailuresPermitControlledRecovery(t *testing.T) { + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + for _, tc := range []struct { + name string + stage startupStage + }{ + {name: "index_prepare", stage: startupStageIndexPrepare}, + {name: "startup_sync", stage: startupStageStartupSync}, + } { + t.Run(tc.name, func(t *testing.T) { + startupDiag := continuationFor(compat.Diagnostic{Code: compat.CodeIndexStale, Summary: "recoverable " + tc.name}, cfg.DatabasePath) + called := false + originalExecute := recoverExecute + recoverExecute = func(_ context.Context, opts recovery.Options) (recovery.Report, error) { + called = true + if !opts.DryRun || opts.FromPath != cfg.DatabasePath || opts.ActivePath != cfg.DatabasePath { + t.Fatalf("recovery options=%+v, want dry-run from/active %q", opts, cfg.DatabasePath) + } + return recovery.Report{ActivePath: opts.ActivePath}, nil + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + var stdout, stderr bytes.Buffer + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg, Failure: &startupFailure{Stage: tc.stage, Diagnostic: startupDiag, Recoverable: true}} + }) + root.SetArgs(startupDiag.Continuation) + if err := root.Execute(); err != nil { + t.Fatalf("recoverable %s did not permit dry-run recovery: %v\nstdout=%q stderr=%q", tc.stage, err, stdout.String(), stderr.String()) + } + if !called { + t.Fatal("recoverExecute was not called for recoverable startup failure") + } + if !strings.Contains(stdout.String(), "recovery dry run") { + t.Fatalf("dry-run report missing: stdout=%q stderr=%q", stdout.String(), stderr.String()) + } + }) + } +} + +func TestSuccessfulStartupRecoveryFailureOmitsTypedNilStartupFailure(t *testing.T) { + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + recoveryErr := errors.New("injected recovery failure") + + originalExecute := recoverExecute + recoverExecute = func(context.Context, recovery.Options) (recovery.Report, error) { + return recovery.Report{}, recoveryErr + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + root := buildRootCmdWithStartup(io.Discard, io.Discard, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg} + }) + root.SetArgs([]string{"recover", "--from", "stranded.db"}) + err := root.Execute() + if !errors.Is(err, recoveryErr) { + t.Fatalf("error=%v does not preserve recovery failure", err) + } + if strings.Contains(err.Error(), "startup failure") { + t.Fatalf("error=%v includes phantom startup failure", err) + } + var failure *startupFailure + if errors.As(err, &failure) { + t.Fatalf("error=%v unexpectedly matches startupFailure target %#v", err, failure) + } +} + +func TestSuccessfulStartupPostInstallSyncFailureOmitsTypedNilStartupFailure(t *testing.T) { + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + syncErr := errors.New("injected post-install sync failure") + + originalExecute := recoverExecute + recoverExecute = func(context.Context, recovery.Options) (recovery.Report, error) { + return recovery.Report{ActivePath: cfg.DatabasePath}, nil + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + originalPostInstallSync := recoverPostInstallSync + recoverPostInstallSync = func(*config.Config, io.Writer) error { return syncErr } + t.Cleanup(func() { recoverPostInstallSync = originalPostInstallSync }) + + var stdout bytes.Buffer + root := buildRootCmdWithStartup(&stdout, io.Discard, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg} + }) + root.SetArgs([]string{"recover", "--from", "stranded.db"}) + err := root.Execute() + if !errors.Is(err, syncErr) { + t.Fatalf("error=%v does not preserve post-install sync failure", err) + } + if strings.Contains(err.Error(), "startup failure") { + t.Fatalf("error=%v includes phantom startup failure", err) + } + var failure *startupFailure + if errors.As(err, &failure) { + t.Fatalf("error=%v unexpectedly matches startupFailure target %#v", err, failure) + } + if stdout.Len() != 0 { + t.Fatalf("report printed before failed post-install sync: %q", stdout.String()) + } +} + +func TestDiagnosticOnlyStartupFailurePlusRecoveryFailurePreservesBothCauses(t *testing.T) { + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + startupDiag := continuationFor(compat.Diagnostic{Code: compat.CodeUnsupportedLineage, Summary: "diagnostic-only startup"}, cfg.DatabasePath) + recoveryErr := errors.New("injected recovery failure") + + originalExecute := recoverExecute + recoverExecute = func(context.Context, recovery.Options) (recovery.Report, error) { + return recovery.Report{}, recoveryErr + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + root := buildRootCmdWithStartup(io.Discard, io.Discard, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg, Failure: &startupFailure{Stage: startupStageIndexPrepare, Diagnostic: startupDiag, Recoverable: true}} + }) + root.SetArgs([]string{"recover", "--from", "stranded.db"}) + err := root.Execute() + if !errors.Is(err, recoveryErr) { + t.Fatalf("error=%v does not preserve recovery failure", err) + } + var failure *startupFailure + if !errors.As(err, &failure) || failure == nil { + t.Fatalf("error=%v does not expose structural startup failure", err) + } + assertDiagnosticAggregateRenderedOnce(t, err, startupDiag, recoveryErr) +} + +func TestDiagnosticOnlyStartupFailurePlusPostInstallSyncFailurePreservesBothCauses(t *testing.T) { + cfg := &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")} + startupDiag := continuationFor(compat.Diagnostic{Code: compat.CodeUnsupportedLineage, Summary: "diagnostic-only startup"}, cfg.DatabasePath) + syncErr := errors.New("injected post-install sync failure") + + originalExecute := recoverExecute + recoverExecute = func(context.Context, recovery.Options) (recovery.Report, error) { + return recovery.Report{ActivePath: cfg.DatabasePath}, nil + } + t.Cleanup(func() { recoverExecute = originalExecute }) + + originalPostInstallSync := recoverPostInstallSync + recoverPostInstallSync = func(*config.Config, io.Writer) error { return syncErr } + t.Cleanup(func() { recoverPostInstallSync = originalPostInstallSync }) + + var stdout bytes.Buffer + root := buildRootCmdWithStartup(&stdout, io.Discard, func(context.Context, io.Writer) startupResult { + return startupResult{Config: cfg, Failure: &startupFailure{Stage: startupStageIndexPrepare, Diagnostic: startupDiag, Recoverable: true}} + }) + root.SetArgs([]string{"recover", "--from", "stranded.db"}) + err := root.Execute() + if !errors.Is(err, syncErr) { + t.Fatalf("error=%v does not preserve post-install sync failure", err) + } + var failure *startupFailure + if !errors.As(err, &failure) || failure == nil { + t.Fatalf("error=%v does not expose structural startup failure", err) + } + assertDiagnosticAggregateRenderedOnce(t, err, startupDiag, syncErr) + if stdout.Len() != 0 { + t.Fatalf("report printed before failed post-install sync: %q", stdout.String()) + } +} + +func assertDiagnosticAggregateRenderedOnce(t *testing.T, err error, diagnostic compat.Diagnostic, cause error) { + t.Helper() + if err == nil { + t.Fatal("error is nil") + } + text := err.Error() + for _, want := range []string{string(diagnostic.Code), diagnostic.Summary, strings.Join(diagnostic.Continuation, " "), cause.Error()} { + if got := strings.Count(text, want); got != 1 { + t.Fatalf("error=%q contains %q %d times, want exactly once", text, want, got) + } + } +} + +func TestStartupFailureNilAndFallbackRendering(t *testing.T) { + var nilFailure *startupFailure + if got := nilFailure.Error(); got != "startup failure" { + t.Fatalf("nil startup failure Error() = %q, want startup failure", got) + } + if got := nilFailure.Unwrap(); got != nil { + t.Fatalf("nil startup failure Unwrap() = %v, want nil", got) + } + + fallback := (&startupFailure{}).Error() + for _, want := range []string{string(startupStageUnknown), "startup failed"} { + if !strings.Contains(fallback, want) { + t.Fatalf("fallback startup failure rendering = %q, want %q", fallback, want) + } + } +} + +func TestStartupFailureMachineDiagnosticsAreStructuredAndUncontaminated(t *testing.T) { + for _, tc := range []struct { + name string + argv []string + mode string + }{ + {name: "config_json", argv: []string{"config", "--json"}, mode: "json"}, + {name: "manifest_robot", argv: []string{"search", "needle", "--robot"}, mode: "robot"}, + } { + t.Run(tc.name, func(t *testing.T) { + var stdout, stderr bytes.Buffer + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Failure: &startupFailure{ + Stage: startupStageActiveManifest, + Cause: errors.New("active manifest invalid"), + Diagnostic: compat.Diagnostic{Code: compat.CodeMigrationFailed, Summary: "active manifest invalid"}, + }} + }) + root.SetArgs(tc.argv) + err := root.Execute() + if err == nil { + t.Fatalf("%v unexpectedly succeeded", tc.argv) + } + if stderr.Len() != 0 { + t.Fatalf("machine diagnostic contaminated stderr: %q", stderr.String()) + } + if strings.Contains(stdout.String(), "source_path") || strings.Contains(stdout.String(), "database") || strings.Contains(stdout.String(), "Error:") { + t.Fatalf("machine diagnostic contaminated stdout: %q", stdout.String()) + } + switch tc.mode { + case "json": + var got struct { + Code string `json:"code"` + Summary string `json:"summary"` + Continuation []string `json:"continuation_argv"` + } + if err := json.Unmarshal(stdout.Bytes(), &got); err != nil { + t.Fatalf("invalid JSON diagnostic %q: %v", stdout.String(), err) + } + if got.Code == "" || got.Summary == "" || len(got.Continuation) != 0 { + t.Fatalf("JSON diagnostic=%+v, want code/summary and no continuation", got) + } + case "robot": + fields := robotDiagnosticFields(t, stdout.String()) + if fields["diagnostic_code"] == "" || fields["diagnostic_summary"] == "" { + t.Fatalf("robot diagnostic missing code/summary: %q", stdout.String()) + } + if _, ok := fields["diagnostic_continuation_argv"]; ok { + t.Fatalf("nonrecoverable robot diagnostic included continuation: %q", stdout.String()) + } + } + }) + } +} + +func TestRobotDiagnosticEscapesMultilineValuesAndEncodesContinuationArgv(t *testing.T) { + var stdout, stderr bytes.Buffer + diag := compat.Diagnostic{ + Code: compat.CodeIndexStale, + Summary: "first line\\with slash\r\nsecond line", + Continuation: []string{"recover", "--from", "path with spaces\\and\\slashes\r\nnext", "--dry-run"}, + } + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Failure: &startupFailure{Stage: startupStageStartupSync, Diagnostic: diag, Recoverable: true}} + }) + root.SetArgs([]string{"search", "needle", "--robot"}) + if err := root.Execute(); err == nil { + t.Fatalf("robot diagnostic command unexpectedly succeeded; stdout=%q", stdout.String()) + } + if stderr.Len() != 0 { + t.Fatalf("robot diagnostic wrote stderr: %q", stderr.String()) + } + fields := robotDiagnosticFields(t, stdout.String()) + if got := fields["diagnostic_summary"]; strings.ContainsAny(got, "\r\n") || !strings.Contains(got, `\\`) || !strings.Contains(got, `\r`) || !strings.Contains(got, `\n`) { + t.Fatalf("robot summary was not escaped as one line: %q in output %q", got, stdout.String()) + } + continuation := fields["diagnostic_continuation_argv"] + if continuation == "" || !strings.HasPrefix(continuation, "[") { + t.Fatalf("robot continuation is not encoded argv JSON: %q", continuation) + } + if strings.ContainsAny(continuation, "\r\n") || !strings.Contains(continuation, `\\`) || !strings.Contains(continuation, `\r`) || !strings.Contains(continuation, `\n`) { + t.Fatalf("robot continuation was not escaped as one line: %q", continuation) + } +} + +func robotDiagnosticFields(t *testing.T, output string) map[string]string { + t.Helper() + fields := map[string]string{} + for _, line := range strings.Split(strings.TrimSuffix(output, "\n"), "\n") { + if line == "" { + continue + } + parts := strings.SplitN(line, "=", 2) + if len(parts) != 2 { + t.Fatalf("robot line is not key=value: %q in %q", line, output) + } + if strings.ContainsAny(parts[1], "\r\n") { + t.Fatalf("robot value contains raw newline: %q in %q", parts[1], output) + } + fields[parts[0]] = parts[1] + } + return fields +} + +func TestDefaultStartupPolicyCallsSyncExactlyOnce(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "index.db") + setIndexPolicyEnv(t, dbPath, t.TempDir()) + calls := 0 + originalSync := startupSync + startupSync = func(*config.Config, io.Writer) error { + calls++ + return nil + } + t.Cleanup(func() { startupSync = originalSync }) + + result := defaultStartupPolicy(context.Background(), io.Discard) + if result.Failure != nil { + t.Fatalf("startup result=%+v", result) + } + if calls != 1 { + t.Fatalf("sync calls=%d, want 1", calls) + } +} + +func TestDefaultStartupPolicyNonrecoverableStages(t *testing.T) { + for _, tc := range []struct { + name string + stage startupStage + setup func(t *testing.T) + }{ + { + name: "legacy_source", + stage: startupStageLegacySource, + setup: func(t *testing.T) { + t.Helper() + home := t.TempDir() + t.Setenv("HOME", home) + configDir := filepath.Join(home, ".config", "backscroll") + if err := os.MkdirAll(configDir, 0o755); err != nil { + t.Fatalf("mkdir config dir: %v", err) + } + if err := os.WriteFile(filepath.Join(configDir, "config.toml"), []byte("[sources]\nke = [\"legacy.md\"]\n"), 0o644); err != nil { + t.Fatalf("write legacy config: %v", err) + } + }, + }, + { + name: "config_load", + stage: startupStageConfigLoad, + setup: func(t *testing.T) { + t.Helper() + dir := t.TempDir() + t.Setenv("HOME", filepath.Join(dir, "home")) + t.Setenv("BACKSCROLL_CONFIG_DIR", filepath.Join(dir, "config")) + t.Chdir(dir) + if err := os.WriteFile(filepath.Join(dir, "backscroll.toml"), []byte("database_path = [\n"), 0o644); err != nil { + t.Fatalf("write malformed local config: %v", err) + } + }, + }, + { + name: "active_manifest", + stage: startupStageActiveManifest, + setup: func(t *testing.T) { + t.Helper() + dir := t.TempDir() + cfgDir := filepath.Join(dir, "config") + setIndexPolicyEnv(t, filepath.Join(dir, "index.db"), cfgDir) + inputsDir := filepath.Join(cfgDir, "backscroll", "inputs") + if err := os.MkdirAll(inputsDir, 0o755); err != nil { + t.Fatalf("mkdir inputs dir: %v", err) + } + if err := os.WriteFile(filepath.Join(inputsDir, "bad.inputs.toml"), []byte("[[inputs]\nid = [\n"), 0o644); err != nil { + t.Fatalf("write malformed manifest: %v", err) + } + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + originalSync := startupSync + startupSync = func(*config.Config, io.Writer) error { + t.Fatal("startup sync should not run after nonrecoverable startup stage") + return nil + } + t.Cleanup(func() { startupSync = originalSync }) + tc.setup(t) + + result := defaultStartupPolicy(context.Background(), io.Discard) + failure := result.startupFailure() + if failure == nil { + t.Fatal("default startup unexpectedly succeeded") + } + if failure.Stage != tc.stage || failure.Recoverable { + t.Fatalf("failure=%+v, want stage %s and nonrecoverable", failure, tc.stage) + } + if failure.Cause == nil || failure.Diagnostic.Code == "" || failure.Diagnostic.Summary == "" || len(failure.Diagnostic.Continuation) != 0 { + t.Fatalf("failure diagnostic/cause not structured as nonrecoverable: %+v", failure) + } + }) + } +} + +func TestDiagnosticAlreadyRenderedOnlySuppressesTopLevelDiagnostic(t *testing.T) { + diagErr := indexDiagnosticError{diagnostic: compat.Diagnostic{Code: compat.CodeIndexStale, Summary: "already rendered"}} + if !diagnosticAlreadyRendered(diagErr) { + t.Fatal("top-level rendered diagnostic should be suppressed by main") + } + joined := errors.Join(diagErr, errors.New("recovery failed")) + if diagnosticAlreadyRendered(joined) { + t.Fatal("joined recovery failure should still be printed by main") + } + if diagErr.Error() == "" { + t.Fatal("diagnostic error should render a non-empty message") + } +} + +func TestMetadataCommandsSkipStartup(t *testing.T) { + for _, argv := range [][]string{{"--help"}, {"--version"}, {"search", "--help"}} { + calls := 0 + root := buildRootCmdWithStartup(io.Discard, io.Discard, func(context.Context, io.Writer) startupResult { + calls++ + return startupResult{} + }) + root.SetArgs(argv) + if err := root.Execute(); err != nil { + t.Fatalf("%v: %v", argv, err) + } + if calls != 0 { + t.Fatalf("%v invoked startup %d times", argv, calls) + } + } +} + +func TestRootExcludesDirectReadCommand(t *testing.T) { + root := buildRootCmd(io.Discard, io.Discard) + for _, cmd := range root.Commands() { + if cmd.Name() == "read" { + t.Fatal("public read command is registered") + } + } +} + +func TestCommandTreeExcludesIndexedOnlyFlag(t *testing.T) { + root := buildRootCmd(io.Discard, io.Discard) + var walk func(*cobra.Command) + walk = func(cmd *cobra.Command) { + if flag := cmd.Flags().Lookup("indexed-only"); flag != nil { + t.Errorf("%s registers forbidden --indexed-only", cmd.CommandPath()) + } + for _, child := range cmd.Commands() { + walk(child) + } + } + walk(root) +} + +func replaceRootCommandRunE(t *testing.T, root *cobra.Command, commandName string, runE func(*cobra.Command, []string) error) { + t.Helper() + for _, child := range root.Commands() { + if child.Name() == commandName { + child.Run = nil + child.RunE = runE + return + } + } + t.Fatalf("root command %q not found", commandName) +} + +func assertStartupResultInContext(t *testing.T, got, want startupResult) { + t.Helper() + if got.Config != want.Config { + t.Fatalf("startup config pointer mismatch: got=%p want=%p", got.Config, want.Config) + } + gotFailure := got.startupFailure() + wantFailure := want.startupFailure() + if (gotFailure == nil) != (wantFailure == nil) { + t.Fatalf("startup failure nil mismatch: got=%+v want=%+v", gotFailure, wantFailure) + } + if gotFailure == nil { + return + } + if gotFailure.Stage != wantFailure.Stage || gotFailure.Recoverable != wantFailure.Recoverable { + t.Fatalf("startup failure metadata=%+v, want %+v", gotFailure, wantFailure) + } + if (gotFailure.Cause == nil) != (wantFailure.Cause == nil) { + t.Fatalf("startup cause nil mismatch: got=%v want=%v", gotFailure.Cause, wantFailure.Cause) + } + if gotFailure.Cause != nil && gotFailure.Cause.Error() != wantFailure.Cause.Error() { + t.Fatalf("startup cause=%q, want %q", gotFailure.Cause.Error(), wantFailure.Cause.Error()) + } + if gotFailure.Diagnostic.Code != wantFailure.Diagnostic.Code || gotFailure.Diagnostic.Summary != wantFailure.Diagnostic.Summary || !reflect.DeepEqual(gotFailure.Diagnostic.Continuation, wantFailure.Diagnostic.Continuation) { + t.Fatalf("startup diagnostic=%+v, want %+v", gotFailure.Diagnostic, wantFailure.Diagnostic) + } +} diff --git a/cmd/backscroll/status.go b/cmd/backscroll/status.go index 367202b..b929274 100644 --- a/cmd/backscroll/status.go +++ b/cmd/backscroll/status.go @@ -16,14 +16,12 @@ import ( ) func newStatusCmd(stdout, stderr io.Writer) *cobra.Command { - var ( - jsonFormat bool - indexedOnly bool - ) + var jsonFormat bool cmd := &cobra.Command{ - Use: "status", - Short: "Show index status and configuration", + Use: "status", + Short: "Show index status and configuration", + SilenceUsage: true, Long: `Status displays information about the backscroll index, including: - Database path and size - Number of indexed files and messages @@ -31,61 +29,45 @@ func newStatusCmd(stdout, stderr io.Writer) *cobra.Command { - Configuration Use --json to output as JSON. -Status is read-only and never auto-syncs.`, +Startup preflight may sync before status runs; status itself only reads current index state.`, RunE: func(cmd *cobra.Command, args []string) error { - return runStatus(stdout, stderr, jsonFormat, indexedOnly) + startup := startupResultFrom(cmd) + if startup.Config == nil { + return fmt.Errorf("startup configuration unavailable") + } + return runStatus(cmd.Context(), stdout, stderr, startup.Config, jsonFormat) }, } cmd.Flags().BoolVar(&jsonFormat, "json", false, "Output as JSON") - cmd.Flags().BoolVar(&indexedOnly, "indexed-only", false, "Deprecated: status is always read-only") return cmd } -func runStatus(stdout, stderr io.Writer, jsonFormat, indexedOnly bool) error { - _ = indexedOnly - cfg, err := config.Load() - if err != nil { - return fmt.Errorf("load config: %w", err) +func runStatus(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, jsonFormat bool) (retErr error) { + db, diag, err := prepareIndex(ctx, cfg, indexDataRead) + if diag != nil { + return refuseDiagnostics(stdout, stderr, []compat.Diagnostic{*diag}, jsonFormat) } - - // Check if database exists without creating it. Status is diagnostic/read-only - // and must not auto-sync or open the index through a writer. - _, err = os.Stat(cfg.DatabasePath) - dbExists := err == nil - if err != nil && !os.IsNotExist(err) { - return fmt.Errorf("stat database: %w", err) + if err != nil { + return fmt.Errorf("prepare index: %w", err) } + defer func() { retErr = closeIndexDB(db, retErr) }() - var stats storage.Stats - if dbExists { - db, diag, err := prepareIndex(context.Background(), cfg, indexDiagnostic, false) - if diag != nil { - return refuseDiagnostics(stdout, stderr, []compat.Diagnostic{*diag}, jsonFormat) - } - if err != nil { - return fmt.Errorf("open database read-only: %w", err) - } - defer func() { _ = db.Close() }() - - stats, err = db.GetStats() - if err != nil { - return fmt.Errorf("get stats: %w", err) - } + stats, err := db.GetStats() + if err != nil { + return fmt.Errorf("get stats: %w", err) } // Resolve active inputs for status display activeInputNames, usingDeclarative := resolveInputsForStatus(cfg.SessionDirs) - // Get database file size - var dbSize int64 - if dbExists { - fileInfo, _ := os.Stat(cfg.DatabasePath) - if fileInfo != nil { - dbSize = fileInfo.Size() - } + // Get database file size after startup has prepared the canonical database. + fileInfo, err := os.Stat(cfg.DatabasePath) + if err != nil { + return fmt.Errorf("stat database: %w", err) } + dbSize := fileInfo.Size() // Format output if jsonFormat { @@ -93,11 +75,11 @@ func runStatus(stdout, stderr io.Writer, jsonFormat, indexedOnly bool) error { data := map[string]interface{}{ "database": map[string]interface{}{ "path": cfg.DatabasePath, - "exists": dbExists, + "exists": true, "size": dbSize, }, "index": map[string]interface{}{ - "usable": dbExists && stats.TotalFiles > 0, + "usable": stats.TotalFiles > 0, "total_files": stats.TotalFiles, "total_messages": stats.TotalMessages, "indexed_at": stats.IndexedAt, @@ -120,27 +102,17 @@ func runStatus(stdout, stderr io.Writer, jsonFormat, indexedOnly bool) error { _, _ = fmt.Fprintf(stdout, "=================\n\n") _, _ = fmt.Fprintf(stdout, "Database:\n") - if dbExists { - _, _ = fmt.Fprintf(stdout, " Path: %s\n", cfg.DatabasePath) - _, _ = fmt.Fprintf(stdout, " Size: %.2f MB\n", float64(dbSize)/1024/1024) - } else { - _, _ = fmt.Fprintf(stdout, " Path: %s (not yet created)\n", cfg.DatabasePath) - } - - if dbExists { - _, _ = fmt.Fprintf(stdout, "\nIndex:\n") - _, _ = fmt.Fprintf(stdout, " Files indexed: %d\n", stats.TotalFiles) - _, _ = fmt.Fprintf(stdout, " Messages indexed: %d\n", stats.TotalMessages) - _, _ = fmt.Fprintf(stdout, " Chunks stored: %d\n", stats.TotalChunks) - _, _ = fmt.Fprintf(stdout, " Embeddings: %d\n", stats.TotalEmbeddings) - _, _ = fmt.Fprintf(stdout, " Vectors stored: %d\n", stats.TotalVectors) - if !stats.IndexedAt.IsZero() { - _, _ = fmt.Fprintf(stdout, " Last indexed: %s\n", stats.IndexedAt.Format("2006-01-02 15:04:05 MST")) - } - } else { - _, _ = fmt.Fprintf(stdout, "\nIndex: Not yet created\n") - _, _ = fmt.Fprintf(stdout, " Files indexed: 0\n") - _, _ = fmt.Fprintf(stdout, " Messages indexed: 0\n") + _, _ = fmt.Fprintf(stdout, " Path: %s\n", cfg.DatabasePath) + _, _ = fmt.Fprintf(stdout, " Size: %.2f MB\n", float64(dbSize)/1024/1024) + + _, _ = fmt.Fprintf(stdout, "\nIndex:\n") + _, _ = fmt.Fprintf(stdout, " Files indexed: %d\n", stats.TotalFiles) + _, _ = fmt.Fprintf(stdout, " Messages indexed: %d\n", stats.TotalMessages) + _, _ = fmt.Fprintf(stdout, " Chunks stored: %d\n", stats.TotalChunks) + _, _ = fmt.Fprintf(stdout, " Embeddings: %d\n", stats.TotalEmbeddings) + _, _ = fmt.Fprintf(stdout, " Vectors stored: %d\n", stats.TotalVectors) + if !stats.IndexedAt.IsZero() { + _, _ = fmt.Fprintf(stdout, " Last indexed: %s\n", stats.IndexedAt.Format("2006-01-02 15:04:05 MST")) } _, _ = fmt.Fprintf(stdout, "\nConfiguration:\n") @@ -176,8 +148,8 @@ func resolveInputsForStatus(sessionDirs []string) ([]string, bool) { return names, true } -func recoveryDiagnosticsForIndex(db *storage.Database, activePath string) ([]compat.Diagnostic, error) { - input, diag, err := storage.ReadRecoveryInput(context.Background(), db) +func recoveryDiagnosticsForIndex(ctx context.Context, db *storage.Database, activePath string) ([]compat.Diagnostic, error) { + input, diag, err := storage.ReadRecoveryInput(ctx, db) if diag != nil { d := continuationFor(*diag, activePath) return []compat.Diagnostic{d}, err diff --git a/cmd/backscroll/sync_helpers.go b/cmd/backscroll/sync_helpers.go index 4d9be8c..3c90723 100644 --- a/cmd/backscroll/sync_helpers.go +++ b/cmd/backscroll/sync_helpers.go @@ -14,12 +14,11 @@ import ( ) var ( - maybeAutoSyncOpen = storage.Open - maybeAutoSyncActiveInputs = input_config.ActiveInputs - maybeAutoSyncLoadGlobalRegistry = projects.LoadGlobalRegistry - maybeAutoSyncNewRegistry = newDefaultAutoSyncRegistry - maybeAutoSyncSyncFiles = func(db *storage.Database, files []storage.IndexedFile) error { return db.SyncFiles(files) } - maybeAutoSyncProgress io.Writer = io.Discard + maybeAutoSyncOpen = storage.Open + maybeAutoSyncActiveInputs = input_config.ActiveInputs + maybeAutoSyncLoadGlobalRegistry = projects.LoadGlobalRegistry + maybeAutoSyncNewRegistry = newDefaultAutoSyncRegistry + maybeAutoSyncSyncFiles = func(db *storage.Database, files []storage.IndexedFile) error { return db.SyncFiles(files) } ) func newDefaultAutoSyncRegistry() *readers.Registry { @@ -35,7 +34,7 @@ func newDefaultAutoSyncRegistry() *readers.Registry { // maybeAutoSync performs an incremental sync operation if the database exists. // It is intended to be called before query commands to ensure fresh index state. // If sync fails, it returns an error (caller decides whether to warn/ignore). -func maybeAutoSync(cfg *config.Config) (retErr error) { +func maybeAutoSync(cfg *config.Config, progress io.Writer) (retErr error) { // Open database for reading to check if it exists // (this will auto-create if missing) db, err := maybeAutoSyncOpen(cfg.DatabasePath) @@ -107,7 +106,7 @@ func maybeAutoSync(cfg *config.Config) (retErr error) { continue } staleParsesDone++ - _, _ = fmt.Fprintf(maybeAutoSyncProgress, "Re-parsing stale file %d/%d: %s\n", staleParsesDone, len(stalePaths), ref) + _, _ = fmt.Fprintf(progress, "Re-parsing stale file %d/%d: %s\n", staleParsesDone, len(stalePaths), ref) } pf, err := reader.Parse(ref, def) @@ -187,7 +186,7 @@ func maybeAutoSync(cfg *config.Config) (retErr error) { return fmt.Errorf("re-mine templates for %s: %w", sourcePath, err) } if deletedCount > 0 { - _, _ = fmt.Fprintf(maybeAutoSyncProgress, "Deleted %d stuck templates from %s\n", deletedCount, sourcePath) + _, _ = fmt.Fprintf(progress, "Deleted %d stuck templates from %s\n", deletedCount, sourcePath) } } } diff --git a/cmd/backscroll/validate.go b/cmd/backscroll/validate.go index 38c761e..b803ac2 100644 --- a/cmd/backscroll/validate.go +++ b/cmd/backscroll/validate.go @@ -5,7 +5,6 @@ import ( "encoding/json" "fmt" "io" - "os" "github.com/spf13/cobra" @@ -14,12 +13,12 @@ import ( ) func newValidateCmd(stdout, stderr io.Writer) *cobra.Command { - var indexedOnly bool var jsonFormat bool cmd := &cobra.Command{ - Use: "validate", - Short: "Validate the index integrity", + Use: "validate", + Short: "Validate the index integrity", + SilenceUsage: true, Long: `Validate checks the integrity of the SQLite index by verifying: - Required tables exist - FTS5 virtual table is set up correctly @@ -27,50 +26,39 @@ func newValidateCmd(stdout, stderr io.Writer) *cobra.Command { Returns an error if validation fails. -Validate is read-only and never auto-syncs.`, +Command startup may synchronize active inputs before this handler runs. The +validate handler then checks the prepared SQLite index without performing a +second sync.`, RunE: func(cmd *cobra.Command, args []string) error { - return runValidate(stdout, stderr, indexedOnly, jsonFormat) + startup := startupResultFrom(cmd) + if startup.Config == nil { + return fmt.Errorf("startup configuration unavailable") + } + return runValidate(cmd.Context(), stdout, stderr, startup.Config, jsonFormat) }, } - cmd.Flags().BoolVar(&indexedOnly, "indexed-only", false, "Deprecated: validate is always read-only") cmd.Flags().BoolVar(&jsonFormat, "json", false, "Output as JSON") return cmd } -func runValidate(stdout, stderr io.Writer, indexedOnly bool, jsonFormat bool) error { - _ = indexedOnly - cfg, err := config.Load() - if err != nil { - return fmt.Errorf("load config: %w", err) - } - - if _, err := os.Stat(cfg.DatabasePath); os.IsNotExist(err) { - if jsonFormat { - return json.NewEncoder(stdout).Encode(map[string]any{"valid": true, "database_exists": false}) - } - _, _ = fmt.Fprintf(stdout, "✓ Index validation skipped: database not found\n") - return nil - } else if err != nil { - return fmt.Errorf("stat database: %w", err) - } - - db, diag, err := prepareIndex(context.Background(), cfg, indexDiagnostic, false) +func runValidate(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, jsonFormat bool) (retErr error) { + db, diag, err := prepareIndex(ctx, cfg, indexDataRead) if diag != nil { return refuseDiagnostics(stdout, stderr, []compat.Diagnostic{*diag}, jsonFormat) } if err != nil { - return fmt.Errorf("open database read-only: %w", err) + return fmt.Errorf("prepare index: %w", err) } - defer func() { _ = db.Close() }() + defer func() { retErr = closeIndexDB(db, retErr) }() if err := db.Validate(); err != nil { activePath, resolveErr := resolveActiveIndexPath(cfg.DatabasePath) if resolveErr != nil { return fmt.Errorf("resolve active index path: %w", resolveErr) } - diagnostics, inspectErr := recoveryDiagnosticsForIndex(db, activePath) + diagnostics, inspectErr := recoveryDiagnosticsForIndex(ctx, db, activePath) if inspectErr != nil { return fmt.Errorf("inspect recovery diagnostics: %w", inspectErr) } diff --git a/docs/adr/.stem b/docs/adr/.stem index 297efaa..883a6a2 100644 --- a/docs/adr/.stem +++ b/docs/adr/.stem @@ -1,3 +1,4 @@ +root: true version: 2 schema: tipo: diff --git a/docs/adr/0002-exigir-sync-y-consulta-desde-sqlite.md b/docs/adr/0002-exigir-sync-y-consulta-desde-sqlite.md new file mode 100644 index 0000000..24b3deb --- /dev/null +++ b/docs/adr/0002-exigir-sync-y-consulta-desde-sqlite.md @@ -0,0 +1,72 @@ +--- +tipo: adr +estado: accepted +fecha: 2026-08-20 +contexto: El port Go reintrodujo el comando público read y varias rutas indexed-only que permiten consultar archivos o snapshots sin pasar por la ingesta y el índice perenne definidos por el North Star. +decision: Ejecutar un sync incremental central antes de toda operación, retirar read e indexed-only y permitir únicamente recover como continuación controlada después de un intento de sync fallido. +consecuencias: SQLite vuelve a ser la única fuente pública de consulta; el CLI pierde superficies incompatibles y todos los comandos operativos asumen un índice recién sincronizado. +--- + +# Exigir sync y consulta desde SQLite + +## Contexto + +Backscroll define los archivos de sesiones, planes y documentos como inputs transitorios. SQLite es el registro episódico perenne: conserva información cuando los archivos expiran y debe suministrar toda consulta visible al usuario. + +El comando `backscroll read` viola esa frontera porque abre un archivo físico mediante `internal/reader` sin ingerirlo ni consultar SQLite. El flag `--indexed-only` crea una segunda excepción al omitir discovery y sync para consultar una snapshot existente. Además, `prepareIndex(..., autoSync bool)` distribuye la decisión de frescura entre handlers. + +La tarea histórica `docs/roadmap/T001-remove-public-read-command.md` ya había identificado y eliminado `read` en `f9c37eb`. El port Go lo reintrodujo en `104c81e` al reconstruir el inventario anterior de comandos. PR #43 alineó las guías con el árbol Cobra resultante, mostrando que validar documentación contra Cobra no basta cuando Cobra contradice la arquitectura. + +## Decisión + +Toda operación pública ejecutará una política central de arranque desde `PersistentPreRunE`: + +1. cargar configuración; +2. rechazar fuentes legacy; +3. validar todos los manifests activos; +4. preparar un índice compatible; +5. ejecutar exactamente un intento de sync incremental de arranque; +6. ejecutar el handler solicitado solamente después del éxito. + +Se retiran sin periodo de deprecación: + +- el comando público `backscroll read`; +- el paquete `internal/reader` cuando quede sin consumidores; +- todos los flags y caminos `--indexed-only`; +- la decisión `autoSync` distribuida entre comandos. + +`recover` es la única continuación permitida después de un intento de sync fallido, porque su función es reparar el estado que impidió completar el arranque. Tras instalar y verificar la base canónica, ejecuta un sync post-instalación antes de informar éxito; ese segundo intento opera sobre la base recuperada, no repite el arranque contra la base fallida. No puede devolver resultados cached. Help y version permanecen libres de efectos laterales porque no ejecutan un cuerpo operativo. + +Los planes y documentos Markdown se ingieren mediante manifests y se consultan desde SQLite. `search --source-path` permanece como búsqueda DB-backed por ruta. + +Esta decisión limita ADR 0001: Cobra sigue siendo la fuente de verdad de la sintaxis publicada, pero no de los invariantes arquitectónicos. Tests dedicados deben impedir que Cobra vuelva a registrar superficies prohibidas. + +Quedan supersedidas las secciones de diseños posteriores que preservaron `read` directo o presentaron `--indexed-only` como contrato público vigente. Los documentos históricos permanecen intactos como evidencia de contexto. + +## Alternativas descartadas + +- **Forzar sync en cada handler:** conserva una política distribuida que un comando nuevo puede omitir. +- **Sincronizar antes de parsear Cobra:** produciría efectos laterales para help, version y argumentos inválidos, y dificultaría la continuación de recovery. +- **Mantener read como diagnóstico:** conserva una segunda fuente pública de verdad y repite la ambigüedad que originó la regresión. +- **Conservar indexed-only para auditorías:** contradice la frescura obligatoria; una futura API de snapshot necesita un contrato separado y versionado. +- **Añadir un daemon:** no es necesario; el hash incremental antes de cada operación satisface la garantía aprobada. + +## Consecuencias + +### Positivas + +- SQLite es la única superficie pública de consulta. +- Todos los comandos nuevos heredan sync sin configuración adicional. +- Los fallos de ingesta abortan sin servir filas stale. +- Los tests pueden validar tanto documentación→CLI como CLI→arquitectura. + +### Negativas + +- `read` y `--indexed-only` se eliminan como breaking changes. +- `config`, `status` y `validate` pasan a ejecutar trabajo de sync antes de responder. +- Consumidores que dependían de snapshots deben migrar. +- Numerosos tests herméticos requieren manifests en lugar del bypass indexed-only. + +### Riesgo aceptado + +Cada invocación paga discovery y hashing incremental. Los archivos sin cambios se omiten; se acepta ese costo para preservar la consistencia de la memoria episódica. diff --git a/docs/audit-integration.md b/docs/audit-integration.md index 7a63b34..9ad7417 100644 --- a/docs/audit-integration.md +++ b/docs/audit-integration.md @@ -2,26 +2,29 @@ Backscroll owns the perennial corpus and supported CLI query surfaces. A downstream audit tool owns deterministic findings, thresholds, redaction, report rendering, and any ADR or backlog creation. -## Establish a read-only boundary +## Supported operational boundary -Use diagnostics before reading an existing snapshot: +Every operational command validates active manifests and attempts one incremental +sync before executing. Session, plan, and Markdown files are ingestion inputs; +SQLite is the perennial record used by search, list, patterns, status, and validate. +Use `--source-path` on search as a filter, paired with query text, for database-backed retrieval scoped to a known input path. + +Use diagnostics at the start of an audit run: ```bash backscroll status --json backscroll validate --json ``` -Both commands are always read-only and never run input discovery or sync. The deprecated `--indexed-only` flag is accepted on them but does not change behavior. - -For query commands, `--indexed-only` suppresses auto-sync and reads the configured SQLite snapshot: +Then query through the database-backed CLI surfaces: ```bash -backscroll list --json --indexed-only --all-projects --order timestamp:asc --limit 100 -backscroll search --text "permission denied" --json --indexed-only --all-projects -backscroll patterns --kind failures --json --indexed-only --all-projects +backscroll list --json --all-projects --order timestamp:asc --limit 100 +backscroll search --text "permission denied" --json --all-projects +backscroll patterns --kind failures --json --all-projects ``` -Without `--indexed-only`, `list`, `search`, and `patterns` validate active manifests and incrementally index changed inputs before querying. +Human startup progress and warnings use stderr. JSON/robot startup progress is discarded so stdout remains machine-readable, and structured diagnostics stay parseable in machine modes. ## Status JSON @@ -35,20 +38,21 @@ Status is preflight metadata. It does not expose transcript content. ## Session discovery -`backscroll list --json --indexed-only` returns a JSON object containing `count` and `sessions`. Each session summary includes its path, project, timestamp, and tags. Supported filters are `--project`, `--all-projects`, `--order`, `--limit`, `--offset`, and legacy `--recent`. +`backscroll list --json` returns a JSON object containing `count` and `sessions`. Each session summary includes its path, project, timestamp, and tags. Supported filters are `--project`, `--all-projects`, `--order`, `--limit`, `--offset`, and legacy `--recent`. `list` does not expose message-level filters such as `--source-path`, `--source`, `--role`, `--after`, `--before`, or `--content-type`. ## Message and tool investigation -`backscroll search` requires a non-empty query and returns ranked matching rows. It supports path, source, project, role, date, tag, and content-type filters. For example: +The search command returns ranked matching rows. It supports path, source, project, role, date, tag, and content-type filters. For example: ```bash -backscroll search --text "go test" --content-type tool --source-path "*/example/*.jsonl" --indexed-only --json +backscroll search --text "go test" --content-type tool --source-path "*/example/*.jsonl" --json +backscroll search --text "$QUERY" --source-path "*session-id*" --all-projects --json ``` Search is an investigation surface, not an exhaustive corpus export: ranking, limits, and token budgets may omit rows. The current public CLI does not provide an empty-query stream of every stored message. Consumers requiring a complete message-level export must not infer one from `list` or `search`; they need a separately designed read-only API or an explicitly versioned database integration. ## Privacy and raw-content boundary -Backscroll stores normalized message text and serialized tool content in SQLite. The public CLI does not make raw provider JSONL a downstream schema contract. `backscroll read --path ` directly parses a user-supplied file when raw-source access is intentional, but it requires that file to remain on disk and is separate from indexed snapshot reads. +Backscroll stores normalized message text and serialized tool content in SQLite. The public CLI does not make raw provider JSONL a downstream schema contract. Database-backed retrieval through search with the `--source-path` filter and query text is the supported drill-down path for a known input path. Raw provider files remain ingestion inputs, not the normal audit read boundary. diff --git a/docs/configuration.md b/docs/configuration.md index 6e97c96..4fa24ea 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -80,7 +80,7 @@ role = "$.message.role" selector = "$.message.content" ``` -Markdown documents use the same input list with `decode.format = "markdown"` for whole-document indexing or `decode.format = "markdown_sections"` for `## ` header splitting: +Markdown documents use the same input list with `decode.format = "markdown_document"` for whole-document indexing or `decode.format = "markdown_sections"` for `## ` header splitting: ```toml version = 1 @@ -107,7 +107,7 @@ roots = ["docs/knowledge"] include = ["**/*.md"] [inputs.decode] -format = "markdown" +format = "markdown_document" ``` Invalid TOML, unknown fields, unsupported versions, invalid selectors/globs/regexes, or invalid active manifests fail with an error that includes the manifest path. Missing discovery roots are skipped so shipped Claude/Pi presets can coexist on machines that only have one tool installed. @@ -119,14 +119,19 @@ Invalid TOML, unknown fields, unsupported versions, invalid selectors/globs/rege backscroll config backscroll config --json -# Inspect the existing index without triggering ingestion. +# Inspect the index and manifest health. backscroll status --json -# Query commands validate manifests and incrementally index changed inputs. +# Operational commands validate manifests and incrementally index changed inputs. backscroll search --text "migration plan" --all-projects backscroll list --order timestamp:desc --limit 20 ``` -There is no public `inputs` or `sync` command. Active manifests are validated during command preflight; invalid manifests fail with their path before indexing begins. `search`, `list`, and `patterns` run incremental auto-sync unless `--indexed-only` is supplied. Use `backscroll rebuild` only to re-derive FTS and other derived data from the perennial database before an incremental sync; it is not a replacement for manifest validation. +Every operational command validates active manifests and attempts one incremental +sync before executing. Session, plan, and Markdown files are ingestion inputs; +SQLite is the perennial record used by search, list, patterns, status, and validate. +Use `--source-path` on search as a filter, paired with query text, for database-backed retrieval scoped to a known input path. -See [the generic input contract](input-contract.md) for the full manifest schema. For read-only audit consumers, see the [downstream audit integration contract](audit-integration.md). +There is no public `inputs` or `sync` command. Active manifests are validated during command preflight; invalid manifests fail with their path before indexing begins. Use `backscroll rebuild` only after the mandatory startup sync has prepared the database; the handler re-derives FTS and other derived data from the perennial database and performs no second sync. It is not a replacement for manifest validation. + +See [the generic input contract](input-contract.md) for the full manifest schema. For downstream audit consumers, see the [downstream audit integration contract](audit-integration.md). diff --git a/docs/eval/README.md b/docs/eval/README.md index 03fa100..6ca7595 100644 --- a/docs/eval/README.md +++ b/docs/eval/README.md @@ -60,8 +60,8 @@ scripts/eval.sh --limit 5 ### Output ``` -Backscroll Evaluation — Recall@5 Metric -======================================== +Backscroll Evaluation — Recall@5 Metric with Ground-Truth Matching +==================================================================== Index: 1719 files, 192507 messages Eval-set: docs/eval/queries.toml @@ -70,9 +70,8 @@ Loaded 20 queries from eval-set Results ======= Queries evaluated: 20 -Results found: 18 -Results at rank ≤5: 16 -Recall@5: 80.0% +Matches found at rank ≤4: 16 +Recall@5 (with ground-truth matching): 80.0% ✓ Recall@5 target met (≥80%) ``` @@ -83,7 +82,7 @@ Exit code: 0 (success) if recall@5 ≥ 80%, else 1 (gate failed). - **Recall@5 ≥ 80%**: Most queries return useful results in the top 5. Agents can rely on backscroll for recall. - **Recall@5 60–80%**: Some queries miss the top 5; scoring or content may be improving. Check `--verbose` output for which queries fail. -- **Recall@5 < 60%**: Significant ranking issue or missing content. Run `scripts/eval.sh --verbose`, then inspect failed queries with `backscroll search --robot --fields full` and check the index with `backscroll status`. +- **Recall@5 < 60%**: Significant ranking issue or missing content. Run `scripts/eval.sh --verbose`, then inspect failed queries with `backscroll search --text "$QUERY" --robot --fields full` and check the index with `backscroll status`. ## Eval-Set Evolution @@ -91,7 +90,7 @@ Exit code: 0 (success) if recall@5 ≥ 80%, else 1 (gate failed). 1. Run `scripts/eval.sh --verbose` and log baseline recall@5. 2. If recall drops, investigate: - New content added by slice (new tool calls, reasoning)? Queries may need refinement. - - Ranking changed? Run `backscroll search --robot --fields full` and inspect scores. + - Ranking changed? Run `backscroll search --text "$QUERY" --robot --fields full` and inspect scores. 3. Document regressions in the PR or commit message. **After M1 completion:** @@ -107,7 +106,7 @@ Queries were extracted from: 4. **Error recovery** — common bugs and investigation patterns. 5. **Cross-project patterns** — behaviors that span multiple projects. -Each query was verified to return a meaningful result on the live index (as of 2026-07-02 snapshot). +Each query was verified to return a meaningful result on the then-current live index as of 2026-07-02. Re-run the eval against the current mandatory-startup-sync index before drawing conclusions. ## Notes diff --git a/docs/input-contract.md b/docs/input-contract.md index f0ea52d..ab90ee1 100644 --- a/docs/input-contract.md +++ b/docs/input-contract.md @@ -33,7 +33,7 @@ user edits are not overwritten. discover -> decode -> record -> map -> content -> text -> emit -> search_items ``` -`search_items` is optimized for both retrieval UX and audit surfaces. Each indexed row carries `source`, `source_path`, `project`, `role`, `content_type`, `timestamp`, `ordinal`, and bounded `text`. Tool inputs, outputs, and errors are indexed with `content_type='tool'` and stored in a separate FTS5 index (`tool_fts`) for substring/exact matching; prose and code use the main messages_fts index for morphological search. Downstream consumers should treat the JSON surfaces described in [Downstream audit integration contract](audit-integration.md) as the stable read boundary. +`search_items` is optimized for both retrieval UX and audit surfaces. Each indexed row carries `source`, `source_path`, `project`, `role`, `content_type`, `timestamp`, `ordinal`, and bounded `text`. Tool inputs, outputs, and errors are indexed with `content_type='tool'` and stored in a separate FTS5 index (`tool_fts`) for substring/exact matching; prose and code use the main messages_fts index for morphological search. Every operational command validates active manifests and attempts one incremental sync before executing. Session, plan, and Markdown files are ingestion inputs; SQLite is the perennial record used by search, list, patterns, status, and validate. Use `--source-path` on search as a filter, paired with query text, for database-backed retrieval scoped to a known input path. Downstream consumers should treat the JSON surfaces described in [Downstream audit integration contract](audit-integration.md) as the stable read boundary. ## File shape @@ -138,7 +138,7 @@ Declares the technical file format. | Field | Type | Default | Meaning | |---|---:|---:|---| -| `format` | enum | required | MVP values: `jsonl`, `json`, `markdown`, `markdown_sections`. | +| `format` | enum | required | MVP values: `jsonl`, `json`, `markdown_document`, `markdown_sections`. | | `encoding` | string | `utf-8` | Text encoding for file reads. | ## `record` @@ -161,7 +161,7 @@ MVP operators are `eq`, `ne`, `in`, `exists`, and `missing`. ## `map` -Maps record fields to Backscroll metadata. Required for `jsonl` and `json` inputs. Markdown inputs (`markdown` and `markdown_sections`) emit document text directly and may omit this section. `project` is also evaluated against the full JSON document (or each JSONL line) before record filtering, so file/session metadata records can provide a project for emitted messages. +Maps record fields to Backscroll metadata. Required for `jsonl` and `json` inputs. Markdown inputs (`markdown_document` and `markdown_sections`) emit document text directly and may omit this section. `project` is also evaluated against the full JSON document (or each JSONL line) before record filtering, so file/session metadata records can provide a project for emitted messages. | Field | Type | Default | Meaning | |---|---:|---:|---| @@ -325,7 +325,7 @@ drop_empty = true ## Markdown document inputs -Plans and external documents are declared as normal inputs. Whole-document markdown uses `decode.format = "markdown"`; sectioned markdown uses `decode.format = "markdown_sections"`, which splits on `## ` headers and preserves any pre-header preamble as the first message. +Plans and external documents are declared as normal inputs. Whole-document markdown uses `decode.format = "markdown_document"`; sectioned markdown uses `decode.format = "markdown_sections"`, which splits on `## ` headers and preserves any pre-header preamble as the first message. ```toml version = 1 @@ -356,7 +356,7 @@ roots = ["docs/knowledge"] include = ["**/*.md"] [inputs.decode] -format = "markdown" +format = "markdown_document" ``` Use `source = "plan"`, `"ke"`, `"decision"`, `"memory"`, `"rule"`, `"spec"`, or `"backlog"` to preserve the semantic source stored in SQLite. Specs can opt into `markdown_sections` when section-level indexing is desired. diff --git a/docs/patterns.md b/docs/patterns.md index 9b6265a..11d80cd 100644 --- a/docs/patterns.md +++ b/docs/patterns.md @@ -23,7 +23,7 @@ backscroll patterns --kind commands|failures|templates|corrections|sequences ``` Base flags for every kind: `--project` / `--all-projects`, `--tag`, -`--limit` / `--offset`, `--indexed-only`, `--json` / `--robot`. +`--limit` / `--offset`, `--json` / `--robot`. ### commands — what runs most @@ -102,10 +102,12 @@ not from loop state: there is nothing to checkpoint. ## Operating notes +- Every operational command validates active manifests and attempts one incremental sync before executing. Session, plan, and Markdown files are ingestion inputs; SQLite is the perennial record used by search, list, patterns, status, and validate. Use `--source-path` on search as a filter, paired with query text, for database-backed retrieval scoped to a known input path. - Zero-result guidance goes to stderr; stdout stays clean for `--json`. +- Robot mode on search emits `result_N_field=value` lines and escapes search string values with backslash as `\\`, carriage return as `\r`, and newline as `\n`. - A malformed `categories.toml` fails the command (non-zero exit) rather than masquerading as an empty result. - Historical supply: rich capture exists for rows synced after migration - v8; `rebuild` backfills expired files from stored text (lossy for tool + v8; `rebuild` backfills expired-file derived data from stored text (lossy for tool events, marked `extraction_version=0`) and stale on-disk files re-parse at full fidelity during sync (capped per run, FIFO). diff --git a/docs/read.md b/docs/read.md index 85b950e..7578102 100644 --- a/docs/read.md +++ b/docs/read.md @@ -1,60 +1,56 @@ --- estado: Completed --- -# Direct File Reading and Indexed Path Search +# Migration from Direct File Reads to Source Path Retrieval -Backscroll exposes two distinct retrieval paths: +The former direct-read CLI path has been removed from living guidance. Backscroll now has one operational retrieval path: -- `backscroll read` parses a session or plan file directly from disk. It does not read from the SQLite index. -- `backscroll search --source-path` searches rows already stored in SQLite and narrows matches to an indexed path or pattern. +```text +active manifests -> mandatory startup sync -> perennial SQLite -> database-backed query +``` -Choose direct reading when the file still exists and you need its contents. Choose indexed search when the database is the source of truth, including sessions whose original files may have expired. +Every operational command validates active manifests and attempts one incremental +sync before executing. Session, plan, and Markdown files are ingestion inputs; +SQLite is the perennial record used by search, list, patterns, status, and validate. +Use `--source-path` on search as a filter, paired with query text, for database-backed retrieval scoped to a known input path. -## Direct file reading +## Database-backed source path lookup -```bash -# Structured message output -backscroll read --path ~/.claude/projects/example/session.jsonl +Use the search `--source-path` filter when you know a stored input path, a path fragment, or a session identifier. The filter matches the `search_items.source_path` value stored in SQLite; it narrows a normal text query and does not parse arbitrary files from disk. -# Positional path is equivalent -backscroll read ~/.claude/projects/example/session.jsonl +```bash +# Exact or glob-style stored input path. +backscroll search --text "query terms" --source-path "/home/user/.claude/projects/example/session.jsonl" --robot +backscroll search --text "query terms" --source-path "*/example/*.jsonl" --robot -# Concise semantic rows, limited to the last 45 rows -backscroll read --path ~/.claude/projects/example/session.jsonl --semantic --tail 45 +# UUID/session-id fragment in an indexed source_path. +backscroll search --text "artifact literal" --source-path "*019e0d38-c437-7565-ba11-5dd57d516744*" --all-projects --json +backscroll search --text "$QUERY" --source session --source-path "*session-id*" --all-projects --json -# Human-readable semantic formatting -backscroll read --path ~/.claude/projects/example/session.jsonl --semantic --pretty +# Tool activity matching a known term within the selected path. +backscroll search --text "go test" --content-type tool --source-path "*/example/*.jsonl" --json ``` -Use either the positional path or `--path`, not both. `--tail` and `--pretty` apply to semantic output. Direct reading requires the source file to exist and does not fall back to the perennial index. - -## Indexed path search +Use `--fields full` and a bounded `--max-tokens` value when drilling into a selected path for agent context: ```bash -# Exact indexed path -backscroll search --text "query terms" --source-path "/home/user/.claude/projects/example/session.jsonl" --robot - -# Glob-style path pattern -backscroll search --text "query terms" --source-path "*/example/*.jsonl" --robot - -# UUID/session-id fragment in an indexed source_path -backscroll search --text "query terms" --source session --source-path "*019e0d38-c437-7565-ba11-5dd57d516744*" --all-projects --robot - -# Tool activity matching a known term within the selected path -backscroll search --text "go test" --content-type tool --source-path "*/example/*.jsonl" --indexed-only --json +backscroll search --text "$QUERY" --source-path "*session-id*" --all-projects --robot --fields full --max-tokens 4000 ``` -`search` requires a non-empty query. `--source-path` filters rows whose stored `source_path` equals the value or matches its `*`/SQL `LIKE` pattern. It does not parse arbitrary files. +## What changed -`backscroll list` lists indexed sessions and supports project, ordering, limit, and offset flags. It does **not** support `--source-path`, source, role, date, or content-type filtering; use `backscroll search` for those filters. +- Raw session, plan, and Markdown files are ingestion inputs, not the normal retrieval boundary. +- SQLite is the perennial record. Rows remain queryable after source files expire unless `purge` removes them explicitly. +- `list` is for session/document summaries. It supports project, ordering, limit, offset, JSON, and robot output; it does not support message-level filters such as `--source-path`, `--source`, `--role`, date windows, or `--content-type`. +- `search` is the drill-down surface for known paths, source types, roles, dates, tags, and content types. -## Auto-sync and snapshot reads +## Raw-file boundary -Normal `search` and `list` calls validate active manifests and incrementally index changed inputs before querying. Add `--indexed-only` to read the existing index without discovery or mutation. `backscroll status` and `backscroll validate` are always read-only. +Do not fall back to `cat`, `jq`, Python, or filesystem session hunting for normal retrieval. Those raw-file techniques are reserved for explicitly authorized indexing-bug diagnosis after the database-backed commands and diagnostics have been reported. ## Exit codes | Code | Meaning | |------|---------| -| `0` | Read or search completed; search results may be empty | -| `1` | Invalid arguments, unreadable file, manifest preflight failure, or database/query failure | +| `0` | Query and any required startup sync completed; results may be empty | +| `1` | Invalid arguments, manifest preflight failure, or database/query failure | diff --git a/docs/search.md b/docs/search.md index 5a278a6..c3c8484 100644 --- a/docs/search.md +++ b/docs/search.md @@ -3,7 +3,7 @@ estado: Completed --- # Search Engine -`backscroll search` performs full-text search across all indexed sessions using BM25 relevance ranking. Results include highlighted snippets showing where the query matched. +The search command performs full-text search across all indexed sessions using BM25 relevance ranking. Results include highlighted snippets showing where the query matched. `--source-path` is a filter: every executable search example must include positional query text or `--text `. ## CLI Usage @@ -13,7 +13,7 @@ backscroll search "error handling" --project "backscroll" backscroll search "architecture" --json backscroll search "deployment" --robot --max-tokens 2000 backscroll search "refactor" --fields full -backscroll search "handoff" --source-path "*/session.jsonl" --robot +backscroll search "artifact literal" --source-path "*/session.jsonl" --robot ``` ### Flags @@ -21,21 +21,22 @@ backscroll search "handoff" --source-path "*/session.jsonl" --robot | Flag | Description | |------|-------------| | `--project ` | Filter results to a specific project | -| `--json` | Output as JSON lines (one object per result) | -| `--robot` | Output as compact tab-separated format | +| `--json` | Output as a JSON array | +| `--robot` | Output compact `result_N_field=value` lines | | `--fields minimal\|full` | Field set to include (default: `minimal`) | | `--max-tokens ` | Approximate token limit for total output | -| `--source-path ` | Filter by indexed `source_path`; exact paths or `*`/SQL `LIKE` patterns | +| `--source-path ` | Filter a normal text query by indexed `source_path`; exact paths or `*`/SQL `LIKE` patterns | ## Output Formats ### Text (default) -Human-readable output with terminal bold for match highlights. Each result shows the session path, relevance score, and a snippet: +Human-readable output with terminal bold for match highlights. Each result uses the exact text-layout envelope emitted by the CLI: ``` ---- -[SESSION] ~/.claude/projects/abc/sessions/session.jsonl (Score: 12.34) +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +Rank: 1 | Source: session | Role: assistant | Score: 12.34 +Path: /home/user/.claude/projects/backscroll/sessions/abc123/session.jsonl ...the migration plan involves three phases... ``` @@ -43,27 +44,50 @@ Match markers (`>>>` and `<<<` in the raw snippet) are rendered as bold text in ### JSON -One JSON object per line. With `--fields minimal`: +`--json` emits one JSON array. With `--fields minimal`: ```json -{"source_path": "~/.claude/.../session.jsonl", "snippet": "...matched text...", "score": 12.34} +[ + {"source_path": "~/.claude/.../session.jsonl", "snippet": "...matched text...", "score": 12.34, "role": "assistant", "timestamp": "2026-08-20T12:34:56Z"} +] ``` -With `--fields full`, includes the complete message text alongside the snippet: +With `--fields full`, the array encodes `models.SearchResult` without JSON tags, so keys are emitted in the current Go field names (PascalCase), not snake_case: ```json -{"source_path": "...", "text": "full message content", "match_snippet": "...matched text...", "score": 12.34} +[ + { + "Source": "session", + "Role": "assistant", + "Content": "...matched text...", + "FilePath": "~/.claude/.../session.jsonl", + "Timestamp": "2026-08-20T12:34:56Z", + "SessionID": "", + "ProjectPath": "backscroll", + "Score": 12.34, + "Tags": null, + "ContentType": "text", + "Rank": 1 + } +] ``` +Current full-mode fields are exactly: `Source`, `Role`, `Content`, `FilePath`, `Timestamp`, `SessionID`, `ProjectPath`, `Score`, `Tags`, `ContentType`, and `Rank`. `ProjectPath` is a legacy field name; its value is the project identifier (for example `backscroll` or `myproj`), not a filesystem path. Only `--fields minimal` uses the snake_case payload (`source_path`, `snippet`, `score`, `role`, `timestamp`). + ### Robot -Compact tab-separated format designed for LLM consumption. Each line contains three fields separated by tabs: +Robot mode on search emits deterministic `result_N_field=value` lines: ``` -source_path\tscore\tsnippet +result_0_source=session +result_0_role=assistant +result_0_filepath=/home/user/.claude/projects/example/session.jsonl +result_0_content=matched content with escaped newlines +result_0_score=12.34 +result_0_rank=1 ``` -No ANSI escape codes. No headers. Minimal overhead — suitable for piping into context windows. +No ANSI escape codes. Search robot string values escape backslash as `\\`, carriage return as `\r`, and newline as `\n`, keeping each field on one line for context windows. ## Token Limiting @@ -71,6 +95,7 @@ The `--max-tokens` flag applies an approximate token limit (characters / 4) to t ```bash backscroll search "decisions" --robot --max-tokens 4000 +backscroll search --text "$QUERY" --source-path "$SOURCE_PATH" --robot --fields full --max-tokens 4000 ``` The limit is approximate — it will not truncate a result mid-output, but will stop before starting a result that would exceed the budget. diff --git a/docs/superpowers/plans/2026-08-20-mandatory-sync-db-only-retrieval.md b/docs/superpowers/plans/2026-08-20-mandatory-sync-db-only-retrieval.md new file mode 100644 index 0000000..d81869f --- /dev/null +++ b/docs/superpowers/plans/2026-08-20-mandatory-sync-db-only-retrieval.md @@ -0,0 +1,916 @@ +# Mandatory Sync and Database-Only Retrieval Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Restore SQLite as Backscroll's only public retrieval source and run one mandatory incremental sync before every operational command, with `recover` as the sole controlled continuation after startup failure. + +**Architecture:** Cobra's root `PersistentPreRunE` owns configuration validation, manifest preflight, compatibility preparation, and one incremental sync attempt. It stores the prepared configuration or typed startup failure in command context; handlers open the already-synchronized index without choosing freshness, while `recover` alone may consume a startup failure and perform a post-install sync. + +**Tech Stack:** Go 1.x, Cobra, modernc.org/sqlite, stdlib `testing`, existing `internal/config`, `internal/input_config`, `internal/storage`, and `internal/recovery` packages. + +**Spec:** `docs/superpowers/specs/2026-08-20-mandatory-sync-db-only-retrieval-design.md` + +## Global Constraints + +- SQLite is the only public retrieval source; transient files are ingestion inputs only. +- Every operational command attempts incremental startup sync exactly once before its handler runs. +- `recover` is the only command allowed to continue after startup failure. +- `--help` and `--version` must not load configuration, open SQLite, or sync inputs. +- Configuration, manifest, compatibility, discovery, parse, and sync failures fail closed without cached rows. +- JSON and robot stdout must remain parseable; human progress and warnings go to stderr. +- `backscroll read` and every `--indexed-only` flag are removed without aliases or deprecated no-op registrations. +- `search --source-path` remains the database-backed path lookup. +- Recovery must verify the installed canonical database and run a post-install sync before reporting success. +- No daemon, watcher, public `sync` command, snapshot bypass, FTS ranking change, or schema migration is introduced. +- Tests must scrub `HOME` and `BACKSCROLL_CONFIG_DIR`; the release gate is aggregate statement coverage >=85%. + +--- + +## File Structure + +- `cmd/backscroll/startup_policy.go`: root startup orchestration, injectable policy function, command-context state, typed failure rendering, recovery exception. +- `cmd/backscroll/startup_policy_test.go`: command-tree invariants, exactly-once ordering, fail-closed behavior, help/version side-effect checks, machine-output checks. +- `cmd/backscroll/main.go`: wire the root policy and remove `read` registration. +- `cmd/backscroll/index_policy.go`: retain compatibility/opening primitives but remove handler-selected `autoSync`. +- `cmd/backscroll/sync_helpers.go`: keep incremental ingestion implementation; accept an explicit progress writer instead of relying on freshness choices in handlers. +- `cmd/backscroll/{search,list,patterns,status,validate,rebuild,purge,annotate,config}.go`: consume startup configuration and open the synchronized index without syncing. +- `cmd/backscroll/recover.go`: consume startup failure context, execute recovery, then sync the installed database before success output. +- `cmd/backscroll/read.go`: delete. +- `internal/reader/`: delete after production-consumer check. +- `cmd/backscroll/*_test.go`: replace snapshot bypass fixtures with hermetic startup inputs and update function signatures. +- `README.md`, `CLAUDE.md`, `docs/{audit-integration,configuration,input-contract,patterns,read,sync}.md`, `.claude/skills/backscroll/{SKILL,ref-context-mode}.md`: publish the mandatory manifest -> sync -> SQLite -> query contract. +- Historical files under `docs/roadmap/` and older `docs/superpowers/` remain unchanged; ADR 0002 already records their supersession. + +--- + +### Task 1: Lock the Public CLI Invariants + +**Files:** +- Create: `cmd/backscroll/startup_policy_test.go` +- Modify: `cmd/backscroll/main.go` +- Delete: `cmd/backscroll/read.go` +- Delete: `internal/reader/reader.go` +- Delete: `internal/reader/semantic.go` +- Delete: `internal/reader/reader_test.go` +- Delete: `internal/reader/semantic_test.go` + +**Interfaces:** +- Consumes: `buildRootCmd(stdout, stderr io.Writer) *cobra.Command`. +- Produces: a root command tree with no `read` command and no flag named `indexed-only` anywhere. + +- [ ] **Step 1: Write failing command-tree invariant tests** + +```go +func TestRootExcludesDirectReadCommand(t *testing.T) { + root := buildRootCmd(io.Discard, io.Discard) + for _, cmd := range root.Commands() { + if cmd.Name() == "read" { + t.Fatal("public read command is registered") + } + } +} + +func TestCommandTreeExcludesIndexedOnlyFlag(t *testing.T) { + root := buildRootCmd(io.Discard, io.Discard) + var walk func(*cobra.Command) + walk = func(cmd *cobra.Command) { + if flag := cmd.Flags().Lookup("indexed-only"); flag != nil { + t.Errorf("%s registers forbidden --indexed-only", cmd.CommandPath()) + } + for _, child := range cmd.Commands() { + walk(child) + } + } + walk(root) +} +``` + +- [ ] **Step 2: Run the invariant tests and verify they fail** + +Run: `go test ./cmd/backscroll -run 'TestRootExcludesDirectReadCommand|TestCommandTreeExcludesIndexedOnlyFlag' -count=1` + +Expected: FAIL because `read` and `--indexed-only` are currently registered. + +- [ ] **Step 3: Remove `newReadCmd` from `root.AddCommand` and delete the direct-reader files** + +```go +root.AddCommand( + newSearchCmd(stdout, stderr), + newListCmd(stdout, stderr), + newPatternsCmd(stdout, stderr), + newRebuildCmd(stdout, stderr), + newPurgeCmd(stdout, stderr), + newValidateCmd(stdout, stderr), + newStatusCmd(stdout, stderr), + newConfigCmd(stdout, stderr), + newAnnotateCmd(stdout, stderr), + newRecoverCmd(stdout, stderr), +) +``` + +Run: `rm cmd/backscroll/read.go internal/reader/reader.go internal/reader/semantic.go internal/reader/reader_test.go internal/reader/semantic_test.go` + +- [ ] **Step 4: Remove all five public flag registrations and parameters** + +Remove `indexedOnly` variables, `BoolVar` calls, help text, and function parameters from `search.go`, `list.go`, `patterns.go`, `status.go`, and `validate.go`. The resulting calls must have these shapes: + +```go +return runSearch(stdout, stderr, query, project, allProjects, jsonFormat, robotFormat, + source, sourcePath, after, before, role, limit, offset, contentType, tag, + fields, maxTokens, lexicalOnly, similarityThreshold) + +return runList(stdout, stderr, project, allProjects, recent, jsonFormat, robotFormat, + order, limit, offset) + +return runPatterns(stdout, stderr, kind, project, allProjects, tag, limit, offset, + jsonFormat, robotFormat, minSupport, minConfidence, pending, batch, + minLength, maxLength, after, before, trend) + +return runStatus(stdout, stderr, jsonFormat) +return runValidate(stdout, stderr, jsonFormat) +``` + +- [ ] **Step 5: Run the invariant tests and compilation check** + +Run: `go test ./cmd/backscroll -run 'TestRootExcludesDirectReadCommand|TestCommandTreeExcludesIndexedOnlyFlag' -count=1` + +Expected: package compilation may still fail at stale test call sites, but neither failure may report a remaining production `read` or `indexed-only` registration. + +- [ ] **Step 6: Commit the public-surface removal** + +```bash +git add cmd/backscroll/main.go cmd/backscroll/search.go cmd/backscroll/list.go cmd/backscroll/patterns.go cmd/backscroll/status.go cmd/backscroll/validate.go cmd/backscroll/startup_policy_test.go +git add -u cmd/backscroll/read.go internal/reader +git commit -m "feat(cli): remove direct and stale retrieval bypasses" +``` + +--- + +### Task 2: Centralize Mandatory Startup Policy + +**Files:** +- Create: `cmd/backscroll/startup_policy.go` +- Modify: `cmd/backscroll/main.go` +- Modify: `cmd/backscroll/index_policy.go` +- Modify: `cmd/backscroll/sync_helpers.go` +- Test: `cmd/backscroll/startup_policy_test.go` +- Test: `cmd/backscroll/index_policy_test.go` + +**Interfaces:** +- Consumes: `config.Load()`, `config.ValidateNoLegacySources(string)`, `input_config.ActiveInputs([]string)`, `prepareIndex(context.Context, *config.Config, indexCommandClass)`, and `maybeAutoSync(*config.Config, io.Writer)`. +- Produces: + - `type startupPolicyFunc func(context.Context, io.Writer) startupResult` + - `type startupResult struct { Config *config.Config; Diagnostic *compat.Diagnostic; Err error }` + - `var startupSync = maybeAutoSync` + - `func defaultStartupPolicy(context.Context, io.Writer) startupResult` + - `func buildRootCmdWithStartup(io.Writer, io.Writer, startupPolicyFunc) *cobra.Command` + - `func startupResultFrom(*cobra.Command) startupResult` + - `func prepareIndex(context.Context, *config.Config, indexCommandClass) (*storage.Database, *compat.Diagnostic, error)` + - `func maybeAutoSync(*config.Config, io.Writer) error` + +- [ ] **Step 1: Write failing exactly-once and ordering tests using an injected policy** + +```go +func TestEveryOperationalCommandRunsStartupBeforeHandler(t *testing.T) { + commands := [][]string{ + {"search", "needle"}, {"list"}, {"patterns", "--kind", "commands"}, + {"annotate", "--uuid", "u", "--kind", "correction", "--label", "x"}, + {"purge", "--before", "2030-01-01"}, {"rebuild"}, {"status"}, + {"validate"}, {"config"}, {"recover", "--from", "missing.db", "--dry-run"}, + } + for _, argv := range commands { + t.Run(strings.Join(argv, "_"), func(t *testing.T) { + calls := 0 + policy := func(context.Context, io.Writer) startupResult { + calls++ + return startupResult{Config: &config.Config{DatabasePath: filepath.Join(t.TempDir(), "index.db")}} + } + root := buildRootCmdWithStartup(io.Discard, io.Discard, policy) + root.SetArgs(argv) + _ = root.Execute() + if calls != 1 { + t.Fatalf("startup calls=%d, want 1", calls) + } + }) + } +} +``` + +Add a focused synthetic command in the same test that appends `"startup"` inside the policy and `"handler"` inside `RunE`, then assert `[]string{"startup", "handler"}`. + +Also exercise the production policy with an injected sync seam: + +```go +func TestDefaultStartupPolicyCallsSyncExactlyOnce(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "index.db") + setIndexPolicyEnv(t, dbPath, t.TempDir()) + calls := 0 + originalSync := startupSync + startupSync = func(*config.Config, io.Writer) error { + calls++ + return nil + } + t.Cleanup(func() { startupSync = originalSync }) + + result := defaultStartupPolicy(context.Background(), io.Discard) + if result.Err != nil || result.Diagnostic != nil { + t.Fatalf("startup result=%+v", result) + } + if calls != 1 { + t.Fatalf("sync calls=%d, want 1", calls) + } +} +``` + +- [ ] **Step 2: Write failing help/version side-effect tests** + +```go +func TestMetadataCommandsSkipStartup(t *testing.T) { + for _, argv := range [][]string{{"--help"}, {"--version"}, {"search", "--help"}} { + calls := 0 + root := buildRootCmdWithStartup(io.Discard, io.Discard, func(context.Context, io.Writer) startupResult { + calls++ + return startupResult{} + }) + root.SetArgs(argv) + if err := root.Execute(); err != nil { + t.Fatalf("%v: %v", argv, err) + } + if calls != 0 { + t.Fatalf("%v invoked startup %d times", argv, calls) + } + } +} +``` + +- [ ] **Step 3: Run the startup tests and verify they fail** + +Run: `go test ./cmd/backscroll -run 'TestEveryOperationalCommandRunsStartupBeforeHandler|TestMetadataCommandsSkipStartup' -count=1` + +Expected: FAIL because `buildRootCmdWithStartup`, `startupResult`, and centralized orchestration do not exist. + +- [ ] **Step 4: Implement startup state and root wiring** + +```go +type startupPolicyFunc func(context.Context, io.Writer) startupResult + +type startupResult struct { + Config *config.Config + Diagnostic *compat.Diagnostic + Err error +} + +type startupContextKey struct{} + +func startupResultFrom(cmd *cobra.Command) startupResult { + result, _ := cmd.Context().Value(startupContextKey{}).(startupResult) + return result +} + +func buildRootCmd(stdout, stderr io.Writer) *cobra.Command { + return buildRootCmdWithStartup(stdout, stderr, defaultStartupPolicy) +} +``` + +In `buildRootCmdWithStartup`, set `PersistentPreRunE` to call the policy once, store its result with `cmd.SetContext(context.WithValue(cmd.Context(), startupContextKey{}, result))`, return nil on success, allow only `cmd.Name() == "recover"` to continue on failure, and otherwise render/return the typed failure before any handler output. + +- [ ] **Step 5: Implement default preflight ordering** + +```go +func defaultStartupPolicy(ctx context.Context, progress io.Writer) startupResult { + inputsDir, err := input_config.InputsDir() + if err != nil { + return startupResult{Err: fmt.Errorf("resolve inputs directory: %w", err)} + } + if err := config.ValidateNoLegacySources(inputsDir); err != nil { + return startupResult{Err: err} + } + cfg, err := config.Load() + if err != nil { + return startupResult{Err: fmt.Errorf("load config: %w", err)} + } + if _, _, err := input_config.ActiveInputs(cfg.SessionDirs); err != nil { + return startupResult{Config: cfg, Err: fmt.Errorf("validate active inputs: %w", err)} + } + db, diag, err := prepareIndex(ctx, cfg, indexMutation) + if db != nil { + err = closeIndexDB(db, err) + } + if diag != nil || err != nil { + return startupResult{Config: cfg, Diagnostic: diag, Err: err} + } + if err := startupSync(cfg, progress); err != nil { + activePath, _ := resolveActiveIndexPath(cfg.DatabasePath) + d := continuationFor(compat.Diagnostic{Code: compat.CodeIndexStale, Summary: fmt.Sprintf("index sync failed: %v", err)}, activePath) + return startupResult{Config: cfg, Diagnostic: &d, Err: err} + } + return startupResult{Config: cfg} +} +``` + +- [ ] **Step 6: Remove `autoSync` from `prepareIndex`** + +Change the signature to: + +```go +func prepareIndex(ctx context.Context, cfg *config.Config, class indexCommandClass) (*storage.Database, *compat.Diagnostic, error) +``` + +Delete the database-existence snapshot branch and the close/sync/reopen block. Map both `indexDataRead` and `indexMutation` to `storage.OpenCompatible`; retain immutable openings only for internal diagnostic/remediation inspection. + +- [ ] **Step 7: Make sync progress explicit** + +Change: + +```go +func maybeAutoSync(cfg *config.Config, progress io.Writer) (retErr error) +``` + +Replace both `fmt.Fprintf(maybeAutoSyncProgress, ...)` calls with `fmt.Fprintf(progress, ...)`. Keep storage/reader injection variables for deterministic failure tests, but remove the `maybeAutoSyncProgress` global. + +- [ ] **Step 8: Run the focused policy tests** + +Run: `go test ./cmd/backscroll -run 'TestEveryOperationalCommandRunsStartupBeforeHandler|TestMetadataCommandsSkipStartup|TestAutoSyncFailuresBlockCachedConsumers|TestResolveActiveIndexPathPropagatesBrokenSymlink' -count=1` + +Expected: PASS after adapting direct `prepareIndex`/`maybeAutoSync` test calls to their new signatures. + +- [ ] **Step 9: Commit centralized startup orchestration** + +```bash +git add cmd/backscroll/startup_policy.go cmd/backscroll/startup_policy_test.go cmd/backscroll/main.go cmd/backscroll/index_policy.go cmd/backscroll/index_policy_test.go cmd/backscroll/sync_helpers.go cmd/backscroll/main_test.go +git commit -m "feat(cli): enforce mandatory startup sync" +``` + +--- + +### Task 3: Make Handlers Consume the Synchronized Index + +**Files:** +- Modify: `cmd/backscroll/search.go` +- Modify: `cmd/backscroll/list.go` +- Modify: `cmd/backscroll/patterns.go` +- Modify: `cmd/backscroll/status.go` +- Modify: `cmd/backscroll/validate.go` +- Modify: `cmd/backscroll/rebuild.go` +- Modify: `cmd/backscroll/purge.go` +- Modify: `cmd/backscroll/annotate.go` +- Modify: `cmd/backscroll/config.go` +- Test: `cmd/backscroll/startup_policy_test.go` +- Test: `cmd/backscroll/index_policy_test.go` + +**Interfaces:** +- Consumes: `startupResultFrom(cmd).Config` and `prepareIndex(ctx, cfg, class)`. +- Produces: handlers that never call `maybeAutoSync`, never choose freshness, and execute only after root startup success. + +- [ ] **Step 1: Write a failing fail-closed handler test** + +```go +func TestStartupFailurePreventsHandlerOutput(t *testing.T) { + var stdout, stderr bytes.Buffer + policyErr := errors.New("injected startup failure") + root := buildRootCmdWithStartup(&stdout, &stderr, func(context.Context, io.Writer) startupResult { + return startupResult{Err: policyErr} + }) + root.SetArgs([]string{"config", "--json"}) + err := root.Execute() + if !errors.Is(err, policyErr) { + t.Fatalf("error=%v, want injected startup failure", err) + } + if stdout.Len() != 0 { + t.Fatalf("handler emitted output after startup failure: %q", stdout.String()) + } +} +``` + +- [ ] **Step 2: Run the test and verify it fails before handler rewiring is complete** + +Run: `go test ./cmd/backscroll -run TestStartupFailurePreventsHandlerOutput -count=1` + +Expected: FAIL if a handler remains reachable or writes output without consuming root policy. + +- [ ] **Step 3: Pass root-loaded configuration into every handler** + +For each command `RunE`, obtain the context value before calling its helper: + +```go +startup := startupResultFrom(cmd) +if startup.Config == nil { + return fmt.Errorf("startup configuration unavailable") +} +return runSearch(cmd.Context(), stdout, stderr, startup.Config, query, /* existing flags */) +``` + +Apply the equivalent signature prefix to all operational helpers: + +```go +func runSearch(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, /* flags */) error +func runList(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, /* flags */) error +func runPatterns(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, /* flags */) error +func runStatus(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, jsonFormat bool) error +func runValidate(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, jsonFormat bool) error +func runRebuild(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config) error +func runPurge(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, before string) error +func runAnnotate(ctx context.Context, stdout, stderr io.Writer, cfg *config.Config, uuid, path string, ordinal int, kind, label string) error +func runConfig(stdout, stderr io.Writer, cfg *config.Config, jsonFormat bool) error +``` + +Delete every handler-local `config.Load()` call. + +- [ ] **Step 4: Open without syncing in index-backed handlers** + +Use: + +```go +db, diag, err := prepareIndex(ctx, cfg, indexDataRead) +``` + +for `search`, `list`, and `patterns`; use `indexMutation` for `annotate`, `purge`, and `rebuild`; use `indexDiagnostic` for post-sync `status` and `validate`. No handler calls `maybeAutoSync`. + +Remove the stale-snapshot no-database special case from `list`, `status`, and `validate`: startup now creates/prepares the canonical database before these handlers run. + +- [ ] **Step 5: Remove rebuild's second sync claim** + +Change its long help from “runs an incremental sync” to “operates on the index synchronized at command startup.” Keep FTS rebuild, derived backfill, and project re-resolution unchanged. + +- [ ] **Step 6: Run command policy and focused handler tests** + +Run: `go test ./cmd/backscroll -run 'TestStartupFailurePreventsHandlerOutput|TestStaleIndexBlocksIndexBackedCommands|TestRebuildFailsOnDerivedMaintenanceError' -count=1` + +Expected: PASS; no cached row appears after startup failure and rebuild performs no second sync. + +- [ ] **Step 7: Commit handler simplification** + +```bash +git add cmd/backscroll/search.go cmd/backscroll/list.go cmd/backscroll/patterns.go cmd/backscroll/status.go cmd/backscroll/validate.go cmd/backscroll/rebuild.go cmd/backscroll/purge.go cmd/backscroll/annotate.go cmd/backscroll/config.go cmd/backscroll/startup_policy_test.go cmd/backscroll/index_policy_test.go +git commit -m "refactor(cli): consume root-synchronized index" +``` + +--- + +### Task 4: Implement Controlled Recovery Continuation + +**Files:** +- Modify: `cmd/backscroll/startup_policy.go` +- Modify: `cmd/backscroll/recover.go` +- Test: `cmd/backscroll/startup_policy_test.go` +- Test: `cmd/backscroll/recover_test.go` + +**Interfaces:** +- Consumes: `startupResultFrom(cmd)`, `recovery.Execute`, and `maybeAutoSync(cfg, stderr)`. +- Produces: `recover` continuation after startup failure, preservation of the original cause, and post-install sync before success output. + +- [ ] **Step 1: Write a failing controlled-continuation test** + +```go +func TestRecoverAloneContinuesAfterStartupFailure(t *testing.T) { + startupErr := errors.New("injected startup failure") + called := false + root := buildRootCmdWithStartup(io.Discard, io.Discard, func(context.Context, io.Writer) startupResult { + return startupResult{Config: &config.Config{DatabasePath: filepath.Join(t.TempDir(), "active.db")}, Err: startupErr} + }) + originalExecute := recoverExecute + recoverExecute = func(context.Context, recovery.Options) (recovery.Report, error) { + called = true + return recovery.Report{}, errors.New("injected recovery failure") + } + t.Cleanup(func() { recoverExecute = originalExecute }) + root.SetArgs([]string{"recover", "--from", "stranded.db"}) + err := root.Execute() + if !called { + t.Fatal("recover handler did not continue after startup failure") + } + if !errors.Is(err, startupErr) { + t.Fatalf("error=%v does not preserve startup failure", err) + } +} +``` + +Also add a table case proving `search`, `list`, and `config` do not continue under the same policy. + +- [ ] **Step 2: Write a failing post-install sync ordering test** + +Inject recovery and sync seams, append `"recover"`, `"sync"`, and `"report"` to a slice, and assert exactly: + +```go +want := []string{"recover", "sync", "report"} +``` + +The report marker is captured by a writer that records its first write. + +- [ ] **Step 3: Add deterministic recovery seams** + +```go +var recoverExecute = recovery.Execute +var recoverPostInstallSync = maybeAutoSync +``` + +Use `recoverExecute` in `newRecoverCmd`. Tests restore both variables with `t.Cleanup`. + +- [ ] **Step 4: Consume startup failure and join causes on failure** + +At the start of `RunE`: + +```go +startup := startupResultFrom(cmd) +cfg := startup.Config +if cfg == nil { + loaded, err := config.Load() + if err != nil { + return errors.Join(startup.Err, fmt.Errorf("load config for recovery: %w", err)) + } + cfg = loaded +} +``` + +If `recoverExecute` fails, return `errors.Join(startup.Err, fmt.Errorf("recovery failed: %w", err))`. If it succeeds, the prior startup failure is considered remediated and is not returned. + +- [ ] **Step 5: Sync the installed database before success output** + +```go +if !dryRun { + if err := recoverPostInstallSync(cfg, stderr); err != nil { + return errors.Join(startup.Err, fmt.Errorf("post-recovery sync: %w", err)) + } +} +printRecoveryReport(stdout, report, dryRun) +``` + +Dry runs do not replace the database and therefore do not run post-install sync. + +- [ ] **Step 6: Run recovery tests** + +Run: `go test ./cmd/backscroll -run 'TestRecoverAloneContinuesAfterStartupFailure|TestRecoverPostInstallSyncBeforeReport|TestRecover' -count=1` + +Expected: PASS, including existing durable backup and canonical union tests. + +- [ ] **Step 7: Commit recovery continuation** + +```bash +git add cmd/backscroll/startup_policy.go cmd/backscroll/startup_policy_test.go cmd/backscroll/recover.go cmd/backscroll/recover_test.go +git commit -m "fix(recovery): continue after failed startup sync" +``` + +--- + +### Task 5: Prove Manifest-Backed Markdown Retrieval and Machine Output + +**Files:** +- Modify: `cmd/backscroll/index_policy_test.go` +- Modify: `cmd/backscroll/diagnostics_test.go` +- Modify: `cmd/backscroll/markdown_registry_test.go` + +**Interfaces:** +- Consumes: `writeInputManifest`, `run`, Markdown readers, and `search --source-path`. +- Produces: integration evidence that Markdown is ingested through manifests and retrieved only from SQLite, with clean JSON/robot output. + +- [ ] **Step 1: Write a failing whole-document integration test** + +```go +func TestMandatoryStartupIndexesMarkdownDocumentForSearch(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "index.db") + root := filepath.Join(t.TempDir(), "notes") + setIndexPolicyEnv(t, dbPath, t.TempDir()) + writeInputManifest(t, root, "markdown_document", root, []string{"*.md"}, nil) + path := filepath.Join(root, "decision.md") + writeFile(t, path, "# Decision\n\nperennial sqlite sentinel\n") + + var stdout, stderr bytes.Buffer + err := run(&stdout, &stderr, []string{"search", "perennial sqlite sentinel", "--all-projects", "--source-path", path, "--json"}) + if err != nil { + t.Fatalf("search: %v stderr=%q", err, stderr.String()) + } + var rows []minimalSearchResult + if err := json.Unmarshal(stdout.Bytes(), &rows); err != nil { + t.Fatalf("invalid JSON %q: %v", stdout.String(), err) + } + if len(rows) != 1 || rows[0].SourcePath != path { + t.Fatalf("rows=%+v, want indexed markdown path %s", rows, path) + } +} +``` + +- [ ] **Step 2: Write a sectioned Markdown integration test** + +Use `format = "markdown_sections"`, a file containing two `##` sections, and search for a token unique to the second section: + +```go +func TestMandatoryStartupIndexesMarkdownSectionsForSearch(t *testing.T) { + dbPath := filepath.Join(t.TempDir(), "index.db") + root := filepath.Join(t.TempDir(), "notes") + setIndexPolicyEnv(t, dbPath, t.TempDir()) + writeInputManifest(t, root, "markdown_sections", root, []string{"*.md"}, nil) + path := filepath.Join(root, "decisions.md") + writeFile(t, path, "# Decisions\n\n## First\nalpha\n\n## Second\nsection sentinel omega\n") + + var stdout, stderr bytes.Buffer + err := run(&stdout, &stderr, []string{"search", "section sentinel omega", "--all-projects", "--source-path", path, "--json"}) + if err != nil { + t.Fatalf("search: %v stderr=%q", err, stderr.String()) + } + var rows []minimalSearchResult + if err := json.Unmarshal(stdout.Bytes(), &rows); err != nil { + t.Fatalf("invalid JSON %q: %v", stdout.String(), err) + } + if len(rows) != 1 || rows[0].SourcePath != path || !strings.Contains(rows[0].Snippet, "sentinel") { + t.Fatalf("rows=%+v, want second indexed section from %s", rows, path) + } +} +``` + +- [ ] **Step 3: Run Markdown tests and verify behavior** + +Run: `go test ./cmd/backscroll -run 'TestMandatoryStartupIndexesMarkdown' -count=1` + +Expected: PASS once startup policy invokes the existing Markdown registry before query handlers. + +- [ ] **Step 4: Update machine-output tests** + +For JSON, unmarshal stdout and assert no progress prose precedes the JSON. For robot mode, validate each result line and ensure a sync failure emits no cached rows: + +```go +var jsonRows []minimalSearchResult +if err := json.Unmarshal(stdout.Bytes(), &jsonRows); err != nil { + t.Fatalf("startup contaminated JSON stdout %q: %v", stdout.String(), err) +} +resultLine := regexp.MustCompile(`^result_[0-9]+_[a-z_]+=`) +for _, line := range strings.Split(strings.TrimSpace(robotStdout), "\n") { + if strings.HasPrefix(line, "result_") && !resultLine.MatchString(line) { + t.Fatalf("invalid robot line %q", line) + } +} +if strings.Contains(failedSyncStdout, "result_") { + t.Fatalf("sync failure emitted cached rows: %q", failedSyncStdout) +} +``` + +- [ ] **Step 5: Run focused output tests** + +Run: `go test ./cmd/backscroll -run 'TestMandatoryStartupIndexesMarkdown|TestMachineModes|TestAutoSyncFailuresBlockCachedConsumers' -count=1` + +Expected: PASS with progress/diagnostics routed according to output mode and no stale rows. + +- [ ] **Step 6: Commit integration evidence** + +```bash +git add cmd/backscroll/index_policy_test.go cmd/backscroll/diagnostics_test.go cmd/backscroll/markdown_registry_test.go +git commit -m "test(cli): prove database-backed markdown retrieval" +``` + +--- + +### Task 6: Convert Snapshot-Based Test Fixtures + +**Files:** +- Modify: `cmd/backscroll/main_test.go` +- Modify: `cmd/backscroll/diagnostics_test.go` +- Modify: `cmd/backscroll/compat_diagnostics_test.go` +- Modify: `cmd/backscroll/legacy_sources_test.go` +- Modify: `cmd/backscroll/list_coverage_test.go` +- Modify: `cmd/backscroll/patterns_coverage_test.go` +- Modify: `cmd/backscroll/patterns_sequences_test.go` +- Modify: `cmd/backscroll/recover_test.go` +- Modify: `cmd/backscroll/index_policy_test.go` + +**Interfaces:** +- Consumes: `setIndexPolicyEnv`, `writeInputManifest`, seeded SQLite helpers, and mandatory root startup. +- Produces: hermetic tests that never request stale snapshot behavior. + +- [ ] **Step 1: Remove forbidden flag arguments from test invocations** + +Run this guarded rewrite, then inspect the diff: + +```bash +python3 - <<'PY' +from pathlib import Path +for path in Path('cmd/backscroll').glob('*_test.go'): + text = path.read_text() + text = text.replace(', "--indexed-only"', '') + text = text.replace('"--indexed-only", ', '') + text = text.replace('"--indexed-only"', '') + path.write_text(text) +PY +gofmt -w cmd/backscroll/*_test.go +git diff -- cmd/backscroll +``` + +Delete obsolete tests whose sole requirement was accepting or bypassing with `--indexed-only`; do not merely rename them. + +- [ ] **Step 2: Update direct helper signatures** + +Change direct calls from: + +```go +maybeAutoSync(cfg) +prepareIndex(context.Background(), cfg, indexDataRead, true) +``` + +into: + +```go +maybeAutoSync(cfg, io.Discard) +prepareIndex(context.Background(), cfg, indexDataRead) +``` + +- [ ] **Step 3: Update legacy-source command tables** + +Remove the `read` entry and use `[]string{"validate"}`. Retain `recover` to prove legacy configuration is still rejected before side effects when no recovery continuation is possible. + +- [ ] **Step 4: Make seeded-index tests survive mandatory startup** + +For each test that seeds SQLite directly, call `setIndexPolicyEnv(t, dbPath, t.TempDir())` and ensure its active input root is empty or contains the intended fixture. This preserves seeded rows while allowing the mandatory sync attempt to succeed. + +- [ ] **Step 5: Run the command package** + +Run: `go test ./cmd/backscroll -count=1` + +Expected: PASS. Any failure mentioning machine-local inputs indicates a missing `HOME`/`BACKSCROLL_CONFIG_DIR` scrub; any missing seeded row indicates the fixture accidentally pointed startup sync at a destructive uuid-less source. + +- [ ] **Step 6: Assert forbidden production/test registrations are gone** + +Run: `rg -n --glob='*.go' -- '--indexed-only|newReadCmd|internal/reader' cmd internal` + +Expected: no matches except quoted assertions that verify the forbidden flag/command is absent. + +- [ ] **Step 7: Commit fixture migration** + +```bash +git add cmd/backscroll/*_test.go +git commit -m "test(cli): migrate fixtures to mandatory sync" +``` + +--- + +### Task 7: Align Living Documentation and Shipped Skill + +**Files:** +- Modify: `README.md` +- Modify: `CLAUDE.md` +- Modify: `docs/audit-integration.md` +- Modify: `docs/configuration.md` +- Modify: `docs/input-contract.md` +- Modify: `docs/patterns.md` +- Rewrite: `docs/read.md` +- Modify: `docs/sync.md` +- Modify: `.claude/skills/backscroll/SKILL.md` +- Modify: `.claude/skills/backscroll/ref-context-mode.md` +- Modify: `cmd/backscroll/skill_contract_test.go` + +**Interfaces:** +- Consumes: final Cobra tree and ADR 0002. +- Produces: one living contract: active manifests -> mandatory startup sync -> perennial SQLite -> database-backed query. + +- [ ] **Step 1: Update skill-contract assertions first** + +Replace stale anchors with: + +```go +anchors := []string{ + "Search discipline (hard rules)", + "Drill the top hit", + "artifact's vocabulary", + "failed invocation is a syntax problem", + "Two empty searches prove nothing", + "Raw-file boundary", + "--source-path", + "mandatory startup sync", + "backscroll validate", +} +``` + +Add assertions that the shipped skill contains neither `--indexed-only` nor an invocation beginning `backscroll read`. + +- [ ] **Step 2: Run contract tests and verify they fail against stale docs** + +Run: `go test ./cmd/backscroll -run 'TestBackscrollSkill|TestBackscrollLivingDocs' -count=1` + +Expected: FAIL with stale `read`/`--indexed-only` references and invalid removed flags. + +- [ ] **Step 3: Rewrite living guidance around the single data path** + +Use this canonical wording where each document explains freshness: + +```text +Every operational command validates active manifests and attempts one incremental +sync before executing. Session, plan, and Markdown files are ingestion inputs; +SQLite is the perennial record used by search, list, patterns, status, and validate. +Use search --source-path for database-backed retrieval scoped to a known input path. +``` + +`docs/read.md` becomes a migration page explaining removal of direct read, with examples using `backscroll search --source-path "*session-id*" --all-projects --json`. + +- [ ] **Step 4: Correct whole-document manifest examples** + +Replace living examples of: + +```toml +[inputs.decode] +format = "markdown" +``` + +with: + +```toml +[inputs.decode] +format = "markdown_document" +``` + +Keep `markdown_sections` examples unchanged. + +- [ ] **Step 5: Update repository structure documentation** + +In `CLAUDE.md`, remove `cmd/backscroll/read.go`, remove `internal/reader`, remove `read`/`--indexed-only` from the command inventory, change the count from eleven to ten commands, and remove the `internal/reader` package-layout row. Add the mandatory root startup-sync decision to Key Design Decisions. + +- [ ] **Step 6: Update the shipped skill and context reference** + +Replace snapshot probes with ordinary mandatory-sync searches. Preserve the rule against raw `cat`, `jq`, Python, or filesystem session hunting, but state that database-backed `search --source-path` is the supported drill-down path. Replace `backscroll validate --indexed-only` and `backscroll status --indexed-only` with their flag-free forms. + +- [ ] **Step 7: Run documentation contracts and stale-reference scan** + +Run: + +```bash +go test ./cmd/backscroll -run 'TestBackscrollSkill|TestBackscrollLivingDocs' -count=1 +rg -n --glob='*.md' --glob='*.toml' -- '(backscroll read|--indexed-only|decode\.format = "markdown"|format = "markdown")' README.md CLAUDE.md docs/audit-integration.md docs/configuration.md docs/input-contract.md docs/patterns.md docs/read.md docs/sync.md .claude/skills/backscroll +``` + +Expected: contract tests PASS and the scan returns no living guidance matches. Historical roadmap/spec/plan records are intentionally outside this scan. + +- [ ] **Step 8: Commit living documentation** + +```bash +git add README.md CLAUDE.md docs/audit-integration.md docs/configuration.md docs/input-contract.md docs/patterns.md docs/read.md docs/sync.md .claude/skills/backscroll/SKILL.md .claude/skills/backscroll/ref-context-mode.md cmd/backscroll/skill_contract_test.go +git commit -m "docs: require manifest sync and SQLite retrieval" +``` + +--- + +### Task 8: Run Full Verification and Update the PR + +**Files:** +- Verify: all modified files +- Existing ADR: `docs/adr/0002-exigir-sync-y-consulta-desde-sqlite.md` + +**Interfaces:** +- Consumes: Tasks 1-7. +- Produces: formatted, tested, coverage-compliant implementation pushed to PR #45. + +- [ ] **Step 1: Format and inspect repository state** + +Run: + +```bash +gofmt -w cmd/backscroll internal +git diff --check +git status --short +``` + +Expected: no formatting errors or whitespace diagnostics; only intended files are modified. + +- [ ] **Step 2: Run static checks** + +Run: `just check` + +Expected: PASS (`gofmt --check` and `go vet`). + +- [ ] **Step 3: Run the complete test suite** + +Run: `just test` + +Expected: PASS across all packages. + +- [ ] **Step 4: Run the release-equivalent CI gate** + +Run: `just ci` + +Expected: build succeeds, scrubbed-HOME tests pass, and aggregate statement coverage is >=85%. + +- [ ] **Step 5: Validate ADR records when Rootline is available** + +```bash +if command -v rootline >/dev/null 2>&1; then + rootline validate docs/adr/0001-declarar-frontera-documentacion-cli.md docs/adr/0002-exigir-sync-y-consulta-desde-sqlite.md --strict --output table +fi +``` + +Expected: both ADRs validate successfully. + +- [ ] **Step 6: Review the final diff against issue #44** + +Run: + +```bash +git diff origin/main...HEAD --stat +git log --oneline origin/main..HEAD +gh issue view 44 --repo pablontiv/backscroll --json body --jq .body +``` + +Check every acceptance checkbox against a test, code path, or living-doc update before claiming completion. + +- [ ] **Step 7: Push the implementation commits to PR #45** + +```bash +git push origin docs/issue-44-db-only-sync-design +gh pr view 45 --repo pablontiv/backscroll --json url,headRefName,statusCheckRollup +``` + +Expected: PR #45 points at the implementation branch and CI starts or reports success. diff --git a/docs/superpowers/specs/2026-08-20-mandatory-sync-db-only-retrieval-design.md b/docs/superpowers/specs/2026-08-20-mandatory-sync-db-only-retrieval-design.md new file mode 100644 index 0000000..b5fe5d8 --- /dev/null +++ b/docs/superpowers/specs/2026-08-20-mandatory-sync-db-only-retrieval-design.md @@ -0,0 +1,264 @@ +# Mandatory Sync and Database-Only Retrieval + +Date: 2026-08-20 +Status: **APPROVED DESIGN** +Issue: [#44](https://github.com/pablontiv/backscroll/issues/44) + +## Context + +Backscroll is the definitive local episodic memory for coding agents. Input files are transient ingestion sources; SQLite is the perennial record that survives source expiry and supplies every user-visible retrieval. + +That boundary was previously explicit and implemented. `docs/roadmap/T001-remove-public-read-command.md` identified `backscroll read` as a violation because it bypassed SQLite, and commit `f9c37eb` removed the command. The Go port later reintroduced it in `104c81e` while reconstructing the historical command inventory, then expanded it in `5080cd9` and `9d63c6f`. + +A second bypass exists through `--indexed-only`. `search`, `list`, and `patterns` can skip discovery and sync and query an existing snapshot. Other commands independently choose whether to sync through `prepareIndex(..., autoSync bool)`. This makes freshness a handler-level option rather than an application invariant. + +Issue #39 and PR #43 exposed the process failure. Their Cobra-backed documentation contract made living guides consistent with the registered CLI, but the CLI had already diverged from the approved North Star. Syntax consistency therefore legitimized an architectural regression. + +## Goals + +1. Restore SQLite as the only public retrieval source. +2. Attempt incremental startup sync exactly once before every operational command. +3. Remove every public stale-snapshot bypass. +4. Preserve one controlled recovery path after a failed mandatory startup attempt, followed by post-install sync. +5. Convert the North Star into executable CLI invariants. +6. Keep machine-readable stdout uncontaminated by sync progress or diagnostics. + +## Non-goals + +- No daemon, watcher, or background filesystem service. +- No public manual `sync` command. +- No direct file or UUID reader replacement. +- No alternate snapshot or stale-read flag. +- No FTS ranking, mining, or tokenizer changes. +- No rewriting of historical plans and research records. + +## Locked invariants + +### Database-only retrieval + +Every public retrieval reads SQLite. Files discovered through active manifests are ingestion inputs only. Plans and declarative Markdown documents follow the same path as sessions: + +```text +manifest -> discovery -> decode -> incremental sync -> SQLite -> query +``` + +`search --source-path` remains the DB-backed way to narrow retrieval to a known path, filename fragment, or session identifier embedded in `search_items.source_path`. + +### Mandatory startup sync + +Every operational subcommand attempts one incremental sync before its handler runs. No handler chooses freshness, and no public flag disables the attempt. + +```text +Cobra parses argv + | + v +root startup policy + 1. load app configuration + 2. reject legacy source configuration + 3. validate every active manifest + 4. prepare a compatible index + 5. run one startup sync attempt + | + v +requested command handler +``` + +Cobra help and version output are metadata operations. They do not execute a command body, open the database, or trigger sync. + +### Fail closed + +Configuration, manifest, compatibility, discovery, parsing, or sync failures stop normal commands. A command never falls back to cached rows and never emits partial query output after startup failure. + +### Controlled recovery + +`recover` is the sole remediation continuation after startup sync fails. The sync attempt still happens first. The root policy makes the original typed diagnostic available to recovery, then allows the recovery handler to inspect and replace the active database. + +A successful recovery must verify the canonical installed database and complete a post-install incremental sync before reporting success. This second attempt belongs to recovery after replacement; it does not repeat the failed startup attempt against the old database. Recovery is not a stale-read path and emits no indexed query results. + +## Architecture + +### Root startup policy + +The existing root `PersistentPreRunE` is the single orchestration boundary. It currently rejects legacy `[sources]`; the new policy composes that check with configuration loading, manifest preflight, index compatibility, and incremental sync. + +The policy must be injectable in tests so command-order behavior can be proven without relying on machine state or physical user inputs. + +Responsibilities: + +- identify whether a command body will execute; +- validate all ingestion configuration before database/file side effects that depend on it; +- prepare/migrate a compatible writable index; +- run incremental sync once; +- return a typed failure for normal commands; +- pass a failed-startup context to `recover` only. + +It does not execute query, formatting, annotation, purge, rebuild, validation, status, or recovery business logic. + +### Index preparation + +`prepareIndex` no longer receives `autoSync bool`. Freshness is established before handlers enter their command-specific paths. + +Read-only stale-snapshot openings remain available only as internal primitives when required for safe compatibility inspection or recovery verification. They are not selectable through a public command flag and cannot produce normal retrieval output. + +Command classes may remain where they express compatibility or mutation requirements, but they must not encode whether startup sync occurs. + +### Command behavior + +| Command | Behavior after successful startup sync | +|---|---| +| `search` | Query the synchronized index. | +| `list` | List sessions from the synchronized index. | +| `patterns` | Compute patterns from the synchronized perennial corpus. | +| `annotate` | Write an annotation after current inputs are ingested. | +| `purge` | Apply explicit retention deletion after current inputs are ingested. | +| `rebuild` | Re-derive FTS/satellites without performing a second sync. | +| `status` | Report the post-sync state. | +| `validate` | Validate the post-sync canonical state. | +| `config` | Print effective configuration only after startup validation and sync succeed. | +| `recover` | Run normally after successful sync; after failed sync, receive the failure and continue as the sole remediation path. | + +### Removed surfaces + +- Remove `newReadCmd` registration and `cmd/backscroll/read.go`. +- Delete `internal/reader` after confirming it has no production consumers. +- Remove `--indexed-only` from `search`, `list`, `patterns`, `status`, and `validate`. +- Remove indexed-only parameters, help text, branching, and test helpers. +- Do not retain hidden aliases or deprecated no-op flags. + +The removal is intentionally breaking. `backscroll read` becomes an unknown command and `--indexed-only` becomes an unknown flag. + +## Data flow and idempotency + +The mandatory sync reuses the existing SHA-256 incremental path: + +1. Load active manifests. +2. Discover source references. +3. Hash each source. +4. Skip unchanged sources unless an extraction-version backfill requires reparse. +5. Parse and synchronize changed sources transactionally. +6. Re-derive bounded stale derived data as currently defined. +7. Open the prepared index for the requested handler. + +Repeated command invocation is safe because unchanged inputs are hash-skipped, perennial UUID-bearing sessions use append-only identity, and derived backfills remain bounded and convergent. + +The root boundary owns the single startup sync attempt. `rebuild` and normal handlers must not call sync again. `recover` alone owns a post-install sync after replacing a database whose startup attempt failed. + +## Error and output contract + +### Human output + +- Sync progress and warnings go to stderr. +- A startup failure identifies its stage and responsible manifest/input/path. +- Normal command output begins only after successful sync. + +### Machine output + +- JSON and robot stdout remain valid and uncontaminated. +- Startup diagnostics preserve their machine-readable code, summary, and continuation argv. +- No partial result rows precede a startup failure. + +### Recovery failure + +If recovery also fails, report both the startup diagnostic and recovery failure without losing the primary cause. Do not install a partially verified database. + +## Documentation contract + +Living guidance expresses one operational route: + +```text +active manifests -> mandatory startup sync -> perennial SQLite -> search/list/patterns +``` + +Required updates include: + +- remove direct `backscroll read` guidance from README, `docs/read.md`, audit docs, and the shipped skill; +- remove all current `--indexed-only` guidance; +- describe `status` and `validate` as post-sync operations; +- keep `search --source-path` as indexed path lookup; +- use `decode.format = "markdown_document"` for whole-document Markdown; +- retain historical records but explicitly mark later decisions that preserved direct read or snapshot semantics as superseded by this design and ADR 0002. + +The living-doc Cobra contract remains useful for syntax drift, but Cobra is not the architectural source of truth. Dedicated invariant tests constrain the command tree itself. + +## Test strategy + +### Root policy + +- Every operational subcommand invokes startup sync exactly once. +- Startup sync completes before the handler begins. +- No handler runs after startup failure. +- Cached rows are not emitted after startup failure. +- `recover` alone can continue with the original failure context. +- A successful recovery verifies and synchronizes before success output. +- Help/version do not invoke startup sync. + +### Public invariants + +- Root command names exclude `read`. +- No root or child command registers `indexed-only`. +- `search --source-path` retrieves only indexed rows. +- Manifest-declared Markdown plans/documents are ingested and retrieved through SQLite. + +### Output + +- Human sync progress uses stderr. +- JSON/robot stdout stays parseable during success and failure. +- Startup failure produces no result payload before its diagnostic. + +### Regression and gates + +- Update tests that currently use `--indexed-only` as a fixture shortcut to use hermetic manifests and mandatory sync. +- Remove direct-reader tests with the deleted package. +- Update module/package layout documentation when `internal/reader` is removed. +- Run `just check`, `just test`, and `just ci`; aggregate statement coverage remains at least 85%. + +## Alternatives rejected + +### Per-command forced sync + +Changing every `prepareIndex` call to sync would be initially smaller but preserves distributed policy. New commands could omit the call, repeating the current regression. + +### Sync before Cobra execution + +Running sync in `run()` before argument parsing would affect help, version, invalid arguments, output-mode detection, and recovery routing. It conflates CLI parsing with operational startup. + +### Preserve direct read as a diagnostic + +A direct reader still creates a second public truth and has already been mistaken for normal retrieval. Files can be inspected with ordinary filesystem tools when explicitly diagnosing ingestion; Backscroll itself remains inside the indexed-evidence boundary. + +### Preserve snapshot mode + +`--indexed-only` intentionally violates mandatory freshness. Reproducible downstream snapshots require a separately designed/versioned API rather than a general CLI bypass. + +## Consequences + +### Positive + +- One enforceable freshness rule for every command. +- One public source of truth. +- New commands inherit sync automatically. +- Documentation cannot legitimize `read` merely because Cobra exposes it. +- Plans and sessions share the same ingestion/retrieval architecture. + +### Negative + +- Breaking removal of `read` and `--indexed-only`. +- Commands such as `config`, `status`, and `validate` now perform startup ingestion work. +- Existing audit consumers relying on snapshot mode must migrate. +- Test fixtures require broad updates because stale snapshots can no longer bypass manifests. + +### Accepted operational trade-off + +Startup cost is bounded by incremental hashing and unchanged-file skipping. Consistency with the episodic-memory contract takes precedence over a public stale-snapshot shortcut. + +## Success criteria + +The design is complete when: + +1. No public retrieval bypasses SQLite. +2. No operational command handler begins before one successful sync, except the controlled recovery continuation after a failed attempt. +3. Sync failure never returns cached query output. +4. Plans are demonstrably ingested and retrieved from the perennial database. +5. CLI invariant tests fail if `read` or `--indexed-only` reappears. +6. Living documentation states the same boundary. +7. All repository gates pass at aggregate coverage >=85%. diff --git a/docs/sync.md b/docs/sync.md index 3eaffc3..5e0bec2 100644 --- a/docs/sync.md +++ b/docs/sync.md @@ -3,7 +3,12 @@ estado: Completed --- # Sync and Indexing -Backscroll has no public `sync` command. Ingestion is integrated into query commands: active global input manifests are validated, changed inputs are detected by SHA-256, and only new or changed content is indexed. +Backscroll has no public `sync` command. Ingestion is integrated into operational commands: active global input manifests are validated, changed inputs are detected by SHA-256, and only new or changed content is indexed. + +Every operational command validates active manifests and attempts one incremental +sync before executing. Session, plan, and Markdown files are ingestion inputs; +SQLite is the perennial record used by search, list, patterns, status, and validate. +Use `--source-path` on search as a filter, paired with query text, for database-backed retrieval scoped to a known input path. ## Normal workflow @@ -11,37 +16,30 @@ Backscroll has no public `sync` command. Ingestion is integrated into query comm # Show the active manifests and resolved paths. backscroll config -# Each command incrementally syncs before reading. +# Commands perform startup sync before querying SQLite. backscroll search --text "migration plan" backscroll list --order timestamp:desc --limit 20 backscroll patterns --kind templates --min-support 5 +backscroll status --json +backscroll validate --json ``` -`search`, `list`, and `patterns` use auto-sync unless `--indexed-only` is supplied. Auto-sync writes progress and warnings to stderr so JSON and robot stdout remain machine-readable. Invalid active manifests fail during preflight instead of being silently ignored. - -`backscroll status` and `backscroll validate` are diagnostic, always read-only, and never auto-sync. Their legacy `--indexed-only` flags are accepted as deprecated no-ops. - -## Indexed-only snapshots +Human startup sync writes progress and warnings to stderr. JSON/robot startup progress is discarded so stdout remains machine-readable, and invalid active manifests fail during preflight instead of being silently ignored. -Use `--indexed-only` when a consumer needs to query one existing SQLite snapshot without file discovery or writes: +A search scoped to a known input path stays database-backed: ```bash -backscroll list --indexed-only --json --all-projects --limit 100 -backscroll search --text "permission denied" --indexed-only --all-projects --json -backscroll patterns --kind failures --indexed-only --all-projects --json -backscroll status --json -backscroll validate --json +backscroll search --text "artifact literal" --source-path "*session-id*" --all-projects --json +backscroll search --text "permission denied" --source-path "*/example/*.jsonl" --all-projects --json ``` -A search still requires a non-empty query. `list` returns session summaries rather than every stored message; see [Downstream audit integration](audit-integration.md) for the supported snapshot boundary. - ## Rebuild semantics ```bash backscroll rebuild ``` -`rebuild` is non-destructive. It runs incremental ingestion through the normal index preparation path, re-derives both FTS5 indexes from the perennial `search_items` table, backfills derived templates/corrections/tool events from stored text where possible, and re-resolves project identities. It does not discard sessions whose files have expired. +`rebuild` is non-destructive. The mandatory root startup sync runs first and prepares the database. The rebuild handler does not perform a second sync: it re-derives both FTS5 indexes from the perennial `search_items` table, backfills derived templates/corrections/tool events from stored text where possible, and re-resolves project identities. It does not discard sessions whose files have expired. Use `rebuild` after index-recovery work or when derived search structures need regeneration. It is not a substitute for a removed manual sync command. `backscroll purge --before ` is the explicit deletion path. @@ -81,14 +79,18 @@ role = "$.message.role" selector = "$.message.content" ``` -Plans and external Markdown documents are also declared as inputs. Use `decode.format = "markdown"` for a whole document or `decode.format = "markdown_sections"` to split on `## ` headings. See [Generic input manifest contract](input-contract.md) for the complete schema. +Plans and external Markdown documents are also declared as inputs. Use `decode.format = "markdown_document"` for a whole document or `decode.format = "markdown_sections"` to split on `## ` headings. See [Generic input manifest contract](input-contract.md) for the complete schema. ## Incremental and perennial behavior -Backscroll stores a SHA-256 hash for each indexed input. Unchanged files are skipped on later auto-syncs. Files with stable message UUIDs are updated append-only; legacy or UUID-less inputs retain wipe-and-reload behavior while the source exists. +Backscroll stores a SHA-256 hash for each indexed input. Unchanged files are skipped on later startup syncs. Files with stable message UUIDs are updated append-only; legacy or UUID-less inputs retain wipe-and-reload behavior while the source exists. The SQLite database is the perennial event store, not a disposable cache. When a source file expires, its indexed rows remain available. Only `purge` removes retained data explicitly. +## Machine output + +`--json` writes one JSON payload to stdout. JSON and robot startup progress is discarded so stdout stays parseable; human progress and warnings use stderr. Structured diagnostics remain parseable in machine modes. `--robot` output shape is command-specific: `backscroll list` and `backscroll patterns` include command-defined sections, while only `backscroll search "" --robot` guarantees deterministic `result_N_field=value` lines. Search robot string values escape backslash as `\\`, carriage return as `\r`, and newline as `\n` so each field remains one line. + ## Noise filtering Text cleanup and record inclusion are defined in each manifest. The shipped presets remove provider noise such as system reminders, task notifications, local command metadata, and configured subagent paths. Empty messages are dropped when `drop_empty = true`. diff --git a/internal/reader/reader.go b/internal/reader/reader.go deleted file mode 100644 index 66d3672..0000000 --- a/internal/reader/reader.go +++ /dev/null @@ -1,12 +0,0 @@ -package reader - -import ( - "github.com/pablontiv/backscroll/internal/models" - internalsync "github.com/pablontiv/backscroll/internal/sync" -) - -// ReadFile reads a session JSONL file and returns its messages. -// Applies the same noise filtering and content extraction as the sync pipeline. -func ReadFile(path string) ([]models.Message, error) { - return internalsync.ParseSessions(path) -} diff --git a/internal/reader/reader_test.go b/internal/reader/reader_test.go deleted file mode 100644 index ec65cb1..0000000 --- a/internal/reader/reader_test.go +++ /dev/null @@ -1,31 +0,0 @@ -package reader_test - -import ( - "testing" - - "github.com/pablontiv/backscroll/internal/reader" -) - -const piFixture = "../../tests/fixtures/pi-session.jsonl" - -func TestReadFile(t *testing.T) { - msgs, err := reader.ReadFile(piFixture) - if err != nil { - t.Fatalf("ReadFile: %v", err) - } - if len(msgs) == 0 { - t.Error("expected at least one message, got zero") - } - for _, m := range msgs { - if m.Role == "" { - t.Errorf("message has empty role: %+v", m) - } - } -} - -func TestReadFileNotExist(t *testing.T) { - _, err := reader.ReadFile("/nonexistent/path/session.jsonl") - if err == nil { - t.Error("expected error for nonexistent file, got nil") - } -} diff --git a/internal/reader/semantic.go b/internal/reader/semantic.go deleted file mode 100644 index 4bc59b6..0000000 --- a/internal/reader/semantic.go +++ /dev/null @@ -1,152 +0,0 @@ -package reader - -import ( - "encoding/json" - "fmt" - "strings" - - internalsync "github.com/pablontiv/backscroll/internal/sync" -) - -const semanticSnippetLimit = 500 - -// SemanticRow is a concise, agent-readable row extracted from a JSONL input file. -type SemanticRow struct { - Path, Timestamp, Role, Kind, Content string - Line, Ordinal int -} - -// ReadSemanticTail returns the last tail semantic rows from path. -func ReadSemanticTail(path string, tail int) ([]SemanticRow, error) { - if tail < 0 { - return nil, fmt.Errorf("tail must be non-negative") - } - var all, ring []SemanticRow - if tail > 0 { - ring = make([]SemanticRow, 0, tail) - } - seen := 0 - err := internalsync.IterateJSONLFile(path, func(lineNumber int, line []byte) error { - for _, row := range semanticRowsFromLine(path, lineNumber, line) { - seen++ - row.Ordinal = seen - if tail <= 0 { - all = append(all, row) - } else if len(ring) < tail { - ring = append(ring, row) - } else { - ring[(seen-1)%tail] = row - } - } - return nil - }) - if err != nil || tail <= 0 || len(ring) < tail { - return append(all, ring...), err - } - rows := make([]SemanticRow, 0, len(ring)) - for i, start := 0, seen%tail; i < len(ring); i++ { - rows = append(rows, ring[(start+i)%tail]) - } - return rows, nil -} - -func semanticRowsFromLine(path string, lineNumber int, line []byte) []SemanticRow { - var rec map[string]any - if err := json.Unmarshal(line, &rec); err != nil { - return nil - } - msg, ok := rec["message"].(map[string]any) - if !ok { - return nil - } - base := SemanticRow{Path: path, Line: lineNumber, Timestamp: stringField(rec, "timestamp"), Role: stringField(msg, "role")} - if text, ok := msg["content"].(string); ok { - return semanticTextRow(base, text, semanticTextKind(base)) - } - blocks, ok := msg["content"].([]any) - if !ok { - return nil - } - rows := make([]SemanticRow, 0, len(blocks)) - for _, rawBlock := range blocks { - block, ok := rawBlock.(map[string]any) - if !ok { - continue - } - row := base - switch stringField(block, "type") { - case "text": - row.Kind, row.Content = semanticTextKind(base), truncateSnippet(stringField(block, "text")) - case "tool_use", "toolCall": - row.Kind = "tool_use" - row.Content = truncateSnippet(toolSnippet(stringField(block, "name"), stringField(block, "id"), jsonField(block, "input", "arguments", "toolCall"))) - case "tool_result", "toolResult": - row.Kind = "tool_result" - row.Content = truncateSnippet(toolSnippet("", firstNonEmpty(stringField(block, "tool_use_id"), stringField(block, "toolCallId")), jsonField(block, "content", "toolResult"))) - } - if row.Kind != "" && row.Content != "" { - rows = append(rows, row) - } - } - return rows -} - -func semanticTextRow(base SemanticRow, text string, kind string) []SemanticRow { - if text = truncateSnippet(text); text == "" { - return nil - } - base.Kind, base.Content = kind, text - return []SemanticRow{base} -} - -func semanticTextKind(base SemanticRow) string { - if base.Role == "toolResult" { - return "tool_result" - } - return "text" -} - -func stringField(m map[string]any, key string) string { - if value, ok := m[key].(string); ok { - return value - } - return "" -} - -func jsonField(m map[string]any, keys ...string) string { - for _, key := range keys { - if value, ok := m[key]; ok && value != nil { - if encoded, err := json.Marshal(value); err == nil && string(encoded) != "null" { - return string(encoded) - } - } - } - return "" -} - -func firstNonEmpty(values ...string) string { - for _, value := range values { - if value != "" { - return value - } - } - return "" -} - -func toolSnippet(name string, id string, payload string) string { - parts := make([]string, 0, 3) - for _, part := range []string{"name=" + name, "id=" + id, "payload=" + payload} { - if !strings.HasSuffix(part, "=") { - parts = append(parts, part) - } - } - return strings.Join(parts, " ") -} - -func truncateSnippet(content string) string { - content = strings.Join(strings.Fields(content), " ") - if len(content) <= semanticSnippetLimit { - return content - } - return content[:semanticSnippetLimit] + "…" -} diff --git a/internal/reader/semantic_test.go b/internal/reader/semantic_test.go deleted file mode 100644 index 06d817f..0000000 --- a/internal/reader/semantic_test.go +++ /dev/null @@ -1,143 +0,0 @@ -package reader - -import ( - "strings" - "testing" -) - -func TestReadSemanticTailIncludesPiToolCallArguments(t *testing.T) { - rows, err := ReadSemanticTail("../../tests/fixtures/pi-session.jsonl", 0) - if err != nil { - t.Fatalf("ReadSemanticTail: %v", err) - } - - for _, row := range rows { - if row.Kind == "tool_use" && strings.Contains(row.Content, "name=read") { - if !strings.Contains(row.Content, `"path":"secret"`) { - t.Fatalf("tool_use row missing Pi arguments payload: %#v", row) - } - return - } - } - t.Fatalf("missing Pi tool_use row in %#v", rows) -} - -func TestReadSemanticTailClassifiesPiToolResultRole(t *testing.T) { - rows, err := ReadSemanticTail("../../tests/fixtures/pi-session.jsonl", 0) - if err != nil { - t.Fatalf("ReadSemanticTail: %v", err) - } - - for _, row := range rows { - if row.Role == "toolResult" { - if row.Kind != "tool_result" { - t.Fatalf("toolResult role row kind = %q, want tool_result: %#v", row.Kind, row) - } - if !strings.Contains(row.Content, "tool result should not index") { - t.Fatalf("toolResult role row missing content snippet: %#v", row) - } - return - } - } - t.Fatalf("missing Pi toolResult role row in %#v", rows) -} - -func TestSemanticTextRow(t *testing.T) { - // Test that semanticTextRow creates a row when text is present - base := SemanticRow{Path: "test.jsonl", Line: 1, Timestamp: "2024-01-01T00:00:00Z", Role: "user"} - rows := semanticTextRow(base, "hello world", "text") - if len(rows) != 1 { - t.Errorf("expected 1 row, got %d", len(rows)) - } - if rows[0].Content != "hello world" { - t.Errorf("expected content 'hello world', got %q", rows[0].Content) - } - - // Test that semanticTextRow returns nil for empty text - rows = semanticTextRow(base, "", "text") - if len(rows) != 0 { - t.Errorf("expected 0 rows for empty text, got %d", len(rows)) - } - - // Test truncation of long text - longText := "word " + strings.Repeat("x", 600) - rows = semanticTextRow(base, longText, "text") - if len(rows) != 1 { - t.Errorf("expected 1 row for long text, got %d", len(rows)) - } - if !strings.HasSuffix(rows[0].Content, "…") { - t.Errorf("expected truncated content to end with …, got %q", rows[0].Content) - } - // The length after Fields() normalization varies, just check it ends with … - // and is roughly the right size (500 + 3 for "…" encoding) - if len(rows[0].Content) < 500 { - t.Errorf("expected long truncated content, got %d", len(rows[0].Content)) - } -} - -func TestFirstNonEmpty(t *testing.T) { - // Test with all non-empty - result := firstNonEmpty("a", "b", "c") - if result != "a" { - t.Errorf("expected 'a', got %q", result) - } - - // Test with empty first value - result = firstNonEmpty("", "b", "c") - if result != "b" { - t.Errorf("expected 'b', got %q", result) - } - - // Test with all empty - result = firstNonEmpty("", "", "") - if result != "" { - t.Errorf("expected empty string, got %q", result) - } - - // Test with single value - result = firstNonEmpty("value") - if result != "value" { - t.Errorf("expected 'value', got %q", result) - } -} - -func TestReadSemanticTailWithLimit(t *testing.T) { - // Test with non-zero tail limit (ringbuffer logic) - rows, err := ReadSemanticTail("../../tests/fixtures/pi-session.jsonl", 2) - if err != nil { - t.Fatalf("ReadSemanticTail with tail=2: %v", err) - } - // Should return last 2 rows - if len(rows) > 2 { - t.Errorf("expected at most 2 rows, got %d", len(rows)) - } -} - -func TestJsonField(t *testing.T) { - // Test finding nested JSON values - m := map[string]any{ - "input": map[string]any{ - "arguments": map[string]any{ - "toolCall": "test_value", - }, - }, - } - - result := jsonField(m, "input", "arguments", "toolCall") - if !strings.Contains(result, "test_value") { - t.Errorf("expected JSON containing 'test_value', got %q", result) - } - - // Test with missing key - result = jsonField(m, "missing", "key") - if result != "" { - t.Errorf("expected empty string for missing key, got %q", result) - } - - // Test with null value - m2 := map[string]any{"key": nil} - result = jsonField(m2, "key") - if result != "" { - t.Errorf("expected empty string for null value, got %q", result) - } -}