fix(compat): whitespace-insensitive schema lineage (#52)

CLAUDE.md

@@ -118,6 +118,7 @@ External knowledge sources are configured with active `*.inputs.toml` manifests
 - **Content-type classification**: Messages classified as `text`/`code`/`tool`/`reasoning` based on message content types during sync. Tool content is indexed in separate `search_items` rows with `content_type='tool'`. Pi agent reasoning blocks are captured when `index_reasoning=true` (default off) in the input manifest and indexed with `content_type='reasoning'`. Sync writes only to `search_items`; the `session_events` table was dropped in migration v5.
 - **Split FTS by retrieval semantics**: tool content (`content_type='tool'`) lives in a separate FTS5 index `tool_fts` (tokenizer `trigram`, substring/exact match for paths/commands/errors); prose content (text, code, reasoning) lives in `messages_fts` (`porter unicode61`). Migration v4 branched the triggers by content type. Migration v7 updated the triggers to route 'reasoning' alongside 'text'/'code' to `messages_fts`. `--content-type tool` queries `tool_fts`; prose queries `messages_fts`; an unfiltered query merges both via Reciprocal Rank Fusion (RRF, k=60), which fuses by rank position, not score magnitude, and is immune to incomparable cross-tokenizer BM25 scales. The trigram tokenizer matches substrings of ≥3 characters, so tool queries shorter than 3 characters will match zero results.
 - **Pure Go SQLite**: `modernc.org/sqlite` — no CGO, trivially cross-compilable.
+- **Whitespace-insensitive schema normalization (fix #52 part 1)**: `normalizeSQL()` collapses runs of whitespace outside SQL string literals, preserving whitespace inside single-quoted literals and handling `''` escapes correctly. This eliminates cosmetic DDL formatting (e.g., `ALTER TABLE ADD COLUMN` inline vs. hand-wrapped multi-line) as a source of signature mismatch. All 76 manifest signatures are regenerated via `RegenerateManifestJSON` under the new normalization (reproducible: load fixtures, compute signatures under current `normalizeSQL`, write back manifest.json). The fix prevents false `unsupported_lineage` rejections when published-release-migrated databases encounter the catalog.
 - **Startup coordination**: `github.com/gofrs/flock` via `internal/startuplock` — persistent `<db>.startup-sync.lock` sidecar with `0600` permissions, OS-owned advisory lock, local-host-only WAL snapshot coordination, and read-only followers that never delete the sidecar.
 - **Connection pragmas via `_pragma`**: `modernc.org/sqlite` honors DSN pragmas only in the `_pragma=name(value)` form; the mattn-style `_name=value` (e.g. `_busy_timeout=5000`, `_journal_mode=WAL`) is silently ignored, which had left the DB in rollback (delete) journal mode with a zero busy timeout — the root cause of `database is locked` (SQLITE_BUSY) errors. Both connections in `internal/storage/storage.go` set `_pragma=journal_mode(WAL)`, `_pragma=synchronous(NORMAL)`, and `_pragma=busy_timeout(5000)` (read-only sets only the busy timeout; journal mode is persisted in the file). Always use `_pragma=name(value)` for any new connection pragma here.
 - **Autoupdate**: `picokit/autoupdate` fetches and stages the latest GitHub release in the background; `run()` waits up to 10s after the command completes so short-lived commands don't kill the download before it finishes. Autoupdate is mandatory: there is no runtime opt-out. `newUpdater()` in `cmd/backscroll/main.go` is the single wiring point — it calls `autoupdate.New` with no `envDisable`, and both `run()` and the wiring test go through it, so re-adding an opt-out would fail the test. Dev builds are exempt by identity — a plain `go build` yields `version="dev"`, which picokit never fetches or applies — so validate against a dev build, not an env var. `scripts/eval.sh` builds its own dev binary for the same reason (a release binary would fetch+wait ~10s per invocation).
@@ -148,6 +149,7 @@ External knowledge sources are configured with active `*.inputs.toml` manifests
 - **Search robot output contract**: robot mode on search emits `result_N_field=value` lines exactly once-wrapped (the robot path writes lines directly; passing pre-formatted lines through the picokit formatter double-wraps them as `result_N=result_N_field=...`). Search robot string values escape backslash as `\\`, carriage return as `\r`, and newline as `\n`.
 - **Cross-host project identity**: `projects.Identify()` normalizes session cwd against registry roots by matching root tails (≥2 components, case-insensitive), so `/home/shared/<proj>` sessions resolve against `/Users/Shared/<proj>` roots on a synced index. Registry roots should keep distinct suffixes — two projects whose roots share the same trailing components could misbucket.
 - **Recall eval-set**: `docs/eval/queries.toml` (~20 real mined queries with `expected_match` ground truth) + `scripts/eval.sh` compute recall@5; a query counts only if its expected target appears in the top 5. Local regression gate, not a required CI step.
+- **Single catalog source of truth (fix #52)**: The lineage catalog lives in `internal/compat/manifest.json` (data, regenerable via `REGEN_MANIFEST=1`). `internal/storage/recovery_records.go` queries `compat.Catalog.IsKnownSignature()` to check if a schema is recognized, rather than carrying a stale hardcoded switch. This eliminates duplication and the risk of signature drift after normalization changes. Whitespace normalization collapsed 76 fixtures into 17 distinct signatures; collision consistency is verified by a test that ensures all entries with the same signature agree on `AppliedVersion` and `HasSourceMetadata`.
 
 ## Dependencies
 

internal/compat/catalog.go

@@ -52,6 +52,14 @@ func (c Catalog) BySignature(signature string) (Lineage, bool) {
 	return lineage, ok
 }
 
+// IsKnownSignature returns true if the signature is in the lineage catalog.
+// Use this to check if a schema is recognized, independent of its migration
+// status or other semantic properties.
+func (c Catalog) IsKnownSignature(signature string) bool {
+	_, ok := c.lineages[signature]
+	return ok
+}
+
 func (c Catalog) CurrentSignature() string {
 	return c.currentSignature
 }

internal/compat/catalog_test.go

@@ -6,6 +6,7 @@ import (
 	"database/sql"
 	"fmt"
 	"io/fs"
+	"os"
 	"path/filepath"
 	"reflect"
 	"sort"
@@ -373,3 +374,76 @@ func loadFixtureMigrationRows(t *testing.T, fixtureSQL []byte) []migrationRow {
 	}
 	return result
 }
+
+// TestCollisionConsistencyGuardsAgainstSignatureAmbiguity verifies that when
+// whitespace normalization collapses multiple fixtures into the same signature,
+// all colliding entries agree on AppliedVersion and HasSourceMetadata. If any
+// collision group disagrees, the Catalog.BySignature map's winner is arbitrary
+// and recovery could plan the wrong migration steps — a real bug.
+func TestCollisionConsistencyGuardsAgainstSignatureAmbiguity(t *testing.T) {
+	catalog, err := LoadCatalog()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// Build collision groups: signature -> list of (AppliedVersion, HasSourceMetadata)
+	collisions := make(map[string][]struct {
+		source      string
+		version     int
+		hasMetaData bool
+	})
+
+	for _, release := range catalog.Releases {
+		collisions[release.Signature] = append(collisions[release.Signature], struct {
+			source      string
+			version     int
+			hasMetaData bool
+		}{fmt.Sprintf("release %s", release.Tag), release.AppliedVersion, release.HasSourceMetadata})
+	}
+
+	for _, fixture := range catalog.UnmanifestedFixtures {
+		collisions[fixture.Signature] = append(collisions[fixture.Signature], struct {
+			source      string
+			version     int
+			hasMetaData bool
+		}{fmt.Sprintf("unmanifested %s", fixture.Fixture), fixture.AppliedVersion, fixture.HasSourceMetadata})
+	}
+
+	// Check each collision group for consistency
+	for sig, entries := range collisions {
+		if len(entries) <= 1 {
+			// No collision; skip
+			continue
+		}
+
+		// All entries in this collision must agree on AppliedVersion and HasSourceMetadata
+		first := entries[0]
+		for i, entry := range entries[1:] {
+			if entry.version != first.version {
+				t.Errorf("signature %s has inconsistent AppliedVersion: %s says %d, %s says %d",
+					sig, first.source, first.version, entry.source, entry.version)
+			}
+			if entry.hasMetaData != first.hasMetaData {
+				t.Errorf("signature %s has inconsistent HasSourceMetadata: %s says %v, %s says %v",
+					sig, first.source, first.hasMetaData, entry.source, entry.hasMetaData)
+			}
+			if i == 0 {
+				t.Logf("collision group %s: %s, %s agree", sig[:16], first.source, entry.source)
+			}
+		}
+	}
+}
+
+// TestRegenerateManifestOnNormalizationChange is an optional helper test that
+// regenerates manifest.json after normalizeSQL changes. Run with:
+// go test -run TestRegenerateManifestOnNormalizationChange ./internal/compat -v
+func TestRegenerateManifestOnNormalizationChange(t *testing.T) {
+	if os.Getenv("REGEN_MANIFEST") == "" {
+		t.Skip("Set REGEN_MANIFEST=1 to regenerate manifest.json")
+	}
+
+	manifestPath := filepath.Join("testdata", "release-schemas", "manifest.json")
+	if err := RegenerateManifestJSON(manifestPath); err != nil {
+		t.Fatalf("regenerate manifest: %v", err)
+	}
+}

internal/compat/regenerate_manifest.go

@@ -0,0 +1,153 @@
+package compat
+
+import (
+	"context"
+	"crypto/sha256"
+	"database/sql"
+	"encoding/json"
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+)
+
+// RegenerateManifestJSON reads all fixtures and regenerates the manifest.json with
+// new signatures computed using the current normalizeSQL implementation.
+// This is called after changing normalizeSQL to ensure all signatures remain valid.
+func RegenerateManifestJSON(manifestPath string) error {
+	// Load the old manifest to preserve the release mappings and unmapped fixtures
+	oldCatalog, err := loadCatalogFromPath(manifestPath)
+	if err != nil {
+		return fmt.Errorf("load old manifest: %w", err)
+	}
+
+	// Scan the testdata/release-schemas directory for all .sql files
+	fixturesDir := filepath.Dir(manifestPath)
+
+	// Build a map of fixture -> new signature
+	fixtureSignatures := make(map[string]string)
+	fixtureProvenances := make(map[string]string)
+
+	entries, err := os.ReadDir(fixturesDir)
+	if err != nil {
+		return fmt.Errorf("read fixtures directory: %w", err)
+	}
+
+	for _, entry := range entries {
+		if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".sql") {
+			continue
+		}
+
+		fixturePath := filepath.Join(fixturesDir, entry.Name())
+		fixtureData, err := os.ReadFile(fixturePath)
+		if err != nil {
+			return fmt.Errorf("read fixture %s: %w", entry.Name(), err)
+		}
+
+		// Compute provenance SHA256
+		provenance := fmt.Sprintf("%x", sha256.Sum256(fixtureData))
+
+		// Open and inspect the fixture
+		db, err := sql.Open("sqlite", ":memory:")
+		if err != nil {
+			return fmt.Errorf("open sqlite for fixture %s: %w", entry.Name(), err)
+		}
+		if _, err := db.Exec(string(fixtureData)); err != nil {
+			db.Close()
+			return fmt.Errorf("execute fixture %s: %w", entry.Name(), err)
+		}
+
+		shape, err := inspectShape(context.Background(), db)
+		db.Close()
+		if err != nil {
+			return fmt.Errorf("inspect fixture %s: %w", entry.Name(), err)
+		}
+
+		fixtureSignatures[entry.Name()] = shape.Signature
+		fixtureProvenances[entry.Name()] = provenance
+	}
+
+	// Update the catalog with new signatures
+	newCatalog := Catalog{
+		FirstGoRelease:       oldCatalog.FirstGoRelease,
+		LatestGoRelease:      oldCatalog.LatestGoRelease,
+		Releases:             []catalogRelease{},
+		UnmanifestedFixtures: []catalogFixture{},
+	}
+
+	// Update manifested releases with new signatures
+	for _, oldRelease := range oldCatalog.Releases {
+		newSig, ok := fixtureSignatures[oldRelease.Fixture]
+		if !ok {
+			return fmt.Errorf("fixture not found for release %s: %s", oldRelease.Tag, oldRelease.Fixture)
+		}
+		newProvenance, ok := fixtureProvenances[oldRelease.Fixture]
+		if !ok {
+			return fmt.Errorf("provenance not computed for fixture %s", oldRelease.Fixture)
+		}
+
+		// Check if provenance matches - if not, warn that fixture may have changed
+		if oldRelease.ProvenanceSHA256 != newProvenance {
+			fmt.Fprintf(os.Stderr, "WARNING: fixture %s for release %s changed (provenance mismatch)\n",
+				oldRelease.Fixture, oldRelease.Tag)
+		}
+
+		newCatalog.Releases = append(newCatalog.Releases, catalogRelease{
+			Tag:               oldRelease.Tag,
+			Fixture:           oldRelease.Fixture,
+			ProvenanceSHA256:  newProvenance, // Use new provenance in case fixture was regenerated
+			Signature:         newSig,
+			AppliedVersion:    oldRelease.AppliedVersion,
+			HasSourceMetadata: oldRelease.HasSourceMetadata,
+		})
+	}
+
+	// Update unmanifested fixtures with new signatures
+	for _, oldUnmanifested := range oldCatalog.UnmanifestedFixtures {
+		newSig, ok := fixtureSignatures[oldUnmanifested.Fixture]
+		if !ok {
+			return fmt.Errorf("fixture not found for unmanifested: %s", oldUnmanifested.Fixture)
+		}
+		newProvenance, ok := fixtureProvenances[oldUnmanifested.Fixture]
+		if !ok {
+			return fmt.Errorf("provenance not computed for fixture %s", oldUnmanifested.Fixture)
+		}
+
+		newCatalog.UnmanifestedFixtures = append(newCatalog.UnmanifestedFixtures, catalogFixture{
+			Fixture:           oldUnmanifested.Fixture,
+			ProvenanceSHA256:  newProvenance,
+			Signature:         newSig,
+			AppliedVersion:    oldUnmanifested.AppliedVersion,
+			HasSourceMetadata: oldUnmanifested.HasSourceMetadata,
+			Provenance:        oldUnmanifested.Provenance,
+		})
+	}
+
+	// Marshal to JSON with nice formatting
+	data, err := json.MarshalIndent(newCatalog, "", "  ")
+	if err != nil {
+		return fmt.Errorf("marshal catalog: %w", err)
+	}
+
+	// Write to file
+	if err := os.WriteFile(manifestPath, append(data, '\n'), 0644); err != nil {
+		return fmt.Errorf("write manifest: %w", err)
+	}
+
+	fmt.Printf("Regenerated manifest with %d releases and %d unmanifested fixtures\n",
+		len(newCatalog.Releases), len(newCatalog.UnmanifestedFixtures))
+	return nil
+}
+
+func loadCatalogFromPath(path string) (Catalog, error) {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return Catalog{}, err
+	}
+
+	var catalog Catalog
+	if err := json.Unmarshal(data, &catalog); err != nil {
+		return Catalog{}, fmt.Errorf("unmarshal catalog: %w", err)
+	}
+	return catalog, nil
+}

internal/compat/schema.go

@@ -412,7 +412,53 @@ func isFTSShadowObject(name string, virtualTables map[string]bool) bool {
 }
 
 func normalizeSQL(sqlText string) string {
-	return strings.TrimSpace(sqlText)
+	// Collapse runs of whitespace outside string literals, but preserve whitespace
+	// within single-quoted SQL strings (including '' escapes).
+	var result strings.Builder
+	inQuote := false
+	lastWasSpace := false
+
+	for i := 0; i < len(sqlText); i++ {
+		ch := sqlText[i]
+
+		// Check for single quote (start or end of string literal, or '' escape)
+		if ch == '\'' {
+			// Check if this is a '' escape (two consecutive single quotes)
+			if inQuote && i+1 < len(sqlText) && sqlText[i+1] == '\'' {
+				// Write both quotes and skip the next one
+				result.WriteByte(ch)
+				result.WriteByte(ch)
+				i++ // skip the next quote
+				lastWasSpace = false
+				continue
+			}
+			// Toggle quote state
+			inQuote = !inQuote
+			result.WriteByte(ch)
+			lastWasSpace = false
+			continue
+		}
+
+		// If inside quotes, preserve character as-is
+		if inQuote {
+			result.WriteByte(ch)
+			lastWasSpace = false
+			continue
+		}
+
+		// Outside quotes: collapse whitespace
+		if ch == ' ' || ch == '\t' || ch == '\n' || ch == '\r' {
+			if !lastWasSpace {
+				result.WriteByte(' ')
+				lastWasSpace = true
+			}
+		} else {
+			result.WriteByte(ch)
+			lastWasSpace = false
+		}
+	}
+
+	return strings.TrimSpace(result.String())
 }
 
 func schemaRecord(kind, table, name, columns, sqlText string) string {

internal/compat/schema_test.go

@@ -281,7 +281,8 @@ func TestInspectIndexRecognizesObservedDevelopmentV13Shape(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	const observedSignature = "sha256:952e517fe590b0c75a02996b6035ac6bb22143b9a707b6448ad05a592bf015b4"
+	// Signature recomputed after normalizeSQL made whitespace-insensitive (issue #52)
+	const observedSignature = "sha256:4d04377754986f0da2f61f2ab73889168f596eb84e1f02df96e23072072e9375"
 	if shape.Signature != observedSignature {
 		t.Fatalf("development V13 signature = %s, want %s", shape.Signature, observedSignature)
 	}
@@ -387,3 +388,59 @@ func openSchema(t *testing.T, schemaSQL string) *sql.DB {
 	}
 	return db
 }
+
+func TestNormalizeSQLIsWhitespaceInsensitiveOutsideStringLiterals(t *testing.T) {
+	tests := []struct {
+		name string
+		sql1 string
+		sql2 string
+	}{
+		{
+			name: "multiple spaces collapsed to single space",
+			sql1: "CREATE   TABLE  foo  (id  INTEGER)",
+			sql2: "CREATE TABLE foo (id INTEGER)",
+		},
+		{
+			name: "newlines and tabs collapsed to single space",
+			sql1: "CREATE\n  TABLE\tfoo\n(\nid\nINTEGER\n)",
+			sql2: "CREATE TABLE foo ( id INTEGER )",
+		},
+		{
+			name: "alter-built schema: inline vs multiline columns normalize the same",
+			sql1: "CREATE TABLE search_items (\n  content_type TEXT DEFAULT 'text',\n  extraction_version INTEGER,\n  was_interrupted INTEGER\n);",
+			sql2: "CREATE TABLE search_items ( content_type TEXT DEFAULT 'text', extraction_version INTEGER, was_interrupted INTEGER );",
+		},
+		{
+			name: "preserve space inside single-quoted literal",
+			sql1: "CREATE TABLE foo (body TEXT DEFAULT 'alpha  beta')",
+			sql2: "CREATE TABLE foo (body TEXT DEFAULT 'alpha  beta')",
+		},
+		{
+			name: "double-single-quote escape inside literal",
+			sql1: "CREATE TABLE foo (body TEXT DEFAULT 'O''Brien')",
+			sql2: "CREATE TABLE foo (body TEXT DEFAULT 'O''Brien')",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			norm1 := normalizeSQL(tt.sql1)
+			norm2 := normalizeSQL(tt.sql2)
+			if norm1 != norm2 {
+				t.Fatalf("normalization differs: %q != %q", norm1, norm2)
+			}
+		})
+	}
+}
+
+func TestNormalizeSQLPreservesWhitespaceInStringLiterals(t *testing.T) {
+	// This is critical: whitespace inside string literals must be preserved exactly
+	sql := "CREATE TABLE foo (body TEXT DEFAULT 'alpha  beta', CHECK (body <> 'gamma  delta'))"
+	norm := normalizeSQL(sql)
+	if !strings.Contains(norm, "'alpha  beta'") {
+		t.Fatalf("whitespace in first literal not preserved: %q", norm)
+	}
+	if !strings.Contains(norm, "'gamma  delta'") {
+		t.Fatalf("whitespace in second literal not preserved: %q", norm)
+	}
+}