Complete managed full-text search: Turso parity, incremental storage, joins, browser

[Marc-André Moreau (mamoreau-devolutions)]

Summary

Closes the remaining gaps between Ahtola's CREATE INDEX … USING fts and the pinned Turso v0.8.0-pre.7 FTS surface, and makes it perform. Turso compiles FTS out of every WebAssembly build (#[cfg(all(feature = "fts", not(target_family = "wasm")))]); Ahtola's pure-managed index now works, and is validated, in the browser too.

Conformance

• turso/fts.sqltest now executes (it was skipped wholesale as @backend cli, though its cases are plain SQL). All pass except five MATCH-context errors that lack only the CLI Parse error: prefix. These are recorded like the existing vector marker, and the inventory's expected-failure count goes from 73 to 78.
• Every FTS test in upstream tests/integration/index_method/mod.rs and tests/fuzz/fts.rs is ported (ManagedFtsUpstreamIntegrationTests, 48 tests). Tantivy-internal counter assertions are dropped with a comment at each site.

Tokenization (Tantivy-exact, host independent)

• default/simple split on Rust char::is_alphanumeric. Previously .NET IsLetterOrDigit split Indic/Thai/Arabic words at vowel signs and dropped ², ½, Ⓐ.
• The remove-long limit is measured on the source before lowercasing, lowercasing is full per-character to_lowercase, and whitespace splits on ASCII whitespace only.
• All character data comes from a generated Unicode 16.0 table (scripts/unicode/Update-ManagedUnicodeData.ps1, fingerprint-tested). string.Normalize silently returns its input under InvariantGlobalization, which had turned unicode61 accent folding off in typical Blazor builds.

Planning

• A method-owned fts_match conjunct on a join arm is pushed into that arm, so its FTS index serves the join in either order (upstream test_fts_join_order_optimization). EXPLAIN QUERY PLAN reports the arm's method search.
• ORDER BY <alias> LIMIT 0 no longer raises no such column. This is a general engine fix.

Storage and incremental maintenance

Every write transaction and statement backup clones the table, and the clone's attachment used to start empty. The next query after any explicit transaction, file-backed write, UPDATE or DELETE therefore rebuilt the whole index.

• Postings are now log structured: immutable column-wise segments shared by reference, a copy-on-write overlay, tiered merges, and one-pass counting-sort bulk builds.
• Clones that preserve row lineage and revision get state-sharing ForkWithState() attachments plus a cloned journal.
• Whole-row-list rewrites (UPDATE, DELETE, UPSERT DO UPDATE, compiled DELETE) and REPLACE conflict deletes record their exact change sets. RecordBulk vouches only for the rewrite's own revision bumps.
• Lazy reconciliation of a published attachment forks, applies and publishes atomically behind a per-attachment gate. Concurrent readers of the shared live catalog therefore never see in-place mutation (this was a latent race).

Query execution

• The residual fts_match re-check on rows a method scan produced now answers from that scan's match set by rowid instead of re-tokenizing each row.
• Hits map back through the table's cached rowid lookup instead of a per-query map over every row.

Performance (50k docs, 46 tokens each, shared dev workstation, approximate)

	before	after	SQLite FTS5
write + search	~2.4 s	45–205 ms	0.2 ms
FTS join arm	658 ms	3.4 ms	2.6 ms
common-term count	543 ms	59 ms	1.2 ms
rare / mid term	0.7 / 6.8 ms	0.03 / 0.9 ms	0.02 / 2.4 ms
first search after reopen	2.3 s	0.59 s	2 ms
index heap	~240 MiB	~52 MiB	native
build index over rows	3.8 s	1.8 s	0.5 s

BenchmarkDotNet (TursoFtsWorkloadBenchmarks, 20k docs) puts search queries at 1.1–2.7× native SQLite FTS5.

Reviewer notes

• Recorded semantic divergence. An unfiltered ORDER BY fts_score(…) DESC LIMIT n keeps non-matching rows at 0.0, preserving SQL semantics on every plan. Upstream's Score cursor returns only ranked hits, which is what WHERE fts_match(…) returns here. Tell me if you want Turso's behavior instead.
• Write latency is engine-bound, not FTS-bound. On the same 50k-row table without any FTS index, a single-row INSERT/UPDATE/DELETE costs 63/178/37 ms, against about 0.04 ms in SQLite. Every INSERT re-validates the whole table, and each statement inside a transaction deep-copies the catalog, so one-row-per-statement loads are quadratic. Documented, with the multi-row INSERT recommendation. The fix is a separate transaction-architecture change: at least two paths mutate stored rows in place, so the copy can't simply be made shallow.
• Postings remain derived state, rebuilt on open; the storage format is unchanged. Scores are still not bit-identical to Tantivy.
• turso-src is untouched. Its local checkout is only initialized shallowly, and the pointer is unchanged.

Validation

• Full net10 suite before the final two query optimizations: 8,296 passed, 0 failed, 16 skipped. After them, all 614 FTS/index-method/vector/EXPLAIN/planner tests pass.
• New suites:
  • ManagedFtsUnicodePinningTests
  • ManagedFtsJoinPlanningTests
  • ManagedFtsSegmentStorageTests (differential fuzz against fresh builds with forks, merges and compaction)
  • ManagedFtsIncrementalMaintenanceTests (zero rebuilds across ten committed write shapes, in memory and on file, plus rollback)
• ./build.ps1 validate-browser-package passes in Chromium, including the new persistent-fts=8 OPFS round trip. ./build.ps1 validate-browser-trim passes, with no Ahtola-originated warnings.
• The Release build is clean on net8/net9/net10. No dotnet format findings on changed lines.

🤖 Generated with Claude Code