Marc-André Moreau (mamoreau-devolutions) · Sep 25, 2026 · Sep 25, 2026
diff --git a/README.md b/README.md
@@ -398,6 +398,12 @@ Treat Ahtola as SQLite-*compatible*, not a full SQLite replacement:
   foreign shadow layouts fail closed. FTS5 storage is likewise not a portable
   shadow-table representation. See
   [docs/managed-vtab-fts-rtree-integration.md](docs/managed-vtab-fts-rtree-integration.md).
+- **Turso full-text search** — `CREATE INDEX … USING fts` with `fts_match`,
+  `fts_score`, `fts_highlight`, `MATCH` and `OPTIMIZE INDEX`, Tantivy-exact
+  tokenizers over a pinned Unicode table, incremental maintenance, and join-arm
+  planning, on the desktop and in the browser (Turso's own WebAssembly builds omit
+  FTS). Postings are derived state rebuilt on open, not a Tantivy file layout. See
+  [docs/managed-index-methods.md](docs/managed-index-methods.md).
 - **SQL CDC** — `PRAGMA capture_data_changes_conn` implements Turso v0.7.2's
   per-connection V1/V2 CDC tables and transactional COMMIT records. It is
   independent of the managed replica's private journal and does not provide a

diff --git a/docs/browser-wasm.md b/docs/browser-wasm.md
@@ -363,6 +363,22 @@ against the on-disk checksum chain again before decryption. A clean,
 authoritative WAL-mode main database can open without a WAL; if its header
 requires WAL recovery and the WAL is missing, open fails.
 
+## Full-text search
+
+`CREATE INDEX … USING fts`, `fts_match`, `fts_score`, `fts_highlight`, `MATCH` and
+`OPTIMIZE INDEX` work in the browser exactly as on the desktop — the pinned Turso
+release compiles its Tantivy-backed FTS out of every WebAssembly build, but Ahtola's
+index is pure managed code. The package lane exercises it over OPFS, including a
+reopen that rebuilds the postings from the persisted rows (see
+`RunPersistentFullTextSearchAsync` in `samples/BrowserWasmConsumer/App.razor`).
+
+- Tokenization reads a pinned Unicode 16.0 table instead of the runtime's ICU, so it is
+  identical in globalization-invariant and hybrid-globalization builds.
+- The postings are derived state held in memory next to the database image: plan for
+  about 20 bytes plus 8 per token position per indexed term occurrence, and a one-pass
+  rebuild the first time an index is queried after the database is opened.
+- Bounded scan connections do not plan FTS queries; use an ordinary connection.
+
 ## EF Core
 
 Create and asynchronously open a browser connection, then give that existing

diff --git a/docs/managed-index-methods.md b/docs/managed-index-methods.md
diff --git a/docs/turso-benchmark-portability.md b/docs/turso-benchmark-portability.md
@@ -16,6 +16,7 @@ corresponding Rust internal API.
 | `core/benches/select_star_benchmark.rs` | Wide scans under WAL and MVCC | Direct SQL port |
 | `core/benches/fts_benchmark.rs` | Cold/warm search, selectivity, ingest, commit/merge churn | Direct Ahtola index-method port |
 | `core/benches/fts_comparison_benchmark.rs` | Ahtola-style FTS versus SQLite FTS5 | Direct SQL port with storage-model labels |
+| (Ahtola workload) `TursoFtsWorkloadBenchmarks` | Zipf 20,000-term corpus: rare/mid/common terms, AND, phrase, prefix, ranked top-10, write-then-search cycle, versus SQLite FTS5 | Managed workload complementing the upstream ports |
 | `core/benches/graph_queries_benchmark.rs` | Analyzed/unanalyzed graph queries | Managed deterministic-fixture adaptation |
 | `core/benches/tpc_h_benchmark.rs` | Supported TPC-H query execution | Managed asset and supported-query adaptation |
 | `core/benches/json_benchmark.rs` | JSONB conversion and JSON Patch | Direct SQL port |

diff --git a/docs/turso-gap-analysis.md b/docs/turso-gap-analysis.md
@@ -1058,7 +1058,7 @@ than absent subsystems.
 | `func-array-postgres-family` | missing | s3-perf | L | 0 | 0 | Postgres-style ARRAY(...)/array_element/array_append/etc. scalar family, always compiled (not behind a cargo feature flag) but not part of stock SQLite semantics and not… |
 | `func-extension-format-btrim` | extension | s4-intentional | S | 0 | 0 | FORMAT (an alias for PRINTF, matching real SQLite's built-in but not present as a distinct entry in Turso's from_str dispatch table) and BTRIM (Postgres-style alias for T… |
 | `func-extension-uuid-family` | extension | s4-intentional | S | 0 | 0 | Ahtola registers a full UUID v4/v7 generation family (text and blob forms, plus gen_random_uuid() for Postgres compatibility) that has no counterpart anywhere in turso-sr… |
-| `func-fts-scalar-family` | partial | s3-perf | L | 0 | 0 | Reconciled 2026-08-29 with pinned v0.8.0-pre.7: method query grammar, pinned tokenizers, MATCH forms, NULL/TEXT-only behavior, varargs highlight, unordered column binding, boosts, seven declared plans, rowid-ordered matching, ranked top-k and OPTIMIZE are managed. Corpus-aware `fts_score` is schema-nondeterministic and rejected from indexes, generated columns, partial predicates, and CHECK constraints. Tantivy storage/exact score bits remain out of scope. |
+| `func-fts-scalar-family` | partial | s3-perf | L | 0 | 0 | Reconciled 2026-08-29 with pinned v0.8.0-pre.7: method query grammar, pinned tokenizers, MATCH forms, NULL/TEXT-only behavior, varargs highlight, unordered column binding, boosts, seven declared plans, rowid-ordered matching, ranked top-k and OPTIMIZE are managed. Corpus-aware `fts_score` is schema-nondeterministic and rejected from indexes, generated columns, partial predicates, and CHECK constraints. 2026-09-25: upstream `turso/fts.sqltest` executes and the upstream FTS integration/fuzz tests are ported; tokenizers are Tantivy-exact over a pinned Unicode 16.0 table; FTS serves join arms; postings are segmented with state-sharing forks, so writes no longer force rebuilds. Tantivy storage/exact score bits remain out of scope. |
 | `func-gcd-lcm-missing` | missing | s3-perf | S | 0 | 0 | gcd()/lcm() (Turso/SQLite-3.41+-style math helpers) have no hits anywhere in src/Ahtola.Core or Ahtola.Core/EmbeddedDatabase.MathFunctions.cs. Not covered by the vendored… |
 | `func-numeric-boolean-ip-helpers-missing` | missing | s4-intentional | M | 0 | 0 | Internal-flavored helper functions supporting Turso's typed BOOLEAN/NUMERIC column extensions and validated IP address type; no SQLite equivalent and no corpus coverage.… |
 | `func-octet-length-missing` | missing | s1-correctness | S | 0 | 0 | octet_length is absent from SqliteBuiltinFunctions.Names and from EmbeddedDatabase.StringFunctions.cs / EmbeddedDatabase.cs (no case-insensitive hit for 'octet' anywhere… |
@@ -1555,6 +1555,45 @@ defects, all now closed with regression coverage in
   is rejected in CHECK constraints while the checked statement could mutate the
   same corpus.
 
+**FTS completeness follow-up (2026-09-25).**
+
+- *Conformance actually runs.* `turso/fts.sqltest` was skipped wholesale as
+  `@backend cli` (upstream routes it to `tursodb` only because the `fts` cargo
+  feature is compiled into the CLI). Its cases are plain SQL, so the corpus now
+  executes them; all pass except five `MATCH`-context errors that lack the CLI's
+  `Parse error:` prefix, recorded like the existing vector marker. Every FTS test in
+  upstream `tests/integration/index_method/mod.rs` and `tests/fuzz/fts.rs` is ported
+  as `ManagedFtsUpstreamIntegrationTests` (48 tests; Tantivy-internal counters such as
+  segment and writer-cache statistics are dropped with a comment at each site).
+- *Tokenization is Tantivy-exact and host independent.* `default`/`simple` split on
+  Rust `char::is_alphanumeric` (previously .NET `IsLetterOrDigit`, which broke Indic,
+  Thai and Arabic words at vowel signs and dropped `²`, `½`, `Ⓐ`), the remove-long
+  limit is measured on the source span before lowercasing, lowercasing is Rust's full
+  per-character mapping, and `whitespace` splits on ASCII whitespace only. All
+  character data comes from a generated Unicode 16.0 table
+  (`scripts/unicode/Update-ManagedUnicodeData.ps1`): `string.Normalize` silently
+  returned its input under `InvariantGlobalization`, which had disabled `unicode61`
+  accent folding in typical Blazor builds.
+- *Joins.* A method-owned `fts_match` conjunct on one join arm is pushed into that
+  arm, so its FTS index serves the join in either table order (upstream
+  `test_fts_join_order_optimization`), and EXPLAIN QUERY PLAN reports the arm's
+  method search instead of `MANAGED EVALUATOR FALLBACK`.
+- *Writes no longer rebuild the index.* Every write transaction and statement backup
+  clones the table, and the clone's attachment started empty, so the next query
+  after any explicit transaction, any file-backed write, and any UPDATE or DELETE
+  rebuilt the whole index (0.5–1.4 s at 20–50k documents). Postings are now
+  log-structured segments shared by state-carrying forks, and whole-row-list
+  rewrites report exact change sets; ten committed write shapes cause zero rebuilds
+  (`ManagedFtsIncrementalMaintenanceTests`). At 50k documents the index needs about
+  40 MiB instead of about 190 MiB, and a cold rebuild takes about 0.3 s instead of 1.3 s.
+- *Browser.* The pinned Turso compiles FTS out of every WebAssembly build; the Chromium
+  package lane now proves Ahtola's FTS over OPFS, including reopen.
+- *Also fixed:* `ORDER BY <result alias> LIMIT 0` raised `no such column`.
+- *Recorded divergence:* an unfiltered `ORDER BY fts_score(…) DESC LIMIT n` keeps
+  non-matching rows at `0.0` (SQL semantics on every plan); upstream's `Score`
+  cursor returns only ranked hits. The `WHERE fts_match(…)` form returns the
+  upstream result.
+
 **Current residual (honest, not scoreboard).** Inventory is **216 closed · 0 open**,
 and the live conformance expected-failures file contains two intentional
 Turso-negative markers for STORED generated columns. “Closed” still includes

diff --git a/docs/turso-gap-inventory.json b/docs/turso-gap-inventory.json
@@ -14,8 +14,8 @@
       "commit_at_analysis": "7b62a3f"
     },
     "conformance_ground_truth": "src/Ahtola.Tests/Conformance/managed-sqltest-expected-failures.txt (606 failure lines at analysis time)",
-    "current_conformance_expected_failures": 73,
-    "current_conformance_expected_failures_note": "Integrated work closes the original SQL/window/CTE correctness groups, 24 new SQL-extension, 14 generated-column/reserved-name, three legacy-view, safe ATTACH routing/CDC/sequence behavior, and the cycling-sequence commit failure. Non-cycling watermarks are repaired; its private row-count representation remains different. Multiple memory-only databases can commit atomically; any physical database write must be the transaction's sole database mutation. The planner and Plan/EXPLAIN parity work closes 31 additional conformance markers, including JSON EQP, grouped-derived probes, and correlated-IN automatic-index cursor allocation. Other assigned gaps and intentional feature/representation/eager-corruption differences remain. These are not 73 independent bugs. The remaining-gap plan tracks current work; the 217 closed records below are historical dispositions.",
+    "current_conformance_expected_failures": 78,
+    "current_conformance_expected_failures_note": "Integrated work closes the original SQL/window/CTE correctness groups, 24 new SQL-extension, 14 generated-column/reserved-name, three legacy-view, safe ATTACH routing/CDC/sequence behavior, and the cycling-sequence commit failure. Non-cycling watermarks are repaired; its private row-count representation remains different. Multiple memory-only databases can commit atomically; any physical database write must be the transaction's sole database mutation. The planner and Plan/EXPLAIN parity work closes 31 additional conformance markers, including JSON EQP, grouped-derived probes, and correlated-IN automatic-index cursor allocation. Other assigned gaps and intentional feature/representation/eager-corruption differences remain. Executing the previously CLI-gated turso/fts.sqltest corpus adds five markers, all the CLI \"Parse error:\" prefix on otherwise identical MATCH-context errors. These are not 78 independent bugs. The remaining-gap plan tracks current work; the 217 closed records below are historical dispositions.",
     "current_execution_plan": "docs/turso-remaining-gap-plan.md",
     "last_reconciled": "2026-09-23",
     "entry_count": 217,
@@ -1742,11 +1742,11 @@
       "layer": "functions",
       "kind": "partial",
       "turso_ref": "turso-src/core/function.rs::FtsFunc::Score (\"fts_score\"), FtsFunc::Match (\"fts_match\"), FtsFunc::Highlight (\"fts_highlight\")",
-      "ahtola_ref": "src/Ahtola.Core/Search/ManagedFtsFunctions.cs; src/Ahtola.Core/Search/ManagedFtsSearchIndex.cs; src/Ahtola.Core/SqliteBuiltinFunctions.cs::Names",
+      "ahtola_ref": "src/Ahtola.Core/Search/ManagedFtsFunctions.cs; src/Ahtola.Core/Search/ManagedFtsSearchIndex.cs; src/Ahtola.Core/Search/ManagedUnicode.cs; src/Ahtola.Core/SqliteBuiltinFunctions.cs::Names",
       "severity": "s3-perf",
       "effort": "L",
-      "conformance_links": [],
-      "notes": "Reconciled again against pinned v0.8.0-pre.7 on 2026-08-29. fts_match, fts_score and the pinned varargs fts_highlight(text..., before, after, query) ship over a pure-managed inverted index; fts_highlight_legacy and fts_snippet are explicit Ahtola extensions. Method queries use a distinct Tantivy-compatible grammar (adjacency OR, AND/NOT, field phrases/prefixes and boosts), while managed FTS5 keeps SQLite grammar. Tokenizers default/raw/simple/whitespace/ngram, default ngram 2..3, TEXT-only indexing, NULL match=0, unbound score=REAL 0.0, unordered column-set binding, duplicate-index ambiguity checks and all seven pinned FTS plan declarations are implemented. Unordered matches preserve rowid order and only complete score ordering uses top-k. Corpus-aware fts_score is schema-nondeterministic and is rejected from indexes, generated columns, partial predicates, and CHECK constraints. See docs/managed-index-methods.md.",
+      "conformance_links": ["conformance/sqlite-sqltests/turso/fts.sqltest"],
+      "notes": "Reconciled again against pinned v0.8.0-pre.7 on 2026-08-29. fts_match, fts_score and the pinned varargs fts_highlight(text..., before, after, query) ship over a pure-managed inverted index; fts_highlight_legacy and fts_snippet are explicit Ahtola extensions. Method queries use a distinct Tantivy-compatible grammar (adjacency OR, AND/NOT, field phrases/prefixes and boosts), while managed FTS5 keeps SQLite grammar. Tokenizers default/raw/simple/whitespace/ngram, default ngram 2..3, TEXT-only indexing, NULL match=0, unbound score=REAL 0.0, unordered column-set binding, duplicate-index ambiguity checks and all seven pinned FTS plan declarations are implemented. Unordered matches preserve rowid order and only complete score ordering uses top-k. Corpus-aware fts_score is schema-nondeterministic and is rejected from indexes, generated columns, partial predicates, and CHECK constraints. Completeness follow-up 2026-09-25: the upstream turso/fts.sqltest corpus now executes (it was skipped as @backend cli) and every upstream tests/integration/index_method FTS test plus tests/fuzz/fts.rs is ported (ManagedFtsUpstreamIntegrationTests); tokenizers reproduce Tantivy 0.26 exactly (char::is_alphanumeric runs, source-length remove-long, full per-character lowercase, ASCII-only whitespace) from a generated Unicode 16.0 table instead of host ICU, so globalization-invariant browser builds fold unicode61 accents correctly; fts_match on a join arm is pushed into that arm so its FTS index serves joins in either order with honest EXPLAIN QUERY PLAN rows; ORDER BY result aliases resolve under LIMIT 0. The one recorded semantic divergence is that an unfiltered ORDER BY fts_score(...) keeps non-matching rows at 0.0 (SQL semantics on every plan) where upstream returns only ranked hits. See docs/managed-index-methods.md.",
       "status": "closed"
     },
     {
@@ -3161,7 +3161,7 @@
       "severity": "s4-intentional",
       "effort": "L",
       "conformance_links": [],
-      "notes": "Opened and closed 2026-08-24 by managed-index-methods-fts as a recorded divergence. A method index is a real sqlite_schema 'index' row with a real rootpage and an ordinary SQLite index b-tree; no page, cell or WAL format changed. A versioned state header rides in a trailing SQL comment on the index's sql text (the same mechanism already shipped for managed virtual tables), and the inverted index (terms, term frequencies, positions, column masks, tombstones) is DERIVED state rebuilt from the base rows inside the catalog snapshot, so rollback/savepoint/crash/VACUUM correctness is inherited. Ahtola does NOT synthesize FTS5 shadow tables and the file is NOT interoperable: stock sqlite3 cannot parse CREATE INDEX ... USING fts and reports a malformed schema for that row, exactly as it would for Turso. Do not describe this layout as SQLite-compatible. Review follow-up 2026-08-24: postings are generation stamped (an upserted or reused rowid retires its previous terms immediately); ranking-only plans retain every unranked base row and are priced accordingly; attachments are published only on success and dropped on every failure; REINDEX/Optimize build detached and publish atomically; DROP INDEX runs Destroy; state envelopes are decoded only for USING-method declarations, length-bounded before the base64 decode allocates, and fully field-validated; detail/columnsize/gram options are implemented or rejected; folding preserves exact source offsets across combining marks and surrogate pairs; prefix limits count live terms only; maintenance is revision-aware through RowStore.Revision plus a per-table mutation journal fed by ReportRowChange, with PreCommit run inside the commit; and the foundation is method generic (generic result rows, per-method planner adapters with KNN hooks, arbitrary SqlValue arguments, no FTS casts in the core), proven by a non-FTS KNN method that plans and executes end to end.",
+      "notes": "Opened and closed 2026-08-24 by managed-index-methods-fts as a recorded divergence. A method index is a real sqlite_schema 'index' row with a real rootpage and an ordinary SQLite index b-tree; no page, cell or WAL format changed. A versioned state header rides in a trailing SQL comment on the index's sql text (the same mechanism already shipped for managed virtual tables), and the inverted index (terms, term frequencies, positions, column masks, tombstones) is DERIVED state rebuilt from the base rows inside the catalog snapshot, so rollback/savepoint/crash/VACUUM correctness is inherited. Ahtola does NOT synthesize FTS5 shadow tables and the file is NOT interoperable: stock sqlite3 cannot parse CREATE INDEX ... USING fts and reports a malformed schema for that row, exactly as it would for Turso. Do not describe this layout as SQLite-compatible. Review follow-up 2026-08-24: postings are generation stamped (an upserted or reused rowid retires its previous terms immediately); ranking-only plans retain every unranked base row and are priced accordingly; attachments are published only on success and dropped on every failure; REINDEX/Optimize build detached and publish atomically; DROP INDEX runs Destroy; state envelopes are decoded only for USING-method declarations, length-bounded before the base64 decode allocates, and fully field-validated; detail/columnsize/gram options are implemented or rejected; folding preserves exact source offsets across combining marks and surrogate pairs; prefix limits count live terms only; maintenance is revision-aware through RowStore.Revision plus a per-table mutation journal fed by ReportRowChange, with PreCommit run inside the commit; and the foundation is method generic (generic result rows, per-method planner adapters with KNN hooks, arbitrary SqlValue arguments, no FTS casts in the core), proven by a non-FTS KNN method that plans and executes end to end. Storage follow-up 2026-09-25: the derived postings are now log structured (immutable column-wise segments of roughly 20 bytes plus 8 per position per posting, a copy-on-write overlay, tiered merges, one-pass counting-sort bulk builds), and table clones that preserve row lineage and revision carry state-sharing attachment forks plus a cloned journal, while whole-row-list rewrites (UPDATE, DELETE, UPSERT DO UPDATE, compiled DELETE) and REPLACE conflict deletes record their exact change sets. Every committed write therefore reconciles incrementally instead of rebuilding the index on the next query (previously every explicit transaction, file-backed write, UPDATE and DELETE forced a full rebuild); a 50,000-document corpus holds its index in about 40 MiB instead of about 190 MiB and rebuilds on open in about 0.3 s instead of 1.3 s. The postings remain derived, not persisted.",
       "status": "closed"
     },
     {