Repository navigation
Linear-time inserts, Turso v0.8.0-pre.13 vector parity, browser vector gate - #73
Merged
Marc-André Moreau (mamoreau-devolutions) merged 3 commits intoSep 26, 2026
Conversation
…r gate Bump the turso-src pin to v0.8.0-pre.13 (64b8ef574) and port its vector changes. The dense float32/float64 cosine rule follows b9414023d: two zero vectors are at distance 0, and a zero dot product is at distance 1, decided before dividing. That is Turso's pure-Rust fallback, the code path Turso runs on wasm. The rest of the core/vector delta is simsimd build plumbing, and turso/vector.sqltest is byte-identical to pre.7. Upstream's index-method OFFSET fix (78395e704) needed no engine change; its FTS case is added as a test. Remove the O(table) work every statement did, which made row-at-a-time inserts quadratic: - Catalog clones share row and rowid lists copy-on-write in 1024-element chunks (CowChunkedList) instead of deep-copying every row of every table. The two sites that patched stored rows in place (incremental blob writes, sqlite_sequence renames) now replace the row. - Rowids are cached per table as an immutable set with their maximum, keyed on row-store lineage, revision and count, and extended by inserts rather than rebuilt by PrepareInsert and ValidateRowids. - INSERT validation checks NOT NULL and non-unique index keys on the new rows only, skips rescanning a rowid alias that ValidateRowids just proved unique, and validates through a view instead of copying the table. 64-dim vector inserts into an indexed table, one row per statement, go from 9.1 ms/row at 20k rows to 87 us/row at 40k rows. The browser package consumer now builds a USING vector index over OPFS, reopens it, and requires the planner to choose it and to match the scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r-search-74db66 # Conflicts: # scripts/Invoke-BrowserPackageConsumer.ps1
The cross-process replica race tests poll for the worker's started,
blocked and completed files with File.Exists and then read them. The
worker wrote them with File.WriteAllText, which creates the file before
writing it, so a read in between saw an empty or still-locked file
("being used by another process"). The assertion then threw inside the
fault-injection callback, and the worker's Dispose masked it with an
exit-code failure. This showed up intermittently on the Windows coverage
leg (ColdOpenRollbackDiscardsCommittedReplacementGenerationWal,
ColdOpenRollbackDiscardsReplacementWalRecreatedAfterQuarantineCrash).
Each signal is now written to a staging file and renamed into place.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Marc-André Moreau (mamoreau-devolutions)
merged commit Sep 26, 2026
34256d4
into
master
11 checks passed
Marc-André Moreau (mamoreau-devolutions)
deleted the
claude/ahtola-vector-search-74db66
branch
September 26, 2026 13:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Vector search parity. Against Turso's Rust engine, Ahtola's vector support was already complete: the vector functions, the serialized formats, and an exact
USING vectorindex forORDER BY vector_distance_*(…) LIMIT k. This PR moves theturso-srcpin tov0.8.0-pre.13(64b8ef574) and ports the one behavioral change in that release:vector_distance_cosfollowsb9414023d: two zero vectors are at distance 0, and a zero dot product is at distance 1, decided before dividing. This is Turso's pure-Rust fallback, which Turso runs on wasm, so native and browser results stay identical.core/vectordelta is simsimd build plumbing.turso/vector.sqltestis byte-identical to pre.7.78395e704, #7523) needed no engine change here. Its FTS case is added as a test.F32_BLOB(N)typing,libsql_vector_idx,vector_top_k, DiskANN) do not exist in the Rust engine through pre.13, so they are out of scope.Insert performance. Every statement did O(table) work, so row-at-a-time inserts were quadratic:
CowChunkedList).sqlite_sequencerenames. They now replace the row instead, because stored row arrays are shared with clones.PrepareInsertandValidateRowidsrebuilt a HashSet of every rowid per statement. Rowids are now cached per table as an immutable set plus their maximum, keyed on row-store lineage, revision and count, and extended by inserts.ValidateRowidsjust proved unique, and validates through a view instead of copying the table. Unique constraints are still checked against every row.One visible behavior change from the cosine port: float32 vectors whose squared norm underflows to zero, while their dot product doesn't, now report
+∞instead of1.ManagedVectorIndexReviewRegressionTests.CosineRecallSurvivesScalarNormCollapseis updated for that. It still requires the index and the scan to agree, and now also compares at full-table LIMIT.Browser.
samples/BrowserWasmConsumernow builds aUSING vectorindex over OPFS, reopens it, and requires the planner to choose it and to return the same rows as the unindexed scan. The expected status inscripts/Invoke-BrowserPackageConsumer.ps1is extended withvector-search=195-1987-643.Testing
CowChunkedListTests(randomized differential againstList<T>with clones),RowStoreCopyOnWriteTests(clone isolation, failed-statement atomicity, AUTOINCREMENT rename rollback, blob write rolled back to a savepoint), cosine zero-norm and zero-dot cases, and FTS ranked LIMIT/OFFSET.validate-project-closurepasses.format-checkreports only pre-existing issues in lines this PR doesn't touch.🤖 Generated with Claude Code