.NET hotspot benchmarks and optimizations for ADO.NET, Dapper and EF Core access patterns

[Marc-André Moreau (mamoreau-devolutions)]

Summary

This adds a benchmark suite for the SQLite access patterns typical .NET applications use: ADO.NET, Dapper, EF Core-style probes, connection-per-operation and paging. It runs Ahtola's Microsoft.Data.Sqlite-compatible facade against Microsoft.Data.Sqlite on the same SQLite-written WAL file. The PR then fixes the hot spots the suite exposed, plus one correctness bug found along the way.

Benchmarks

• DotNetHotspotBenchmarks (BenchmarkDotNet, Engine = Sqlite/Ahtola), 25 cases:
  • Connections: pooled open/close, and open/lookup/close.
  • Point reads: lookup by PK (new command, reused command, Dapper-style GetValue) and by GUID, COUNT by category, schema probe.
  • Row reads: typed, Dapper-style and GetFieldValue readers.
  • Queries: indexed equality, LIMIT/OFFSET and keyset paging, IN lists, LIKE prefix, JOIN + GROUP BY, FK lookup, multiple result sets.
  • Writes: autocommit and batched INSERT, UPDATE by PK, upsert, a unit of work with RETURNING, blobs.
• --hotspots-quick [filter] [ms] [engine]: an in-process stopwatch runner that prints Ahtola/SQLite ratios in about a minute, for the optimize-measure loop. BenchmarkDotNet remains the source of record. Documented in src/Benchmarks/README.md.

Results (quick runner, same machine; noisy, so read the trend rather than single numbers)

Case	Before	After	SQLite
Pooled open + close	370 µs	20 µs	0.1 µs
Pooled open + PK lookup + close	2.0 ms	50–65 µs	12–15 µs
PK lookup, Dapper-style GetValue	118 µs	37–49 µs	12–16 µs
Lookup by GUID (unique index)	140 µs	32–42 µs	10–14 µs
Indexed equality	751 µs	90–102 µs	37–45 µs
ORDER BY id LIMIT/OFFSET paging	620 µs	46–53 µs	35–42 µs
UPDATE by PK	1.98 ms	100–200 µs	13–19 µs
Unit of work (UPDATE + INSERT…RETURNING + DELETE)	1.1 ms	360–410 µs	50–80 µs
Batched INSERT in a transaction (per row)	26–35 µs	14–17 µs	0.8–1 µs

Read paths

• Shared per-table derived caches (TableDerivedCache), validated by copy-on-write content stamps and shared across statement clones:
  • rowid scan order
  • equality buckets
  • sorted managed-index orders
  • rowid→position lookup (a binary search when rowids ascend, which is the normal case)
• Index seeks:
  • An index used by repeated equality seeks on an unchanged table switches to its in-memory order after 32 seeks. Any write resets the count, so write-heavy workloads keep the cheap durable seek.
  • IN lists, rowid ranges, covering COUNT/EXISTS probes, ORDER BY satisfied by an index or by rowid order with early stop, and small-left join probes avoid full scans.
• Paging: a rowid-ordered LIMIT/OFFSET scan starts at the offset (rows are built lazily) instead of building and discarding the skipped rows.
• Facade:
  • Declared-type metadata (PRAGMA table_info/index_list) is cached per connection, validated by a new IManagedConnectionAdapter.GetTableSchemaIdentity.
  • GetValue only resolves declared types for TEXT/BLOB values.
• Parse cache: EmbeddedConnection.Prepare reuses the parse tree for the same SQL text while every table/view name the parser asked about still gets the same answer.
• Allocation:
  • Compare and SourceRow column lookup no longer allocate closures on every call.
  • The qualified-column map is reused across rows.
  • Joined rows share their qualifier→rowid layout.

Write paths

• Commit diff: copy-on-write chunks shared with the committed version are skipped, rows are compared by reference first, and appends count as inserts. A single-row change no longer diffs the whole table.
• Rowid-alias DML: WHERE id = ? on the rowid alias goes straight to the row's position.
• Batched b-tree inserts: SqliteIncrementalTableBtree.InsertMany merges every row that lands in the same leaf into one leaf write, instead of re-parsing and rewriting the leaf per row.
• Savepoint checkpoints: transaction mutation overlays use immutable maps, so the checkpoint taken before every statement is O(1). Previously a long transaction was quadratic in its row changes.
• Persist validation: the rowid-alias proof and CREATE INDEX round-trip checks are memoized; the alias proof only re-checks changed chunks.
• No-op UPDATEs (non-MVCC) skip the persist entirely.
• The catalog mutex name is cached for fully qualified paths.

Pooling and WAL protocol

• Returning a connection to the pool no longer adopts peers' commits: renting it, and every statement, does that anyway.
• Connection-local PRAGMAs (foreign_keys, busy_timeout, …) skip the committed-view refresh. The facade issues PRAGMA foreign_keys on every open.
• The WAL/wal-index validation and frame reads resume from the validated prefix, the write stamps read from open handles, and a 16-entry interior-page view cache is added to the table cursor.
• The foreign-key parent probe uses the rowid set, with a probe cache keyed by table revision. This also fixes a pre-existing memory leak.

Correctness fix

SQLite stores an integral REAL with |x| < 2^51 in integer form (MEM_IntReal) and applies REAL affinity when reading it back. Ahtola previously returned such values as INTEGER. It now applies stored REAL affinity on read, and writes integral REALs in SQLite's integer form. Index records compare int/real-tolerantly during validation.

Reviewer notes

• Public API addition: two default interface members on IManagedConnectionAdapter. GetTableSchemaIdentity returns null by default, which disables the facade cache. ResetForPoolReturn falls back to ResetForPooling().
• Behavior change in a test: Stat4SampleValidationReusesOneRowIdMapPerTableRevision became …PerRowIdSet, because the rowid→position lookup now survives in-place UPDATEs (STAT4 sample keys are still re-read from live rows).
• Deliberately not changed: the committed-view token still stats the database and WAL files, which pooled opens and statements pay for. Those stamps detect peer engines writing outside the wal-index. Remaining large gaps are structural (JOIN + GROUP BY ~30x and LIKE ~7x materialize rows), along with the pager's commit protocol.

Testing

• Full managed suite: pwsh ./scripts/Invoke-ManagedTestSuite.ps1 -Framework net10.0 → 8,840 passed, 0 failed.
• New tests:
  • HotspotAccessPathParityTests runs every optimized access path differentially against Microsoft.Data.Sqlite on identical fixtures. It covers IN lists, rowid ranges, covering probes, joins (rowids through 2- and 3-way joins), DML edge cases on the rowid alias, the warm-order switch, chunk-spanning commits with gap fills and appends, and integral REALs; SQLite runs integrity_check on files Ahtola wrote.
  • ManagedReaderDeclaredTypeCacheTests: TEMP shadowing, ALTER TABLE ADD COLUMN, CREATE/DROP INDEX, rolled-back DDL, and DDL from another connection.
  • InsertManyMatchesRowByRowInsertsAcrossLeavesAndAppends: gap fills across leaves, appends, an overflow record and duplicate rejection.
• dotnet format whitespace: no issues in changed lines.

🤖 Generated with Claude Code