Motivation
Import is single-core-bound: the bottleneck is our Rust-side PGN parsing (read text, replay moves, compute zobrist hashes), which runs in one loop. Observed directly on Linux during a first-run import — one core pinned at ~98%, the other 21 logical cores mostly idle (Core Ultra 7 165H). DuckDB's own thread pool can't help here because the work isn't inside a DuckDB query.
This is separate from — and stacks with — the WAL checkpoint_threshold fix (v0.14.11), which stops the writer from stalling on fsync. That fix keeps the pipeline flowing; this one lets the producer side use many cores instead of one, speeding up both Linux and Windows beyond the current single-core ceiling.
Key point: single writer ≠ single core
A single DuckDB write connection still gives two kinds of parallelism:
- Intra-query (already have it): DuckDB runs each statement multi-threaded (morsel-driven), governed by
SET threads. The set-based passes — dedup UPDATEs, CREATE INDEX (position index build), dedup_games joins — already fan out across cores. Note we currently hardcode threads=4 in chess-db/src/db/mod.rs; scaling that toward the core count is a cheap, separate win for those steps.
- Application-side (missing): the PGN parse, which sits upstream of the writer, is single-threaded today.
Proposed design
Parallelize the parse, keep the write serialized:
parse workers (N cores) → bounded channel → one append thread → single writer
Each game is independent to parse, so fan the parsing across a worker pool (rayon or a fixed worker set), then funnel finished rows into the single DuckDB Appender from one thread. The DB write stays fully serialized (respects the single-writer model and DuckDB's one-writer rule); only the CPU-heavy parsing goes wide.
- Applies to both
flush_games and position generation (flush_positions) — both are per-game CPU.
- Keep it behind the existing
fast/bulk path the wizard uses.
Caveat to get right
The importer assigns sequential game IDs, so parallel parsing needs either order-preserving chunking or ID assignment on the serial (append) side. Very doable — just the part to handle carefully.
What the single writer genuinely rules out (fine)
Two independent write transactions at once (e.g. importing two files into the same DB concurrently). Not a real loss: DuckDB only allows one writer anyway, and the append itself isn't the bottleneck — the parse is, and it's before the writer.
Files
chess-db/src/importer/mod.rs — parse loop, flush_games, flush_positions (BATCH_SIZE)chess-db/src/db/mod.rs — threads=4 cap
Related
- v0.14.11 import
checkpoint_threshold fix (fsync stalls) — orthogonal, already done.
Motivation
Import is single-core-bound: the bottleneck is our Rust-side PGN parsing (read text, replay moves, compute zobrist hashes), which runs in one loop. Observed directly on Linux during a first-run import — one core pinned at ~98%, the other 21 logical cores mostly idle (Core Ultra 7 165H). DuckDB's own thread pool can't help here because the work isn't inside a DuckDB query.
This is separate from — and stacks with — the WAL
checkpoint_thresholdfix (v0.14.11), which stops the writer from stalling on fsync. That fix keeps the pipeline flowing; this one lets the producer side use many cores instead of one, speeding up both Linux and Windows beyond the current single-core ceiling.Key point: single writer ≠ single core
A single DuckDB write connection still gives two kinds of parallelism:
SET threads. The set-based passes — dedupUPDATEs,CREATE INDEX(position index build), dedup_games joins — already fan out across cores. Note we currently hardcodethreads=4inchess-db/src/db/mod.rs; scaling that toward the core count is a cheap, separate win for those steps.Proposed design
Parallelize the parse, keep the write serialized:
Each game is independent to parse, so fan the parsing across a worker pool (rayon or a fixed worker set), then funnel finished rows into the single DuckDB Appender from one thread. The DB write stays fully serialized (respects the single-writer model and DuckDB's one-writer rule); only the CPU-heavy parsing goes wide.
flush_gamesand position generation (flush_positions) — both are per-game CPU.fast/bulk path the wizard uses.Caveat to get right
The importer assigns sequential game IDs, so parallel parsing needs either order-preserving chunking or ID assignment on the serial (append) side. Very doable — just the part to handle carefully.
What the single writer genuinely rules out (fine)
Two independent write transactions at once (e.g. importing two files into the same DB concurrently). Not a real loss: DuckDB only allows one writer anyway, and the append itself isn't the bottleneck — the parse is, and it's before the writer.
Files
chess-db/src/importer/mod.rs— parse loop,flush_games,flush_positions(BATCH_SIZE)chess-db/src/db/mod.rs—threads=4capRelated
checkpoint_thresholdfix (fsync stalls) — orthogonal, already done.