bf_tree migration away from diskann-providers by JordanMaples · Pull Request #1020 · microsoft/DiskANN

JordanMaples · 2026-05-05T18:45:45Z

So far, this is a mostly mechanical movement of the bf_tree prorovider from inside of diskann-providers crate into its own crate. This is largely a copilot authored PR, guided by me.

The general flow of the work was:

create benchmarks, similar to the inmem ones, for bf_tree. (search and insert)
extract the bf_tree provider.
move tests and perform cleanup.
validate that there wasn't a perf degradation, using the benchmarks, post move.

There are a couple remaining things to do here before we take this PR:

Look at removing the tableprovider functionality for deletes from the bf_tree and instead directly dropping the keys
dropping PQ functionality from the bf_tree & add SQ functionality

Some concerns that need to be addressed before this is ready for review.
As part of the move, there are some concepts that were moved into glue.rs and had their visibility expanded. This will need to be cleaned up and moved back to their previous location (and visibility level) once the tableprovider for the delete dealt with.

Add bf_tree insert benchmarks (Criterion + IAI) Add pre-migration bf_tree provider benchmarks mirroring the existing inmem insert benchmark. Both Criterion (wall-clock) and IAI-Callgrind (instruction-count) benchmarks use the same sift-256 dataset and PQ training as the inmem benchmarks for apples-to-apples comparison. Gated behind #[cfg(feature = "bf_tree")] so existing benchmarks continue to work without the feature flag. Run with: cargo bench --bench bench_main --features virtual_storage,bf_tree Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Add bf_tree search benchmarks for migration parity Add search benchmarks (Criterion + IAI-Callgrind) alongside existing insert benchmarks. Search exercises the read path through BfTreeProvider (vector lookups, neighbor access, greedy search traversal) which is critical for detecting regressions post-migration. The search benchmark builds the index once during setup, then measures 10 KNN queries (k=10, l=20) against the populated index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Extract bf_tree provider into diskann-bf_tree-provider crate Move the bf_tree provider and caching modules from diskann-providers into the dedicated diskann-bf_tree-provider crate. This makes the bf-tree dependency explicit at the crate level rather than hidden behind a feature flag. Changes: - Move bf_tree/ and caching/ modules to diskann-bf_tree-provider/src/ - Set up Cargo.toml with proper dependencies (bf-tree, diskann, diskann-providers, diskann-utils, diskann-vector, etc.) - Convert all crate:: imports to diskann_providers:: imports - Bump visibility of postprocess module, AsDeletionCheck/DeletionCheck traits, and TableDeleteProviderAsync methods to pub - Remove bf_tree feature flag and optional deps from diskann-providers - Remove cfg(feature = "bf_tree") gates from table_delete_provider.rs - Rename provider/provider.rs to provider/bf_tree_provider.rs (clippy) - Disable caching example.rs tests (cross-crate cfg(test) limitation) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Move bf_tree benchmarks to diskann-bf_tree-provider Migrate Criterion and IAI-Callgrind benchmark targets from diskann-providers to the new diskann-bf_tree-provider crate. Update imports to use the new crate's module paths. Remove bf_tree benchmark remnants (cfg gates, modules) from diskann-providers benches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> cargo.lock got missed by last commit Move delete bitmap serialization to diskann-bf_tree-provider Move to_bytes/from_bytes logic out of TableDeleteProviderAsync (where it was bf_tree-scoped) into a dedicated delete_bitmap_serde module in the new crate. The reimplementation works through the public API and produces byte-identical output (LE u32 words). Revert the methods in diskann-providers to pub(crate) + #[cfg(test)]. Adds format-level compatibility tests including known fixtures and padding bit rejection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Move DeletionCheck traits and RemoveDeletedIdsAndCopy to diskann::graph::glue These are core to the inplace delete algorithm's search post-processing and belong in the diskann crate alongside SearchPostProcess, CopyIds, FilterStartPoints, and Pipeline. This removes the need to widen diskann-providers' postprocess module to pub for external crates. Also removes the now-dead to_bytes/from_bytes methods and their tests from TableDeleteProviderAsync in diskann-providers, since that serialization logic has been moved to diskann-bf_tree-provider. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Use workspace_root() for benchmark test data paths Replace env!("CARGO_MANIFEST_DIR")-relative paths with diskann_utils::workspace_root() so benchmarks find test_data/ without requiring symlinks in each crate directory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The bf_tree feature was removed from diskann-providers when the provider was extracted into its own crate (diskann-bf_tree-provider). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

codecov-commenter · 2026-05-05T21:17:25Z

Codecov Report

❌ Patch coverage is 88.48315% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.92%. Comparing base (95b6593) to head (deba40e).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
...n-bf_tree-provider/src/provider/hybrid_computer.rs	13.88%	31 Missing ⚠️
...-bf_tree-provider/src/provider/bf_tree_provider.rs	88.57%	8 Missing ⚠️
..._tree-provider/src/provider/delete_bitmap_serde.rs	98.23%	2 Missing ⚠️

❌ Your patch status has failed because the patch coverage (88.48%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1020      +/-   ##
==========================================
+ Coverage   89.51%   89.92%   +0.40%     
==========================================
  Files         460      480      +20     
  Lines       85424    90781    +5357     
==========================================
+ Hits        76469    81631    +5162     
- Misses       8955     9150     +195

Flag	Coverage Δ
miri	`89.92% <88.48%> (+0.40%)`	⬆️
unittests	`89.88% <88.48%> (+0.52%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-bf_tree-provider/src/caching/bf_cache.rs	`93.39% <100.00%> (ø)`
diskann-bf_tree-provider/src/caching/error.rs	`98.86% <ø> (ø)`
diskann-bf_tree-provider/src/caching/provider.rs	`28.57% <ø> (ø)`
diskann-bf_tree-provider/src/caching/utils.rs	`92.39% <100.00%> (ø)`
diskann-bf_tree-provider/src/provider/mod.rs	`0.00% <ø> (ø)`
...bf_tree-provider/src/provider/neighbor_provider.rs	`92.21% <ø> (ø)`
...ree-provider/src/provider/quant_vector_provider.rs	`89.96% <100.00%> (ø)`
...n-bf_tree-provider/src/provider/vector_provider.rs	`86.03% <100.00%> (ø)`
...roviders/src/model/graph/provider/async_/common.rs	`84.79% <ø> (-3.51%)`	⬇️
...odel/graph/provider/async_/inmem/full_precision.rs	`98.39% <ø> (ø)`
... and 8 more

... and 72 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Rewrite QuantVectorProvider to use Arc<Poly<dyn Quantizer>> - Add local HybridComputer and QuantQueryComputer wrapper - Update CreateQuantProvider impl for Poly<dyn Quantizer> - Update serialization to use Quantizer::serialize/try_deserialize - Rewrite all tests and doc examples for spherical quantizer - Update benchmarks to use SphericalQuantizer::train with ReciprocalMeanNorm - Move rand to regular dependencies (needed by doc tests) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hildebrandmw · 2026-05-08T15:59:16Z

+//! bits. Bit `b` of word `w` corresponds to vector ID `w * 32 + b`. The number of words
+//! is `ceil(max_size / 32)`.
+
+use diskann_providers::model::graph::provider::async_::TableDeleteProviderAsync;


Weren't we getting rid of this?

this was from the earlier "migration only" phase of the changes. This'll be removed when I attack the D param removal.

hildebrandmw · 2026-05-08T16:05:47Z

+            .quantizer
+            .fused_query_computer(
+                &query_f32,
+                QueryLayout::FullPrecision,


One minor detail - for index construction, you might want to leave this SameAsData so the distances are comparable to the distances computed during pruning.

JordanMaples and others added 4 commits May 5, 2026 13:58

Remove bf_tree from CI feature list

d255a25

The bf_tree feature was removed from diskann-providers when the provider was extracted into its own crate (diskann-bf_tree-provider). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fmt

7c4f497

fix cargo.lock

44be934

JordanMaples force-pushed the jordanmaples/bf_tree_migration branch from b94f4df to 44be934 Compare May 5, 2026 20:58

JordanMaples and others added 2 commits May 7, 2026 15:49

move product logic from using pq to spherical

587f7a8

hildebrandmw reviewed May 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bf_tree migration away from diskann-providers#1020

bf_tree migration away from diskann-providers#1020
JordanMaples wants to merge 6 commits intomainfrom
jordanmaples/bf_tree_migration

JordanMaples commented May 5, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented May 5, 2026 •

edited

Loading

Uh oh!

hildebrandmw May 8, 2026

Uh oh!

JordanMaples May 8, 2026

Uh oh!

hildebrandmw May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

JordanMaples commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hildebrandmw May 8, 2026

Choose a reason for hiding this comment

Uh oh!

JordanMaples May 8, 2026

Choose a reason for hiding this comment

Uh oh!

hildebrandmw May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JordanMaples commented May 5, 2026 •

edited

Loading

codecov-commenter commented May 5, 2026 •

edited

Loading