Repository navigation
Commit 102a162
authored
fix: report Parquet Bloom filter pruning metrics accurately (#25822)
## Which issue does this PR close?
- Closes #18355.
## Rationale for this change
`row_groups_pruned_bloom_filter` currently reports retained row groups
as Bloom filter matches even when Bloom pruning was not evaluated—for
example, when Bloom filters are disabled or unavailable, no relevant
Bloom statistics were loaded, or predicate evaluation failed.
This makes `EXPLAIN ANALYZE` suggest that Bloom filters evaluated and
matched row groups when they did not participate in the pruning
decision. It also displays an idle Bloom pruning metric when all its
counters are zero.
A retained row group is not necessarily a Bloom filter match. The metric
should describe actual Bloom pruning outcomes.
## What changes are included in this PR?
- Record a Bloom filter match only when the Bloom pruning predicate is
successfully evaluated and determines that the row group may match.
- Leave Bloom pruning counters unchanged when Bloom pruning is
unavailable or cannot be evaluated.
- Continue retaining row groups conservatively when Bloom predicate
evaluation fails, while recording the failure in
`predicate_evaluation_errors`.
- Omit `row_groups_pruned_bloom_filter` from physical-plan displays when
all of its pruning counters are zero.
- Preserve the registered metric and direct metric lookup; only its
accounting and displayed output change.
- Update affected Rust assertions and SQL logic test snapshots.
The row-group access decisions and query results are unchanged.
## What is the testing strategy for this PR?
Added `bloom_filter_pruning_error_retains_row_group_without_match`,
which verifies that a Bloom predicate evaluation error:
- retains the row group;
- increments `predicate_evaluation_errors`;
- does not increment either the Bloom matched or pruned counter.
The display tests verify that:
- an idle Bloom pruning metric is omitted from aggregated and full
Indent, Graphviz, and PostgreSQL JSON output;
- unrelated zero-valued metrics remain visible;
- genuine nonzero Bloom pruning results remain visible.
The following existing test coverage was also run:
- Parquet Bloom-filter and row-group-filter unit tests;
- `parquet_integration` row-group-pruning tests;
- `core_integration` explain-analyze tests;
- physical-plan display tests;
- affected SQL logic tests for dynamic filtering, dynamic row-group
pruning, explain-analyze, limit pruning, and Parquet filter pushdown.
Formatting checks also pass.
## Are there any user-facing changes?
Yes. Parquet Bloom filter pruning metrics in `EXPLAIN ANALYZE` and other
physical-plan displays more accurately represent actual Bloom filter
evaluation. An entirely idle `row_groups_pruned_bloom_filter` metric is
no longer displayed.
There are no public API changes and no changes to query results or
row-group access decisions.1 parent 97c7593 commit 102a162
10 files changed
Lines changed: 617 additions & 217 deletions
File tree
- datafusion
- core/tests
- parquet
- sql
- datasource-parquet/src
- opener
- physical-plan/src
- sqllogictest/test_files
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
880 | 880 | | |
881 | 881 | | |
882 | 882 | | |
883 | | - | |
884 | | - | |
885 | | - | |
886 | | - | |
| 883 | + | |
887 | 884 | | |
888 | 885 | | |
889 | 886 | | |
| |||
896 | 893 | | |
897 | 894 | | |
898 | 895 | | |
899 | | - | |
900 | | - | |
901 | 896 | | |
902 | 897 | | |
903 | 898 | | |
904 | 899 | | |
905 | 900 | | |
906 | | - | |
907 | | - | |
908 | | - | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
909 | 904 | | |
910 | 905 | | |
911 | 906 | | |
| |||
1179 | 1174 | | |
1180 | 1175 | | |
1181 | 1176 | | |
1182 | | - | |
| 1177 | + | |
1183 | 1178 | | |
1184 | 1179 | | |
1185 | 1180 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1336 | 1336 | | |
1337 | 1337 | | |
1338 | 1338 | | |
1339 | | - | |
1340 | | - | |
1341 | | - | |
1342 | | - | |
1343 | | - | |
1344 | | - | |
1345 | | - | |
1346 | | - | |
1347 | | - | |
1348 | 1339 | | |
1349 | 1340 | | |
| 1341 | + | |
1350 | 1342 | | |
1351 | 1343 | | |
1352 | 1344 | | |
1353 | 1345 | | |
1354 | 1346 | | |
1355 | | - | |
1356 | | - | |
1357 | | - | |
1358 | | - | |
1359 | 1347 | | |
1360 | 1348 | | |
1361 | 1349 | | |
| |||
3413 | 3401 | | |
3414 | 3402 | | |
3415 | 3403 | | |
| 3404 | + | |
| 3405 | + | |
| 3406 | + | |
| 3407 | + | |
| 3408 | + | |
| 3409 | + | |
| 3410 | + | |
| 3411 | + | |
| 3412 | + | |
| 3413 | + | |
| 3414 | + | |
| 3415 | + | |
| 3416 | + | |
| 3417 | + | |
| 3418 | + | |
| 3419 | + | |
| 3420 | + | |
| 3421 | + | |
| 3422 | + | |
| 3423 | + | |
| 3424 | + | |
| 3425 | + | |
| 3426 | + | |
| 3427 | + | |
| 3428 | + | |
| 3429 | + | |
3416 | 3430 | | |
3417 | 3431 | | |
3418 | 3432 | | |
| |||
4743 | 4757 | | |
4744 | 4758 | | |
4745 | 4759 | | |
| 4760 | + | |
| 4761 | + | |
| 4762 | + | |
| 4763 | + | |
| 4764 | + | |
| 4765 | + | |
| 4766 | + | |
| 4767 | + | |
| 4768 | + | |
| 4769 | + | |
4746 | 4770 | | |
4747 | 4771 | | |
4748 | 4772 | | |
| |||
4781 | 4805 | | |
4782 | 4806 | | |
4783 | 4807 | | |
| 4808 | + | |
| 4809 | + | |
| 4810 | + | |
| 4811 | + | |
4784 | 4812 | | |
4785 | 4813 | | |
4786 | 4814 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
472 | 472 | | |
473 | 473 | | |
474 | 474 | | |
475 | | - | |
| 475 | + | |
| 476 | + | |
476 | 477 | | |
477 | 478 | | |
478 | 479 | | |
| |||
481 | 482 | | |
482 | 483 | | |
483 | 484 | | |
484 | | - | |
485 | 485 | | |
486 | 486 | | |
487 | 487 | | |
| |||
493 | 493 | | |
494 | 494 | | |
495 | 495 | | |
496 | | - | |
| 496 | + | |
497 | 497 | | |
498 | 498 | | |
499 | 499 | | |
| |||
622 | 622 | | |
623 | 623 | | |
624 | 624 | | |
625 | | - | |
626 | | - | |
| 625 | + | |
627 | 626 | | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
628 | 631 | | |
629 | 632 | | |
630 | 633 | | |
| |||
1640 | 1643 | | |
1641 | 1644 | | |
1642 | 1645 | | |
1643 | | - | |
| 1646 | + | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
| 1658 | + | |
| 1659 | + | |
| 1660 | + | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
| 1679 | + | |
| 1680 | + | |
| 1681 | + | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
| 1686 | + | |
| 1687 | + | |
| 1688 | + | |
1644 | 1689 | | |
1645 | 1690 | | |
1646 | 1691 | | |
| |||
0 commit comments