Repository navigation

feat: adaptive placement of Parquet scan filters (row filter, post-scan, skip) - #25727

Draft

adriangb wants to merge 45 commits into

apache:mainfrom

pydantic:parquet-adaptive-placement-v2

Commits on Sep 24, 2026
- feat(parquet-datasource): always accept pushable filters, run rejected conjuncts post-scan
  
  adriangb
  and
  claude
  committed
  fcb0470
- Update tests added on main for always-accepted parquet filters
  
  adriangb
  and
  claude
  committed
  22ce8bf
Commits on Sep 25, 2026
- fix: stop the Parquet push decoder stream after the final flush
  
  adriangb
  and
  claude
  committed
  31199fc
Commits on Sep 26, 2026
- refactor(physical-plan): share the round-robin row check of EnforceDistribution
  
  adriangb
  and
  claude
  committed
  c2d8f85
- fix(datasource): keep a filter above a scan that cannot give the target partitions
  
  adriangb
  and
  claude
  committed
  9ff1146
- feat: add OptionalFilterPhysicalExpr to mark filters not needed for correctness
  
  adriangb
  and
  claude
  committed
  fe6aa04
- feat: keep optional filters pruning-only in the Parquet post-scan path
  
  adriangb
  and
  claude
  committed
  2246afc
- feat: add filter_stats (FilterCost, Clock) for filter measurements
  
  adriangb
  and
  claude
  committed
  ab2b6e2
- feat: add OptionalFilterGate to pause optional filters that cost more than they save
  
  adriangb
  and
  claude
  committed
  ab74ffa
- feat(gate): add a measured overhead for each evaluated row to the cost
  
  adriangb
  and
  claude
  committed
  6ae18bc
- feat(gate): share pauses between the gates of one plan site
  
  adriangb
  and
  claude
  committed
  ac838c4
- feat(gate): decide only on windows of at least MIN_OBSERVED_ROWS rows
  
  adriangb
  and
  claude
  committed
  36c73a0
- feat(gate): use the measured work of the filter producer as the saving
  
  adriangb
  and
  claude
  committed
  a352cb7
- feat: add optional_filter_mode config
  
  adriangb
  and
  claude
  committed
  ded1dd6
- feat: let the Parquet scan skip optional filters that are not worth their cost
  
  adriangb
  and
  claude
  committed
  0a51a93
- test: end-to-end tests for optional filters in the Parquet post-scan path
  
  adriangb
  and
  claude
  committed
  de1b263
- feat(parquet): adaptive placement of scan filter conjuncts
  
  adriangb
  and
  claude
  committed
  e616293
- test: sqllogictests for adaptive filter placement
  
  adriangb
  and
  claude
  committed
  1fbda42
- feat(parquet): carry row selections across placement rebuilds
  
  adriangb
  and
  claude
  committed
  b0088dc
- feat(parquet): measure skippable rows on the stage input
  
  adriangb
  and
  claude
  committed
  98259e5
- feat(parquet): start filters post-scan and add a row filter stage cost
  
  adriangb
  and
  claude
  committed
  ed19bdc
- fix(parquet): hand coalesced rows downstream at row group boundaries
  
  adriangb
  and
  claude
  committed
  9b24e9b
- fix(parquet): count the actual rows in optional_filter_rows_skipped
  
  adriangb
  and
  claude
  committed
  824dc27
- fix(parquet): count only skippable rows in the saving of an optional row filter
  
  adriangb
  and
  claude
  committed
  49b7bb1
- feat(parquet): share the pauses of optional filter gates in a scan
  
  adriangb
  and
  claude
  committed
  9af896f
- feat(parquet): place and order required and optional conjuncts with one model
  
  adriangb
  and
  claude
  committed
  e3fb0f0
- feat(parquet): add hysteresis to the placement changes of a file
  
  adriangb
  and
  claude
  committed
  aa00419
- feat(parquet): measure unmeasured conjuncts first and forget old filters
  
  adriangb
  and
  claude
  committed
  30ec574
- test(parquet): gate windows of MIN_OBSERVED_ROWS rows in row filter tests
  
  adriangb
  and
  claude
  committed
  3f9098e
- refactor(parquet): use the shared MIN_OBSERVED_ROWS in the placement model
  
  adriangb
  and
  claude
  committed
  15f1b67
- feat(parquet): let measured evidence decide the placement, drop the statistics prior
  
  adriangb
  and
  claude
  committed
  33a188a
- feat(parquet): order row filter predicates by measured rows removed per nanosecond
  
  adriangb
  and
  claude
  committed
  86531c1
- refactor(parquet): share the ranking and the hysteresis with the row filter order
  
  adriangb
  and
  claude
  committed
  0322fd6
- feat(parquet): add the measured post-scan copy to the benefit of a row filter
  
  adriangb
  and
  claude
  committed
  102627d
- test(parquet): size the optional filter integration tests for gate windows
  
  adriangb
  and
  claude
  committed
  b964d74
- feat(parquet): order the post-scan conjuncts by their measurements at each batch
  
  adriangb
  and
  claude
  committed
  48be981
- fix(parquet): fix rustdoc links of the adaptive filter placement
  
  adriangb
  and
  claude
  committed
  e6a798e
- refactor(physical-expr): make PRE_SELECTION_THRESHOLD public
  
  adriangb
  and
  claude
  committed
  4a3820e
- fix(parquet): compact the post-scan batch with the pre-selection threshold of AND
  
  adriangb
  and
  claude
  committed
  2627130
Commits on Sep 28, 2026
- Merge apache/main into parquet-post-scan-filter
  
  adriangb
  and
  claude
  committed
  8590f0c
- fix(parquet): no scan equivalences from a pruning-only predicate
  
  adriangb
  and
  claude
  committed
  a9aa655
- Merge the current #22384 into parquet-post-scan-optional
  
  adriangb
  and
  claude
  committed
  07096a0
- fix(parquet): no scan equivalences from optional conjuncts
  
  adriangb
  and
  claude
  committed
  4cd30d7
- Merge the current #25722 into parquet-adaptive-placement-v2
  
  adriangb
  and
  claude
  committed
  a11aa41
- docs(parquet): optional conjuncts of the placement are not exact
  
  adriangb
  and
  claude
  committed
  bd867a0

Commits

Commits on Sep 24, 2026

Commits on Sep 25, 2026

Commits on Sep 26, 2026

Commits on Sep 28, 2026