From fcb04700c6469ad3e1c4e4fd9e62192ef7f02d43 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Wed, 23 Sep 2026 19:44:12 -0500
Subject: [PATCH 01/42] feat(parquet-datasource): always accept pushable
 filters, run rejected conjuncts post-scan
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Rebased onto main. The original commit 1 of this PR ("extract
DecoderProjection from build_stream") landed independently on main as
current `decoder_projection` / single-decoder + `rg_plan` model.

Two changes, both applied inside the parquet scan so the parent
`FilterExec` can be removed unconditionally for pushable filters:

1. Never drop conjuncts the `RowFilter` cannot place.
   `build_row_filter` previously `.flatten()`-ed away conjuncts that
   `FilterCandidateBuilder::build` rejected (whole-struct references,
   per-file physical-schema mismatches) and swallowed whole-build
   errors. By the time it runs, `try_pushdown_filters` has already
   removed the `FilterExec`, so those conjuncts were applied nowhere —
   wrong results. `build_row_filter` now returns
   `(Option<RowFilter>, Vec<rejected>)`, `RowFilterGenerator` exposes
   `rejected_conjuncts()`, and a whole-file build error routes every
   conjunct to the rejected list rather than relaxing the predicate.

2. Always accept pushable filters and run the remainder post-scan.
   `try_pushdown_filters` reports each pushable filter as accepted so the
   `FilterExec` is always removed; the scan owns the predicate. The
   opener routes conjuncts to two places, applying every one:
     - pushdown_filters=true  -> row-filterable conjuncts via the parquet
       `RowFilter`; rejected conjuncts via the in-scan post-scan filter.
     - pushdown_filters=false -> the whole predicate runs as a post-scan
       filter on decoded batches (behaviorally identical to `FilterExec`).

Implementation:
  - `DecoderProjection` (main's `decoder_projection` module) grows a
    `post_scan_conjuncts` parameter: it widens the decoder mask over
    (user projection ∪ post-scan filter columns), rebases the conjuncts
    onto the stream schema, and returns a `PostScanFilter` applied to
    every decoded batch with SQL `WHERE` semantics. Virtual-column
    conjuncts are stripped from the read-plan mask (they aren't file
    columns) but kept in the post-scan predicate, which sees the
    reader-appended virtual columns.
  - `PushDecoderStreamState` applies the post-scan filter in the decoded-
    batch arm, skips empty batches, and re-introduces a stream-level
    `remaining_limit` (main enforces LIMIT decoder-locally, which is
    unsafe once a post-scan filter can reject rows). The opener routes the
    limit to `remaining_limit` iff a post-scan filter is present.
  - New `post_scan_rows_pruned` / `post_scan_rows_matched` counters and
    `post_scan_filter_eval_time` on `ParquetFileMetrics`.

Tests:
  - `build_row_filter_surfaces_rejected_struct_conjunct` (row_filter.rs)
    asserts the rejected struct conjunct is returned, not dropped.
  - `rejected_struct_conjunct_runs_post_scan_not_dropped` (opener) is
    end-to-end: `s IS NOT NULL` over a struct column with pushdown on
    returns 2 (was 3 before the fix).
  - Parquet `.slt` files regenerated: `FilterExec` above `DataSourceExec`
    gone, predicate on the scan, `post_scan_rows_*` metrics on EXPLAIN
    ANALYZE. Opener / core insta / page_pruning assertions updated for the
    now-applied predicate.

Squashed follow-up commits (see their original messages on the
adriangb/parquet-post-scan-filter branch):
  - perf(parquet-datasource): narrow to the projector's columns before
    filtering
  - perf(parquet-datasource): coalesce post-scan filter output back to
    batch_size
  - perf(parquet-datasource): compact between conjuncts in the post-scan
    filter

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 datafusion/core/src/dataframe/parquet.rs      |   9 +-
 .../src/datasource/physical_plan/parquet.rs   |  77 ++-
 datafusion/core/src/datasource/view_test.rs   |   9 +-
 datafusion/core/tests/parquet/page_pruning.rs |  53 +-
 datafusion/core/tests/sql/explain_analyze.rs  |  17 +-
 .../benches/parquet_nested_filter_pushdown.rs |   6 +-
 .../benches/parquet_struct_filter_pushdown.rs |   6 +-
 .../src/decoder_projection.rs                 | 472 +++++++++++++++++-
 datafusion/datasource-parquet/src/metrics.rs  |  24 +
 .../datasource-parquet/src/opener/mod.rs      | 292 +++++++++--
 .../datasource-parquet/src/push_decoder.rs    | 174 ++++++-
 .../datasource-parquet/src/row_filter.rs      | 191 +++++--
 datafusion/datasource-parquet/src/source.rs   |  16 +-
 .../sqllogictest/test_files/clickbench.slt    | 144 ++----
 datafusion/sqllogictest/test_files/cte.slt    |   4 +-
 .../sqllogictest/test_files/explain_tree.slt  |  29 +-
 .../test_files/grouping_set_repartition.slt   |  25 +-
 .../test_files/monotonic_projection_test.slt  |   7 +-
 .../sqllogictest/test_files/parquet.slt       |  20 +-
 .../test_files/parquet_filter_pushdown.slt    |  20 +-
 .../parquet_max_row_group_bytes.slt           |  12 +-
 .../test_files/parquet_statistics.slt         |  35 +-
 .../test_files/preserve_file_partitioning.slt |  14 +-
 .../sqllogictest/test_files/projection.slt    |   5 +-
 .../test_files/projection_pushdown.slt        | 130 ++---
 .../test_files/repartition_scan.slt           |  14 +-
 .../repartition_subset_satisfaction.slt       |  10 +-
 27 files changed, 1279 insertions(+), 536 deletions(-)

diff --git a/datafusion/core/src/dataframe/parquet.rs b/datafusion/core/src/dataframe/parquet.rs
index dd751eb97779f..ae55f672b44db 100644
--- a/datafusion/core/src/dataframe/parquet.rs
+++ b/datafusion/core/src/dataframe/parquet.rs
@@ -162,9 +162,14 @@ mod tests {
             .select_columns(&["bool_col", "int_col"])?;
 
         let plan = df.explain(false, false)?.collect().await?;
-        // Filters all the way to Parquet
+        // Filters all the way to Parquet. The parquet scan now accepts the
+        // pushable filter unconditionally so the `FilterExec` is removed —
+        // the predicate appears as `predicate=` on the `DataSourceExec`.
         let formatted = pretty::pretty_format_batches(&plan)?.to_string();
-        assert!(formatted.contains("FilterExec: id@0 = 1"), "{formatted}");
+        assert!(
+            formatted.contains("predicate=id@0 = 1"),
+            "expected predicate=id@0 = 1 in {formatted}"
+        );
 
         Ok(())
     }
diff --git a/datafusion/core/src/datasource/physical_plan/parquet.rs b/datafusion/core/src/datasource/physical_plan/parquet.rs
index d562bbe8490f4..a8c766c1b020b 100644
--- a/datafusion/core/src/datasource/physical_plan/parquet.rs
+++ b/datafusion/core/src/datasource/physical_plan/parquet.rs
@@ -752,17 +752,12 @@ mod tests {
             .await
             .unwrap();
 
-        insta::assert_snapshot!(batches_to_sort_string(&read),@r"
-        +-----+----+----+
-        | c1  | c3 | c2 |
-        +-----+----+----+
-        |     |    |    |
-        |     | 10 | 1  |
-        |     | 20 |    |
-        |     | 20 | 2  |
-        | Foo | 10 |    |
-        | bar |    |    |
-        +-----+----+----+
+        insta::assert_snapshot!(batches_to_sort_string(&read),@"
+        +----+----+----+
+        | c1 | c3 | c2 |
+        +----+----+----+
+        |    | 20 | 2  |
+        +----+----+----+
         ");
     }
 
@@ -897,7 +892,9 @@ mod tests {
         assert_eq!(get_value(&metrics, "pushdown_rows_matched"), 0);
         assert_eq!(rt.batches.unwrap().len(), 0);
 
-        // Predicate should prune no row groups
+        // Predicate should prune no row groups. The scan now applies the
+        // predicate post-scan (since `pushdown_filters` is off in this test),
+        // so only the matching row survives.
         let filter = col("c1").eq(lit(ScalarValue::Utf8(Some("foo".to_string()))));
         let rt = RoundTrip::new()
             .with_predicate(filter)
@@ -914,7 +911,7 @@ mod tests {
             .iter()
             .map(|b| b.num_rows())
             .sum::<usize>();
-        assert_eq!(read, 2, "Expected 2 rows to match the predicate");
+        assert_eq!(read, 1, "Expected 1 row to match the predicate");
     }
 
     #[tokio::test]
@@ -938,7 +935,9 @@ mod tests {
         assert_eq!(get_value(&metrics, "predicate_evaluation_errors"), 0);
         assert_eq!(rt.batches.unwrap().len(), 0);
 
-        // Predicate should prune no row groups
+        // Predicate should prune no row groups. The scan now applies the
+        // predicate post-scan (since `pushdown_filters` is off in this test),
+        // so only the matching row survives.
         let filter = col("c1").eq(lit(ScalarValue::UInt64(Some(1))));
         let rt = RoundTrip::new()
             .with_predicate(filter)
@@ -954,7 +953,7 @@ mod tests {
             .iter()
             .map(|b| b.num_rows())
             .sum::<usize>();
-        assert_eq!(read, 2, "Expected 2 rows to match the predicate");
+        assert_eq!(read, 1, "Expected 1 row to match the predicate");
     }
 
     #[tokio::test]
@@ -1052,17 +1051,12 @@ mod tests {
         // In a real query where this predicate was pushed down from a filter stage instead of created directly in the `DataSourceExec`,
         // the filter stage would be preserved as a separate execution plan stage so the actual query results would be as expected.
 
-        insta::assert_snapshot!(batches_to_sort_string(&read),@r"
-        +-----+----+
-        | c1  | c2 |
-        +-----+----+
-        |     |    |
-        |     |    |
-        |     | 1  |
-        |     | 2  |
-        | Foo |    |
-        | bar |    |
-        +-----+----+
+        insta::assert_snapshot!(batches_to_sort_string(&read),@"
+        +----+----+
+        | c1 | c2 |
+        +----+----+
+        |    | 1  |
+        +----+----+
         ");
     }
 
@@ -1148,23 +1142,14 @@ mod tests {
             .round_trip(vec![batch1, batch2, batch3, batch4])
             .await;
 
-        insta::assert_snapshot!(batches_to_sort_string(&rt.batches.unwrap()), @r"
-        +------+----+
-        | c1   | c2 |
-        +------+----+
-        |      | 1  |
-        |      | 2  |
-        | Bar  |    |
-        | Bar  | 2  |
-        | Bar  | 2  |
-        | Bar2 |    |
-        | Bar3 |    |
-        | Foo  |    |
-        | Foo  | 1  |
-        | Foo  | 1  |
-        | Foo2 |    |
-        | Foo3 |    |
-        +------+----+
+        insta::assert_snapshot!(batches_to_sort_string(&rt.batches.unwrap()), @"
+        +-----+----+
+        | c1  | c2 |
+        +-----+----+
+        |     | 1  |
+        | Foo | 1  |
+        | Foo | 1  |
+        +-----+----+
         ");
         let metrics = rt.parquet_exec.metrics().unwrap();
 
@@ -1233,11 +1218,10 @@ mod tests {
             .await
             .unwrap();
 
-        insta::assert_snapshot!(batches_to_sort_string(&read),@r"
+        insta::assert_snapshot!(batches_to_sort_string(&read),@"
         +-----+----+
         | c1  | c2 |
         +-----+----+
-        |     | 2  |
         | Foo | 1  |
         | bar |    |
         +-----+----+
@@ -1756,12 +1740,11 @@ mod tests {
 
         let metrics = rt.parquet_exec.metrics().unwrap();
 
-        assert_snapshot!(batches_to_sort_string(&rt.batches.unwrap()),@r"
+        assert_snapshot!(batches_to_sort_string(&rt.batches.unwrap()),@"
         +-----+
         | int |
         +-----+
         | 4   |
-        | 5   |
         +-----+
         ");
         let (page_index_rows_pruned, page_index_rows_matched) =
diff --git a/datafusion/core/src/datasource/view_test.rs b/datafusion/core/src/datasource/view_test.rs
index 35418d6dea632..a5b21450d6619 100644
--- a/datafusion/core/src/datasource/view_test.rs
+++ b/datafusion/core/src/datasource/view_test.rs
@@ -322,11 +322,16 @@ mod tests {
 
         let plan = df.explain(false, false)?.collect().await?;
 
-        // Filters all the way to Parquet
+        // Filters all the way to Parquet. The parquet scan now accepts the
+        // pushable filter unconditionally so the `FilterExec` is removed —
+        // the predicate appears as `predicate=` on the `DataSourceExec`.
         let formatted = arrow::util::pretty::pretty_format_batches(&plan)
             .unwrap()
             .to_string();
-        assert!(formatted.contains("FilterExec: id@0 = 1"));
+        assert!(
+            formatted.contains("predicate=id@0 = 1"),
+            "expected predicate=id@0 = 1 in {formatted}"
+        );
         Ok(())
     }
 
diff --git a/datafusion/core/tests/parquet/page_pruning.rs b/datafusion/core/tests/parquet/page_pruning.rs
index 70052ec6b7d60..865429efe5d9c 100644
--- a/datafusion/core/tests/parquet/page_pruning.rs
+++ b/datafusion/core/tests/parquet/page_pruning.rs
@@ -120,13 +120,10 @@ async fn page_index_filter_one_col() {
     let filter = col("month").eq(lit(1_i32));
 
     let batches = get_filter_results(&state, filter.clone(), false).await;
-    // `month = 1` from the page index should create below RowSelection
-    //  vec.push(RowSelector::select(312));
-    //  vec.push(RowSelector::skip(3330));
-    //  vec.push(RowSelector::select(339));
-    //  vec.push(RowSelector::skip(3319));
-    // total 651 row
-    assert_eq!(batches[0].num_rows(), 651);
+    // `month = 1` from the page index narrows IO to ~651 candidate rows
+    // (RowSelector skip/select pattern), and the in-scan post-scan filter
+    // then rejects the 31 page-aligned rows that don't actually match.
+    assert_eq!(batches[0].num_rows(), 620);
 
     let batches = get_filter_results(&state, filter, true).await;
     assert_eq!(batches[0].num_rows(), 620);
@@ -134,16 +131,9 @@ async fn page_index_filter_one_col() {
     // 2. create filter month == 1 or month == 2;
     let filter = col("month").eq(lit(1_i32)).or(col("month").eq(lit(2_i32)));
     let batches = get_filter_results(&state, filter.clone(), false).await;
-    // `month = 1` or `month = 2` from the page index should create below RowSelection
-    //  vec.push(RowSelector::select(312));
-    //  vec.push(RowSelector::skip(900));
-    //  vec.push(RowSelector::select(312));
-    //  vec.push(RowSelector::skip(2118));
-    //  vec.push(RowSelector::select(339));
-    //  vec.push(RowSelector::skip(873));
-    //  vec.push(RowSelector::select(318));
-    //  vec.push(RowSelector::skip(2128));
-    assert_eq!(batches[0].num_rows(), 1281);
+    // Page-index pruning leaves a 1281-row candidate set; the scan applies
+    // the predicate (RowFilter or post-scan) to land on 1180 matches.
+    assert_eq!(batches[0].num_rows(), 1180);
 
     let batches = get_filter_results(&state, filter, true).await;
     assert_eq!(batches[0].num_rows(), 1180);
@@ -161,8 +151,8 @@ async fn page_index_filter_one_col() {
     // 4.create filter 0 < month < 2 ;
     let filter = col("month").gt(lit(0_i32)).and(col("month").lt(lit(2_i32)));
     let batches = get_filter_results(&state, filter.clone(), false).await;
-    // should same with `month = 1`
-    assert_eq!(batches[0].num_rows(), 651);
+    // Same matching set as `month = 1`.
+    assert_eq!(batches[0].num_rows(), 620);
     let batches = get_filter_results(&state, filter, true).await;
     assert_eq!(batches[0].num_rows(), 620);
 
@@ -170,14 +160,9 @@ async fn page_index_filter_one_col() {
     // Note this test doesn't apply type coercion so the literal must match the actual view type
     let filter = col("date_string_col").eq(lit(ScalarValue::new_utf8view("01/01/09")));
     let batches = get_filter_results(&state, filter.clone(), false).await;
-    assert_eq!(batches[0].num_rows(), 14);
-
-    // there should only two pages match the filter
-    //                                  min                                        max
-    // page-20                        0  01/01/09                                  01/02/09
-    // page-21                        0  01/01/09                                  01/01/09
-    // each 7 rows
-    assert_eq!(batches[0].num_rows(), 14);
+    // Page index narrows to two pages of 7 rows each (14 rows); the scan
+    // then rejects the 4 page-aligned rows that don't actually match.
+    assert_eq!(batches[0].num_rows(), 10);
     let batches = get_filter_results(&state, filter, true).await;
     assert_eq!(batches[0].num_rows(), 10);
 }
@@ -190,11 +175,9 @@ async fn page_index_filter_multi_col() {
     // create filter month == 1 and year = 2009;
     let filter = col("month").eq(lit(1_i32)).and(col("year").eq(lit(2009)));
     let batches = get_filter_results(&state, filter.clone(), false).await;
-    //  `year = 2009` from the page index should create below RowSelection
-    //  vec.push(RowSelector::select(3663));
-    //  vec.push(RowSelector::skip(3642));
-    //  combine with `month = 1` total 333 row
-    assert_eq!(batches[0].num_rows(), 333);
+    // Page index narrows IO to ~333 candidate rows; the scan then rejects
+    // the page-aligned rows that don't actually match, landing on 310.
+    assert_eq!(batches[0].num_rows(), 310);
     let batches = get_filter_results(&state, filter, true).await;
     assert_eq!(batches[0].num_rows(), 310);
 
@@ -204,7 +187,7 @@ async fn page_index_filter_multi_col() {
         .eq(lit(1_i32))
         .and(col("year").eq(lit(2009)).or(col("id").eq(lit(1))));
     let batches = get_filter_results(&state, filter.clone(), false).await;
-    assert_eq!(batches[0].num_rows(), 651);
+    assert_eq!(batches[0].num_rows(), 310);
     let batches = get_filter_results(&state, filter, true).await;
     assert_eq!(batches[0].num_rows(), 310);
 
@@ -212,7 +195,7 @@ async fn page_index_filter_multi_col() {
     // this filter use two columns will not push down
     let filter = col("year").eq(lit(2009)).or(col("id").eq(lit(1)));
     let batches = get_filter_results(&state, filter.clone(), false).await;
-    assert_eq!(batches[0].num_rows(), 7300);
+    assert_eq!(batches[0].num_rows(), 3650);
     let batches = get_filter_results(&state, filter, true).await;
     assert_eq!(batches[0].num_rows(), 3650);
 
@@ -225,7 +208,7 @@ async fn page_index_filter_multi_col() {
         .and(col("id").eq(lit(1)))
         .or(col("year").eq(lit(2010)));
     let batches = get_filter_results(&state, filter.clone(), false).await;
-    assert_eq!(batches[0].num_rows(), 7300);
+    assert_eq!(batches[0].num_rows(), 3651);
     let batches = get_filter_results(&state, filter, true).await;
     assert_eq!(batches[0].num_rows(), 3651);
 }
diff --git a/datafusion/core/tests/sql/explain_analyze.rs b/datafusion/core/tests/sql/explain_analyze.rs
index 67614cfbdea9a..aa355260446e8 100644
--- a/datafusion/core/tests/sql/explain_analyze.rs
+++ b/datafusion/core/tests/sql/explain_analyze.rs
@@ -877,8 +877,12 @@ async fn parquet_explain_analyze() {
         .unwrap()
         .to_string();
 
-    // should contain aggregated stats
-    assert_contains!(&formatted, "output_rows=8");
+    // should contain aggregated stats. The scan now applies the predicate
+    // post-scan (the `FilterExec` above it is removed), so `output_rows` is
+    // the matching row count (5) rather than the decoded row count (8).
+    // The 3 rejected rows show up as `post_scan_rows_pruned=3`.
+    assert_contains!(&formatted, "output_rows=5");
+    assert_contains!(&formatted, "post_scan_rows_pruned=3");
     assert_contains!(
         &formatted,
         "row_groups_pruned_bloom_filter=1 total \u{2192} 1 matched"
@@ -1069,14 +1073,13 @@ async fn parquet_recursive_projection_pushdown() -> Result<()> {
 
     assert_snapshot!(
         actual,
-        @r"
+        @"
     SortExec: expr=[id@0 ASC NULLS LAST], preserve_partitioning=[false]
       RecursiveQueryExec: name=number_series, is_distinct=false
         CoalescePartitionsExec
-          ProjectionExec: expr=[CAST(id@0 AS Int64) as id, CAST(1 AS Int64) as level]
-            FilterExec: id@0 = 1
-              RepartitionExec: partitioning=RoundRobinBatch(NUM_CORES), input_partitions=1
-                DataSourceExec: file_groups={1 group: [[TMP_DIR/hierarchy.parquet]]}, projection=[id], file_type=parquet, predicate=id@0 = 1, pruning_predicate=id_null_count@2 != row_count@3 AND id_min@0 <= 1 AND 1 <= id_max@1, required_guarantees=[id in (1)]
+          ProjectionExec: expr=[CAST(id@0 AS Int64) as id, CAST(level@1 AS Int64) as level]
+            RepartitionExec: partitioning=RoundRobinBatch(NUM_CORES), input_partitions=1
+              DataSourceExec: file_groups={1 group: [[TMP_DIR/hierarchy.parquet]]}, projection=[id, 1 as level], file_type=parquet, predicate=id@0 = 1, pruning_predicate=id_null_count@2 != row_count@3 AND id_min@0 <= 1 AND 1 <= id_max@1, required_guarantees=[id in (1)]
         CoalescePartitionsExec
           ProjectionExec: expr=[id@0 + 1 as id, level@1 + 1 as level]
             FilterExec: id@0 < 10
diff --git a/datafusion/datasource-parquet/benches/parquet_nested_filter_pushdown.rs b/datafusion/datasource-parquet/benches/parquet_nested_filter_pushdown.rs
index 02137b5a1d288..6a1ed3bdb25e2 100644
--- a/datafusion/datasource-parquet/benches/parquet_nested_filter_pushdown.rs
+++ b/datafusion/datasource-parquet/benches/parquet_nested_filter_pushdown.rs
@@ -115,9 +115,9 @@ fn scan_with_predicate(
     let file_metrics = ParquetFileMetrics::new(0, &path.display().to_string(), &metrics);
 
     let builder = if pushdown {
-        if let Some(row_filter) =
-            build_row_filter(predicate, file_schema, &metadata, false, &file_metrics)?
-        {
+        let (row_filter, _rejected) =
+            build_row_filter(predicate, file_schema, &metadata, false, &file_metrics)?;
+        if let Some(row_filter) = row_filter {
             builder.with_row_filter(row_filter)
         } else {
             builder
diff --git a/datafusion/datasource-parquet/benches/parquet_struct_filter_pushdown.rs b/datafusion/datasource-parquet/benches/parquet_struct_filter_pushdown.rs
index b52408d4222d8..17eab65983720 100644
--- a/datafusion/datasource-parquet/benches/parquet_struct_filter_pushdown.rs
+++ b/datafusion/datasource-parquet/benches/parquet_struct_filter_pushdown.rs
@@ -210,9 +210,9 @@ fn scan(
 
     let mut filter_applied = false;
     let builder = if pushdown {
-        if let Some(row_filter) =
-            build_row_filter(predicate, file_schema, &metadata, false, &file_metrics)?
-        {
+        let (row_filter, _rejected) =
+            build_row_filter(predicate, file_schema, &metadata, false, &file_metrics)?;
+        if let Some(row_filter) = row_filter {
             filter_applied = true;
             builder.with_row_filter(row_filter)
         } else {
diff --git a/datafusion/datasource-parquet/src/decoder_projection.rs b/datafusion/datasource-parquet/src/decoder_projection.rs
index 89fdc01af4eda..ac8499fa3f9c9 100644
--- a/datafusion/datasource-parquet/src/decoder_projection.rs
+++ b/datafusion/datasource-parquet/src/decoder_projection.rs
@@ -33,20 +33,201 @@
 
 use std::sync::Arc;
 
-use arrow::array::{RecordBatch, RecordBatchOptions};
+use arrow::array::{Array, BooleanArray, RecordBatch, RecordBatchOptions};
+use arrow::compute::kernels::boolean::and;
+use arrow::compute::kernels::filter::{filter_record_batch, prep_null_mask_filter};
 use arrow::datatypes::SchemaRef;
 
-use datafusion_common::Result;
+use datafusion_common::cast::as_boolean_array;
+use datafusion_common::{Result, internal_err};
 use datafusion_physical_expr::projection::{ProjectionExprs, Projector};
-use datafusion_physical_expr::utils::reassign_expr_columns;
+use datafusion_physical_expr::split_conjunction;
+use datafusion_physical_expr::utils::{collect_columns, reassign_expr_columns};
 use datafusion_physical_expr_adapter::replace_columns_with_literals;
+use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
+use datafusion_physical_plan::metrics::{Count, Time};
 
 use parquet::arrow::ProjectionMask;
 use parquet::schema::types::SchemaDescriptor;
 
+use crate::ParquetFileMetrics;
 use crate::opener::{VirtualColumnsState, append_fields};
 use crate::projection_read_plan::build_projection_read_plan;
 
+/// Stream-schema column indices referenced by `projection`, in ascending
+/// order, or `None` when the projection already reads every column of
+/// `stream_schema` (so narrowing to them would be an identity).
+///
+/// `projection` must already be rebased onto `stream_schema`.
+fn projector_input_indices(
+    projection: &ProjectionExprs,
+    stream_schema: &SchemaRef,
+) -> Option<Vec<usize>> {
+    let mut indices: Vec<usize> = projection
+        .expr_iter()
+        .flat_map(|expr| collect_columns(&expr))
+        .map(|col| col.index())
+        .collect();
+    indices.sort_unstable();
+    indices.dedup();
+
+    // Nothing to drop: keep the batch (and the projector) as-is. This is the
+    // no-post-scan-filter case, where the decoder mask is already exactly the
+    // projection.
+    (indices.len() < stream_schema.fields().len()).then_some(indices)
+}
+
+/// Compact the working batch to the surviving rows once a conjunct leaves at
+/// most this fraction of them alive. Above the threshold the copy would not be
+/// repaid by the (smaller) saving on the conjuncts that follow, so the masks are
+/// just combined with a cheap bitwise `AND` instead.
+///
+/// The threshold is higher than the equivalent constant in `FilterExec`'s
+/// adaptive evaluator because the post-scan filter's conjunct mix is skewed:
+/// the conjuncts that land here at `pushdown_filters = false` are typically a
+/// cheap static range predicate followed by a *much* more expensive dynamic
+/// filter (a `CASE` over per-partition hash-table probes), so even a modest
+/// reduction in the row count reaching the later conjunct pays for the copy.
+const COMPACTION_SELECTIVITY_THRESHOLD: f64 = 0.8;
+
+/// Outcome of running the post-scan predicate over one decoded batch.
+///
+/// The surviving rows are described rather than fully materialized so the
+/// caller can drop filter-only columns (see [`DecoderProjection::narrow`])
+/// before paying for the final filter kernel.
+pub(crate) enum PostScanSelection {
+    /// No row survived; nothing to hand downstream.
+    Empty,
+    /// Some rows survived. `batch` is the working batch — the input batch, or a
+    /// compacted copy of it when the loop compacted at least once — and `mask`
+    /// is the residual selection over `batch`'s rows (`None` when every row of
+    /// `batch` survived).
+    Rows {
+        batch: RecordBatch,
+        mask: Option<BooleanArray>,
+    },
+}
+
+/// Predicate applied to decoded record batches inside the parquet scan.
+///
+/// Semantically identical to a `FilterExec` over the scan: rows where the
+/// predicate is not `true` are dropped. `NULL` predicate results drop the row
+/// (SQL `WHERE` semantics — `filter_record_batch` treats null mask entries as
+/// false, same as `FilterExec`'s `batch_filter`).
+///
+/// # Compact-once evaluation
+///
+/// The predicate is kept split into its `AND` conjuncts and evaluated
+/// sequentially, compacting the working batch to the survivors whenever a
+/// conjunct proves selective enough. A fused `BinaryExpr` `AND` does *not*
+/// compact between conjuncts, so every conjunct is evaluated on ~every decoded
+/// row; with `pushdown_filters = false` the whole predicate lands here, which
+/// means an expensive dynamic filter would run on all decoded rows even though
+/// a cheap static conjunct ahead of it already rejected most of them. Compacting
+/// is what the parquet `RowFilter` path gets for free from arrow-rs (each
+/// conjunct is its own `ArrowPredicate`, applied against an accumulating
+/// `RowSelection`); this loop is the post-scan equivalent.
+///
+/// Holds metric handles so per-batch rows-pruned / matched / time accumulate
+/// into [`ParquetFileMetrics`] for `EXPLAIN ANALYZE`.
+pub(crate) struct PostScanFilter {
+    /// The `AND` conjuncts of the predicate, rebased onto the decoder's stream
+    /// schema, in the order they are evaluated. Never empty.
+    conjuncts: Vec<Arc<dyn PhysicalExpr>>,
+    rows_pruned: Count,
+    rows_matched: Count,
+    eval_time: Time,
+}
+
+impl PostScanFilter {
+    /// Evaluate the conjuncts against `batch` and describe the surviving rows.
+    ///
+    /// Takes the batch by value because a compaction replaces it; the caller
+    /// gets the working batch back inside [`PostScanSelection::Rows`].
+    ///
+    /// Note the batch handed back is still on the *full* stream schema (the
+    /// decoder mask widened for the predicate's columns): every conjunct may
+    /// need those columns, so narrowing to the projector's inputs must not
+    /// happen until the loop is done.
+    pub(crate) fn evaluate(&self, batch: RecordBatch) -> Result<PostScanSelection> {
+        // Scoped timer: stops on drop, so the early-return paths still record.
+        let _timer = self.eval_time.timer();
+
+        let input_rows = batch.num_rows();
+        if input_rows == 0 {
+            return Ok(PostScanSelection::Rows { batch, mask: None });
+        }
+
+        // `working` is what conjuncts are evaluated against; `acc` is the
+        // accumulated (null-free) selection over `working`'s rows since the last
+        // compaction, `None` meaning "all of them are still live".
+        let mut working = batch;
+        let mut acc: Option<BooleanArray> = None;
+
+        let last = self.conjuncts.len() - 1;
+        for (position, conjunct) in self.conjuncts.iter().enumerate() {
+            let rows_in = working.num_rows();
+            let array = conjunct.evaluate(&working)?.into_array(rows_in)?;
+            let Ok(mask) = as_boolean_array(array.as_ref()) else {
+                return internal_err!(
+                    "post-scan filter predicate did not evaluate to a BooleanArray"
+                );
+            };
+            // `filter_record_batch` treats a null mask entry as false, so a
+            // surviving row is one that is `true` and non-null.
+            let mask = match mask.nulls() {
+                Some(_) => prep_null_mask_filter(mask),
+                None => mask.clone(),
+            };
+            // An all-true conjunct leaves the accumulated selection untouched.
+            if mask.true_count() == rows_in {
+                continue;
+            }
+
+            let folded = match &acc {
+                None => mask,
+                Some(previous) => and(previous, &mask)?,
+            };
+            let alive = folded.true_count();
+            if alive == 0 {
+                self.record(input_rows, 0);
+                return Ok(PostScanSelection::Empty);
+            }
+
+            // Compaction only benefits the conjuncts that come *after* this
+            // one, so never compact on the last: its residual mask goes to the
+            // caller, which narrows away the filter-only columns before
+            // applying it.
+            if position < last
+                && (alive as f64) <= COMPACTION_SELECTIVITY_THRESHOLD * rows_in as f64
+            {
+                working = filter_record_batch(&working, &folded)?;
+                acc = None;
+            } else {
+                acc = Some(folded);
+            }
+        }
+
+        let survivors = match &acc {
+            Some(mask) => mask.true_count(),
+            None => working.num_rows(),
+        };
+        self.record(input_rows, survivors);
+        Ok(PostScanSelection::Rows {
+            batch: working,
+            mask: acc,
+        })
+    }
+
+    /// Record one batch's contribution to the rows-matched / rows-pruned
+    /// metrics. `rows_pruned` stays "total rows in, minus final survivors"
+    /// regardless of how many intermediate compactions the loop performed.
+    fn record(&self, input_rows: usize, survivors: usize) {
+        self.rows_matched.add(survivors);
+        self.rows_pruned.add(input_rows - survivors);
+    }
+}
+
 /// Per-file decoder projection: the [`ProjectionMask`] installed on the
 /// parquet decoder, plus the per-batch transform that maps the decoder's
 /// output onto the scan's `output_schema`.
@@ -63,6 +244,23 @@ pub(crate) struct DecoderProjection {
     /// in metadata / nullability and [`map`](Self::map) must rebuild the batch
     /// with `output_schema`.
     replace_schema: bool,
+    /// Predicate to apply on each decoded batch, after any row-level
+    /// `RowFilter` and before the projector. Carries conjuncts the `RowFilter`
+    /// machinery could not evaluate, plus the whole predicate when
+    /// `pushdown_filters = false`. `None` when no conjunct needs post-scan
+    /// evaluation, in which case the decoder mask covers exactly the user
+    /// projection and there is no extra per-batch work.
+    post_scan_filter: Option<PostScanFilter>,
+    /// Stream-schema column indices the [`Projector`] reads, used by
+    /// [`narrow`](Self::narrow) to drop filter-only columns before the filter
+    /// kernel runs. `None` when the projector already reads every stream
+    /// column — always the case without a post-scan filter, where the decoder
+    /// mask covers exactly the projection and narrowing would be an identity.
+    narrow_indices: Option<Vec<usize>>,
+    /// Schema of a narrowed, filtered batch: what [`map`](Self::map) consumes,
+    /// and the schema any coalescer buffering these batches must be built with.
+    /// Equals the full stream schema when [`Self::narrow_indices`] is `None`.
+    filtered_schema: SchemaRef,
 }
 
 impl DecoderProjection {
@@ -78,12 +276,25 @@ impl DecoderProjection {
     /// columns are stripped from the projection fed into
     /// `build_projection_read_plan` (which only understands file columns) and
     /// appended to the stream schema so the projector can resolve them.
+    ///
+    /// `post_scan_conjuncts` are predicate conjuncts that must be evaluated on
+    /// decoded batches inside the scan (conjuncts the parquet `RowFilter`
+    /// machinery could not place, plus the whole predicate when
+    /// `pushdown_filters = false`). They must reference columns in
+    /// `physical_file_schema` (virtual-column predicates are never pushed into
+    /// the scan). When non-empty the decoder mask is widened to include their
+    /// columns, the conjuncts are rebased onto the (widened) stream schema, and
+    /// they become [`Self::post_scan_filter`] — kept split so it can compact
+    /// between them. When empty this is exactly the prior projection-only
+    /// behaviour.
     pub(crate) fn try_new(
         projection: &ProjectionExprs,
+        post_scan_conjuncts: &[Arc<dyn PhysicalExpr>],
         physical_file_schema: &SchemaRef,
         parquet_schema: &SchemaDescriptor,
         output_schema: &SchemaRef,
         virtual_state: Option<&VirtualColumnsState>,
+        file_metrics: &ParquetFileMetrics,
     ) -> Result<Self> {
         // Virtual columns are produced by the reader separately from the
         // projection mask, so strip them from the expressions we feed into
@@ -97,8 +308,33 @@ impl DecoderProjection {
                 replace_columns_with_literals(expr, state.null_replacements())
             })?,
         };
+        // Decoder reads (user projection ∪ post-scan filter columns). Row-level
+        // filter columns live inside the parquet RowFilter's per-predicate
+        // masks, so they don't need to be in this read plan.
+        //
+        // A post-scan conjunct may reference a virtual column (e.g. parquet
+        // `row_number`): the reader produces those separately, so — like the
+        // projection — strip them to null literals before feeding the read
+        // plan, which only understands file columns. The *original* conjuncts
+        // (with the virtual references intact) are still used below to build
+        // the post-scan predicate, which is rebased onto the stream schema
+        // where the reader has appended the virtual columns.
+        let post_scan_for_read_plan: Vec<Arc<dyn PhysicalExpr>> = match virtual_state {
+            None => post_scan_conjuncts.to_vec(),
+            Some(state) => post_scan_conjuncts
+                .iter()
+                .map(|expr| {
+                    replace_columns_with_literals(
+                        Arc::clone(expr),
+                        state.null_replacements(),
+                    )
+                })
+                .collect::<Result<Vec<_>>>()?,
+        };
         let read_plan = build_projection_read_plan(
-            projection_for_read_plan.expr_iter(),
+            projection_for_read_plan
+                .expr_iter()
+                .chain(post_scan_for_read_plan.iter().map(Arc::clone)),
             physical_file_schema,
             parquet_schema,
         );
@@ -118,18 +354,67 @@ impl DecoderProjection {
         let rebased_projection = projection
             .clone()
             .try_map_exprs(|expr| reassign_expr_columns(expr, &stream_schema))?;
-        let projector = rebased_projection.make_projector(&stream_schema)?;
+
+        // When the mask was widened for post-scan filter columns, the stream
+        // batch carries columns the projector never reads. Filtering those
+        // through the (expensive) filter kernel only to drop them afterwards
+        // is pure waste, so narrow the batch to the projector's inputs first
+        // and rebase the projector onto that narrower schema — the same
+        // project-then-filter order `FilterExec::filter_and_project` uses.
+        let narrow_indices = projector_input_indices(&rebased_projection, &stream_schema);
+        let (projector, narrow_indices, filtered_schema) = match narrow_indices {
+            Some(indices) => {
+                let narrowed = Arc::new(stream_schema.project(&indices)?);
+                let renarrowed = rebased_projection
+                    .clone()
+                    .try_map_exprs(|expr| reassign_expr_columns(expr, &narrowed))?;
+                let projector = renarrowed.make_projector(&narrowed)?;
+                (projector, Some(indices), narrowed)
+            }
+            None => (
+                rebased_projection.make_projector(&stream_schema)?,
+                None,
+                Arc::clone(&stream_schema),
+            ),
+        };
 
         // Compare against the projector's *output* schema rather than the
-        // stream schema, so future widening of the mask (e.g. for post-scan
-        // filter columns) does not flip this flag.
+        // (possibly widened) stream schema, so widening the mask for post-scan
+        // filter columns does not flip this flag.
         let replace_schema = projector.output_schema() != output_schema;
 
+        // Rebase the post-scan conjuncts onto the same (widened) stream schema
+        // and conjoin them into a single predicate for per-batch evaluation.
+        let post_scan_filter = if post_scan_conjuncts.is_empty() {
+            None
+        } else {
+            // Split each conjunct again after rebasing: the caller's list is
+            // already conjunct-wise, but a single entry may still be a nested
+            // `AND` (e.g. a `RowFilter`-rejected conjunct). The compact-once
+            // loop can only compact between the pieces it can see.
+            let rebased = post_scan_conjuncts
+                .iter()
+                .map(|expr| reassign_expr_columns(Arc::clone(expr), &stream_schema))
+                .collect::<Result<Vec<_>>>()?
+                .iter()
+                .flat_map(|expr| split_conjunction(expr).into_iter().map(Arc::clone))
+                .collect::<Vec<_>>();
+            Some(PostScanFilter {
+                conjuncts: rebased,
+                rows_pruned: file_metrics.post_scan_rows_pruned.clone(),
+                rows_matched: file_metrics.post_scan_rows_matched.clone(),
+                eval_time: file_metrics.post_scan_filter_eval_time.clone(),
+            })
+        };
+
         Ok(Self {
             projection_mask: read_plan.projection_mask,
             projector,
             output_schema: Arc::clone(output_schema),
             replace_schema,
+            post_scan_filter,
+            narrow_indices,
+            filtered_schema,
         })
     }
 
@@ -138,6 +423,43 @@ impl DecoderProjection {
         &self.projection_mask
     }
 
+    /// The post-scan filter for this file, if any conjunct needs per-batch
+    /// evaluation. Applied by the push-decoder stream to each decoded batch
+    /// (after any row-level `RowFilter`, before the projector).
+    pub(crate) fn post_scan_filter(&self) -> Option<&PostScanFilter> {
+        self.post_scan_filter.as_ref()
+    }
+
+    /// Drop the columns only the post-scan filter needed, leaving just what
+    /// the [`Projector`] reads.
+    ///
+    /// Call this *after* [`PostScanFilter::evaluate`] (every conjunct may need
+    /// the filter-only columns) but *before* applying the residual mask it
+    /// returned: `RecordBatch::project` is a cheap `Arc` reslice, so narrowing
+    /// first keeps the final filter kernel off columns that would be discarded
+    /// immediately afterwards. Returns the batch unchanged when the projector
+    /// already reads every stream column.
+    pub(crate) fn narrow(&self, batch: RecordBatch) -> Result<RecordBatch> {
+        match &self.narrow_indices {
+            Some(indices) => Ok(batch.project(indices)?),
+            None => Ok(batch),
+        }
+    }
+
+    /// Schema of the batches [`map`](Self::map) consumes, i.e. what
+    /// [`narrow`](Self::narrow) produces. Used to build the coalescer that
+    /// reassembles post-filter batches back to the target batch size.
+    pub(crate) fn filtered_schema(&self) -> &SchemaRef {
+        &self.filtered_schema
+    }
+
+    /// Whether this file has a post-scan filter. Used by the opener to decide
+    /// whether a decoder-local LIMIT is safe (it is not, because the filter
+    /// can reject rows after the decoder counts them).
+    pub(crate) fn has_post_scan_filter(&self) -> bool {
+        self.post_scan_filter.is_some()
+    }
+
     /// Map a decoded batch onto the scan's output schema.
     ///
     /// Applies the [`Projector`] and, when the projector's output schema
@@ -159,3 +481,139 @@ impl DecoderProjection {
         )?)
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    use arrow::array::Int32Array;
+    use arrow::datatypes::{DataType, Field, Schema};
+    use datafusion_common::ScalarValue;
+    use datafusion_expr::Operator;
+    use datafusion_physical_expr::expressions::{BinaryExpr, Column, Literal};
+
+    fn batch(values: Vec<Option<i32>>) -> RecordBatch {
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("a", DataType::Int32, true),
+            Field::new("b", DataType::Int32, true),
+        ]));
+        let a = Int32Array::from(values.clone());
+        // `b` mirrors `a` so both conjuncts can be expressed over real data
+        // while still exercising two separate columns.
+        let b = Int32Array::from(values);
+        RecordBatch::try_new(schema, vec![Arc::new(a), Arc::new(b)]).unwrap()
+    }
+
+    fn gt(column: &str, index: usize, value: i32) -> Arc<dyn PhysicalExpr> {
+        Arc::new(BinaryExpr::new(
+            Arc::new(Column::new(column, index)),
+            Operator::Gt,
+            Arc::new(Literal::new(ScalarValue::Int32(Some(value)))),
+        ))
+    }
+
+    fn filter(conjuncts: Vec<Arc<dyn PhysicalExpr>>) -> PostScanFilter {
+        PostScanFilter {
+            conjuncts,
+            rows_pruned: Count::new(),
+            rows_matched: Count::new(),
+            eval_time: Time::new(),
+        }
+    }
+
+    /// Apply the selection the way the push-decoder does, so the test asserts
+    /// on the rows that actually reach the coalescer.
+    fn survivors(filter: &PostScanFilter, input: RecordBatch) -> Vec<Option<i32>> {
+        match filter.evaluate(input).unwrap() {
+            PostScanSelection::Empty => Vec::new(),
+            PostScanSelection::Rows { batch, mask } => {
+                let batch = match mask {
+                    Some(mask) => filter_record_batch(&batch, &mask).unwrap(),
+                    None => batch,
+                };
+                batch
+                    .column(0)
+                    .as_any()
+                    .downcast_ref::<Int32Array>()
+                    .unwrap()
+                    .iter()
+                    .collect()
+            }
+        }
+    }
+
+    #[test]
+    fn compacts_between_conjuncts_without_losing_rows() {
+        // 100 rows; `a > 79` keeps 20 (20% — below the compaction threshold, so
+        // the loop compacts), then `b > 89` keeps 10 of those.
+        let input = batch((0..100).map(Some).collect());
+        let f = filter(vec![gt("a", 0, 79), gt("b", 1, 89)]);
+        assert_eq!(
+            survivors(&f, input),
+            (90..100).map(Some).collect::<Vec<_>>()
+        );
+        assert_eq!(f.rows_matched.value(), 10);
+        assert_eq!(f.rows_pruned.value(), 90);
+    }
+
+    #[test]
+    fn conjunct_order_does_not_change_the_result() {
+        let a = filter(vec![gt("a", 0, 79), gt("b", 1, 89)]);
+        let b = filter(vec![gt("b", 1, 89), gt("a", 0, 79)]);
+        let expected = (90..100).map(Some).collect::<Vec<_>>();
+        assert_eq!(survivors(&a, batch((0..100).map(Some).collect())), expected);
+        assert_eq!(survivors(&b, batch((0..100).map(Some).collect())), expected);
+    }
+
+    /// A `NULL` predicate result drops the row, matching `filter_record_batch`
+    /// and `FilterExec`. The null must be dropped even when it is the *first*
+    /// conjunct that produced it and a later conjunct would have said `true`.
+    #[test]
+    fn null_predicate_results_drop_the_row() {
+        let values: Vec<Option<i32>> = (0..100)
+            .map(|i| if i == 95 { None } else { Some(i) })
+            .collect();
+        let f = filter(vec![gt("a", 0, 79), gt("b", 1, 89)]);
+        let expected: Vec<Option<i32>> =
+            (90..100).filter(|i| *i != 95).map(Some).collect();
+        assert_eq!(survivors(&f, batch(values)), expected);
+        assert_eq!(f.rows_matched.value(), 9);
+        assert_eq!(f.rows_pruned.value(), 91);
+    }
+
+    #[test]
+    fn all_rows_rejected_reports_empty() {
+        let input = batch((0..100).map(Some).collect());
+        let f = filter(vec![gt("a", 0, 500), gt("b", 1, 0)]);
+        assert!(matches!(
+            f.evaluate(input).unwrap(),
+            PostScanSelection::Empty
+        ));
+        assert_eq!(f.rows_matched.value(), 0);
+        assert_eq!(f.rows_pruned.value(), 100);
+    }
+
+    #[test]
+    fn all_rows_kept_needs_no_mask() {
+        let input = batch((0..100).map(Some).collect());
+        let f = filter(vec![gt("a", 0, -1), gt("b", 1, -1)]);
+        match f.evaluate(input).unwrap() {
+            PostScanSelection::Rows { batch, mask } => {
+                assert!(mask.is_none(), "an all-true predicate needs no mask");
+                assert_eq!(batch.num_rows(), 100);
+            }
+            PostScanSelection::Empty => panic!("expected every row to survive"),
+        }
+        assert_eq!(f.rows_matched.value(), 100);
+        assert_eq!(f.rows_pruned.value(), 0);
+    }
+
+    #[test]
+    fn empty_batch_is_passed_through() {
+        let input = batch(Vec::new());
+        let f = filter(vec![gt("a", 0, 79)]);
+        assert_eq!(survivors(&f, input), Vec::<Option<i32>>::new());
+        assert_eq!(f.rows_matched.value(), 0);
+        assert_eq!(f.rows_pruned.value(), 0);
+    }
+}
diff --git a/datafusion/datasource-parquet/src/metrics.rs b/datafusion/datasource-parquet/src/metrics.rs
index b82bd54839f4f..4abadd2c4e13a 100644
--- a/datafusion/datasource-parquet/src/metrics.rs
+++ b/datafusion/datasource-parquet/src/metrics.rs
@@ -94,6 +94,14 @@ pub struct ParquetFileMetrics {
     pub pushdown_rows_matched: Count,
     /// Total time spent evaluating row-level pushdown filters
     pub row_pushdown_eval_time: Time,
+    /// Total rows filtered out by the in-scan post-scan filter
+    /// (predicate conjuncts that could not be applied as a parquet
+    /// `RowFilter` and were instead evaluated on decoded batches).
+    pub post_scan_rows_pruned: Count,
+    /// Total rows that passed the in-scan post-scan filter.
+    pub post_scan_rows_matched: Count,
+    /// Total time spent evaluating the in-scan post-scan filter.
+    pub post_scan_filter_eval_time: Time,
     /// Total time spent evaluating row group-level statistics filters
     pub statistics_eval_time: Time,
     /// Total time spent evaluating row group Bloom Filters
@@ -267,6 +275,19 @@ impl ParquetFileMetrics {
         let row_pushdown_eval_time = builder
             .clone()
             .subset_time("row_pushdown_eval_time", partition);
+
+        let post_scan_rows_pruned = builder
+            .clone()
+            .with_category(MetricCategory::Rows)
+            .counter("post_scan_rows_pruned", partition);
+        let post_scan_rows_matched = builder
+            .clone()
+            .with_category(MetricCategory::Rows)
+            .counter("post_scan_rows_matched", partition);
+        let post_scan_filter_eval_time = builder
+            .clone()
+            .subset_time("post_scan_filter_eval_time", partition);
+
         let statistics_eval_time = builder
             .clone()
             .subset_time("statistics_eval_time", partition);
@@ -307,6 +328,9 @@ impl ParquetFileMetrics {
             pushdown_rows_pruned,
             pushdown_rows_matched,
             row_pushdown_eval_time,
+            post_scan_rows_pruned,
+            post_scan_rows_matched,
+            post_scan_filter_eval_time,
             statistics_eval_time,
             bloom_filter_eval_time,
             page_index_rows_pruned,
diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index 1f4a09a58b2d6..4b97fcd66e657 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -38,6 +38,7 @@ use crate::{
     apply_file_schema_type_coercions,
 };
 use arrow::array::RecordBatch;
+use arrow::compute::BatchCoalescer;
 use arrow::datatypes::DataType;
 use datafusion_datasource::morsel::{Morsel, MorselPlan, MorselPlanner, Morselizer};
 use datafusion_physical_expr::projection::ProjectionExprs;
@@ -540,37 +541,63 @@ impl DecoderReadPlans {
         prepared: &PreparedParquetOpen,
         metadata: &ArrowReaderMetadata,
     ) -> Result<Self> {
-        // Build the decoder projection (mask + per-batch transform) in a
-        // single call. Encapsulating it behind `DecoderProjection` keeps the
-        // opener's orchestration body focused on filter / decoder / stream
-        // wiring. The file-column projection excludes virtual columns and
-        // respects nested field projections.
+        // ---------------------------------------------------------------
+        // Filter placement
+        //
+        // The scan accepts every pushable filter (the parent `FilterExec`
+        // is gone), so the predicate must be applied here. Each conjunct is
+        // routed to one of two places:
+        //
+        // * the parquet `RowFilter` (during decode, only when
+        //   `pushdown_filters = true`), or
+        // * the in-scan post-scan filter (otherwise, plus any conjunct the
+        //   `RowFilter` machinery cannot evaluate on this file — the rejected
+        //   conjuncts returned by `RowFilterContext::try_new`).
+        //
+        // Either way every conjunct is applied; nothing is silently dropped.
+        // ---------------------------------------------------------------
+        let (row_filter_context, post_scan_conjuncts) =
+            match (prepared.pushdown_filters, prepared.predicate.as_ref()) {
+                // Pushdown enabled: precompute the candidate list once per file.
+                // Both the initial `RowFilter` and any per-RG rebuilds (via
+                // `RowFilterContext::build_row_filter`) reuse it, so tree walks
+                // (`reassign_expr_columns`) and column resolution only run once —
+                // not once per row group. Only what the `RowFilter` could not
+                // place falls through to post-scan.
+                (true, Some(predicate)) => RowFilterContext::try_new(
+                    predicate,
+                    &prepared.physical_file_schema,
+                    metadata.metadata(),
+                    prepared.reorder_predicates,
+                    prepared.file_metrics.clone(),
+                    prepared.max_predicate_cache_size,
+                ),
+                // Pushdown disabled: the whole predicate runs post-scan (in-scan
+                // equivalent of a `FilterExec`).
+                (false, Some(predicate)) => (
+                    None,
+                    datafusion_physical_expr::split_conjunction(predicate)
+                        .into_iter()
+                        .cloned()
+                        .collect(),
+                ),
+                (_, None) => (None, Vec::new()),
+            };
+
+        // Build the decoder projection (mask + per-batch transform + optional
+        // post-scan filter) in a single call. Encapsulating it behind
+        // `DecoderProjection` keeps the opener's orchestration body focused on
+        // filter / decoder / stream wiring. The file-column projection
+        // excludes virtual columns and respects nested field projections.
         let projection = DecoderProjection::try_new(
             &prepared.projection,
+            &post_scan_conjuncts,
             &prepared.physical_file_schema,
             metadata.parquet_schema(),
             &prepared.output_schema,
             prepared.virtual_state.as_deref(),
+            &prepared.file_metrics,
         )?;
-        let pushdown_predicate = prepared
-            .pushdown_filters
-            .then_some(prepared.predicate.as_ref())
-            .flatten();
-        // Precompute the candidate list once per file. Both the initial
-        // `RowFilter` and any per-RG rebuilds (via
-        // `RowFilterContext::build_row_filter`) reuse it, so tree walks
-        // (`reassign_expr_columns`) and column resolution only run once —
-        // not once per row group.
-        let row_filter_context = pushdown_predicate.and_then(|predicate| {
-            RowFilterContext::try_new(
-                predicate,
-                &prepared.physical_file_schema,
-                metadata.metadata(),
-                prepared.reorder_predicates,
-                prepared.file_metrics.clone(),
-                prepared.max_predicate_cache_size,
-            )
-        });
         Ok(Self {
             projection,
             row_filter_context,
@@ -1711,6 +1738,14 @@ impl RowGroupsPrunedParquetOpen {
                 prepared.partition_index,
                 &prepared.file_name,
             );
+        // Decoder-local LIMIT is only safe when no post-decode work can reject
+        // rows. A post-scan filter can — so when one is present the limit is
+        // enforced at the stream level via `remaining_limit` and kept out of
+        // the decoder; otherwise it is pushed into the decoder.
+        let has_post_scan_filter = decoder_projection.has_post_scan_filter();
+        let decoder_limit = prepared.limit.filter(|_| !has_post_scan_filter);
+        let remaining_limit = prepared.limit.filter(|_| has_post_scan_filter);
+
         let InitialDecoderState {
             decoder,
             rg_plan,
@@ -1730,7 +1765,7 @@ impl RowGroupsPrunedParquetOpen {
                 batch_size: prepared.batch_size,
                 arrow_reader_metrics: &arrow_reader_metrics,
                 force_filter_selections: prepared.force_filter_selections,
-                decoder_limit: prepared.limit,
+                decoder_limit,
             };
 
             let prepared_access_plan = prepare_access_plan(access_plan)?;
@@ -1870,6 +1905,10 @@ impl RowGroupsPrunedParquetOpen {
             .file_metrics
             .row_groups_pruned_dynamic_filter
             .clone();
+
+        // Captured before `decoder_projection` is moved into the stream state.
+        let filtered_schema = Arc::clone(decoder_projection.filtered_schema());
+
         let stream = PushDecoderStreamState {
             decoder: Some(decoder),
             active_reader: None,
@@ -1886,6 +1925,12 @@ impl RowGroupsPrunedParquetOpen {
             filter_installed,
             row_filter_skipped_fully_matched,
             byte_progress,
+            remaining_limit,
+            // A post-scan filter can leave only a few rows per decoded batch;
+            // reassemble them so the operator above sees full-size batches.
+            batch_coalescer: has_post_scan_filter
+                .then(|| BatchCoalescer::new(filtered_schema, prepared.batch_size)),
+            flushed: false,
         }
         .into_stream();
 
@@ -3164,14 +3209,17 @@ mod test {
                 .build()
         };
 
-        // A filter on "a" should not exclude any rows even if it matches the data
+        // A filter on "a" cannot be excluded by file-level stats (no stats on
+        // column 0). The scan now accepts the filter and applies it post-scan
+        // (in-scan equivalent of `FilterExec`), so only the matching row
+        // survives.
         let expr = col("a").eq(lit(1));
         let predicate = logical2physical(&expr, &schema);
         let opener = make_opener(predicate);
         let stream = open_file(&opener, file.clone()).await.unwrap();
         let (num_batches, num_rows) = count_batches_and_rows(stream).await;
         assert_eq!(num_batches, 1);
-        assert_eq!(num_rows, 3);
+        assert_eq!(num_rows, 1);
 
         // A filter on `b = 5.0` should exclude all rows
         let expr = col("b").eq(lit(ScalarValue::Float32(Some(5.0))));
@@ -3292,14 +3340,16 @@ mod test {
                 .build()
         };
 
-        // Filter should match the partition value and file statistics
+        // Filter should match the partition value and file statistics (i.e. no
+        // file-level pruning). The scan now accepts the filter and applies it
+        // post-scan, leaving only the single row where `b = 1.0`.
         let expr = col("part").eq(lit(1)).and(col("b").eq(lit(1.0)));
         let predicate = logical2physical(&expr, &table_schema);
         let opener = make_opener(predicate);
         let stream = open_file(&opener, file.clone()).await.unwrap();
         let (num_batches, num_rows) = count_batches_and_rows(stream).await;
         assert_eq!(num_batches, 1);
-        assert_eq!(num_rows, 3);
+        assert_eq!(num_rows, 1);
 
         // Should prune based on partition value but not file statistics
         let expr = col("part").eq(lit(2)).and(col("b").eq(lit(1.0)));
@@ -3816,10 +3866,12 @@ mod test {
             Ok(count_batches_and_rows(stream).await)
         };
 
+        // The scan accepts the `a = 1` filter and applies it (RowFilter or
+        // post-scan), so only the matching row survives (data is a=[1, 2, 2]).
         let (num_batches, num_rows) =
             query_file(schema.clone()).await.expect("query_file");
         assert_eq!(num_batches, 1);
-        assert_eq!(num_rows, 3);
+        assert_eq!(num_rows, 1);
 
         let mismatching_schema = Schema::new(vec![
             Field::new("a", DataType::Int32, true),
@@ -4560,13 +4612,17 @@ mod test {
         )
         .await;
 
+        // The scan now always applies the predicate (RowFilter or post-scan),
+        // so both paths return the same matching rows. The page index only
+        // affects IO — it decides whether the 90 non-matching rows are
+        // physically read before being rejected.
         assert_eq!(
             rows_with_page_index, 10,
             "page index should prune 9 of 10 pages"
         );
         assert_eq!(
-            rows_without_page_index, 100,
-            "without page index all rows are returned"
+            rows_without_page_index, 10,
+            "without page index the post-scan filter still rejects non-matching rows"
         );
     }
 
@@ -5229,6 +5285,167 @@ mod test {
         assert_eq!(values, vec![7, 4, 5, 6, 3]);
     }
 
+    /// A selective post-scan filter must not fragment the output stream.
+    ///
+    /// The decoder hands back one batch per `batch_size` rows; a predicate
+    /// matching a single row in each of them used to be emitted as one
+    /// sliver-sized batch per decoded batch, leaving every operator above the
+    /// scan to pay per-batch overhead on batches of one row. `FilterExec`
+    /// coalesces for exactly this reason, and the in-scan filter now does too.
+    #[tokio::test]
+    async fn selective_post_scan_filter_coalesces_output_batches() {
+        use arrow::array::Int32Array;
+
+        let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+        let schema =
+            Arc::new(Schema::new(vec![Field::new("id", DataType::Int32, false)]));
+
+        // 8 write batches of 1000 rows. The default test `batch_size` is 1024,
+        // so the decoder yields several batches regardless of how the writer
+        // chunked the data.
+        let batches: Vec<RecordBatch> = (0..8)
+            .map(|chunk: i32| {
+                let base = chunk * 1000;
+                RecordBatch::try_new(
+                    Arc::clone(&schema),
+                    vec![Arc::new(Int32Array::from(
+                        (base..base + 1000).collect::<Vec<i32>>(),
+                    ))],
+                )
+                .unwrap()
+            })
+            .collect();
+
+        let data_size =
+            write_parquet_batches(Arc::clone(&store), "coalesce.parquet", batches, None)
+                .await;
+        let file = PartitionedFile::new("coalesce.parquet".to_string(), data_size as u64);
+
+        // One match per ~1024-row decoded batch, spread across the file, so an
+        // uncoalesced stream would emit one batch per match.
+        let matches = [0i32, 1500, 3000, 4500, 6000, 7500];
+        let predicate = logical2physical(
+            &matches
+                .iter()
+                .map(|v| col("id").eq(lit(*v)))
+                .reduce(|acc, e| acc.or(e))
+                .unwrap(),
+            &schema,
+        );
+
+        let morselizer = ParquetMorselizerBuilder::new()
+            .with_store(Arc::clone(&store))
+            .with_schema(Arc::clone(&schema))
+            .with_predicate(predicate)
+            // pushdown_filters=false routes the whole predicate to the
+            // post-scan filter — the path this test is about.
+            .with_pushdown_filters(false)
+            .build();
+
+        let stream = open_file(&morselizer, file).await.unwrap();
+        let (num_batches, num_rows) = count_batches_and_rows(stream).await;
+
+        assert_eq!(num_rows, matches.len(), "every match must survive");
+        // All survivors fit in one target-size batch. Without coalescing this
+        // was one batch per match.
+        assert_eq!(
+            num_batches, 1,
+            "expected the survivors to be coalesced into a single batch, got \
+             {num_batches} batches for {num_rows} rows"
+        );
+    }
+
+    /// End-to-end regression test for the "drop-on-floor" bug fixed by
+    /// `build_row_filter` now returning rejected conjuncts and the opener
+    /// routing them to the post-scan filter.
+    ///
+    /// Setup: a parquet file with a struct column where some rows have a NULL
+    /// struct. Predicate `s IS NOT NULL` is set on the source with
+    /// `pushdown_filters = true`. `ParquetSource::try_pushdown_filters` would
+    /// have already removed the parent `FilterExec` (the conjunct is pushable
+    /// at table schema level). Inside `build_row_filter`,
+    /// `FilterCandidateBuilder::build` rejects the whole-struct reference as
+    /// non-primitive.
+    ///
+    /// Before the fix the rejected conjunct was silently dropped, leaving the
+    /// scan with no `RowFilter` and no post-scan filter, so every row was
+    /// returned — i.e. the predicate was relaxed and the query returned wrong
+    /// results. After the fix the conjunct is surfaced and applied as a
+    /// post-scan filter, so only the rows with a non-null struct survive.
+    #[tokio::test]
+    async fn rejected_struct_conjunct_runs_post_scan_not_dropped() {
+        use arrow::array::{Int32Array, StringArray, StructArray};
+        use arrow::buffer::NullBuffer;
+        use arrow::datatypes::Fields;
+
+        let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+
+        // Schema: id (Int32), s (Struct{value: Int32, label: Utf8}).
+        let struct_fields: Fields = vec![
+            Arc::new(Field::new("value", DataType::Int32, true)),
+            Arc::new(Field::new("label", DataType::Utf8, true)),
+        ]
+        .into();
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("id", DataType::Int32, false),
+            Field::new("s", DataType::Struct(struct_fields.clone()), true),
+        ]));
+
+        // Data: rows 0 and 2 have a non-null struct, row 1 is null.
+        let batch = RecordBatch::try_new(
+            Arc::clone(&schema),
+            vec![
+                Arc::new(Int32Array::from(vec![1, 2, 3])),
+                Arc::new(StructArray::new(
+                    struct_fields,
+                    vec![
+                        Arc::new(Int32Array::from(vec![Some(10), None, Some(30)])) as _,
+                        Arc::new(StringArray::from(vec![Some("a"), None, Some("c")]))
+                            as _,
+                    ],
+                    Some(NullBuffer::from(vec![true, false, true])),
+                )),
+            ],
+        )
+        .unwrap();
+
+        let data_size = write_parquet_batches(
+            Arc::clone(&store),
+            "rejected.parquet",
+            vec![batch],
+            None,
+        )
+        .await;
+
+        let file = PartitionedFile::new("rejected.parquet".to_string(), data_size as u64);
+
+        // `s IS NOT NULL` references a whole struct, which `PushdownChecker`
+        // flags as non-primitive — `FilterCandidateBuilder::build` returns
+        // `Ok(None)` and the conjunct lands in `rejected`.
+        let predicate = logical2physical(&col("s").is_not_null(), &schema);
+
+        let morselizer = ParquetMorselizerBuilder::new()
+            .with_store(Arc::clone(&store))
+            .with_schema(Arc::clone(&schema))
+            .with_predicate(predicate)
+            // The RowFilter path: emulates the post-`try_pushdown_filters`
+            // state where the parent `FilterExec` has already been removed
+            // and the scan owns the conjunct.
+            .with_pushdown_filters(true)
+            .build();
+
+        let stream = open_file(&morselizer, file).await.unwrap();
+        let (_, rows) = count_batches_and_rows(stream).await;
+
+        // 2 rows have a non-null struct. Before the fix this returned 3
+        // (the conjunct was silently dropped).
+        assert_eq!(
+            rows, 2,
+            "expected 2 rows with non-null struct; the rejected conjunct must \
+             be applied post-scan, not silently dropped"
+        );
+    }
+
     /// Helpers for tests that exercise parquet virtual columns
     /// (e.g. `row_number`) plumbed through `TableSchema`/`ParquetOpener`.
     mod virtual_columns {
@@ -5694,9 +5911,12 @@ mod test {
 
         #[tokio::test]
         async fn test_row_index_predicate_allowed_when_pushdown_disabled() {
-            // Guards the `pushdown_filters=false` path: the predicate is only
-            // used for stats pruning (a no-op for row_number) and must not
-            // trip the virtual-column check.
+            // Guards the `pushdown_filters=false` path with a virtual-column
+            // predicate set directly on the opener: it must not trip the
+            // virtual-column check in the read-plan mask. With always-accept
+            // semantics the predicate runs as an in-scan post-scan filter
+            // (evaluated against the reader-appended `row_number` column), so
+            // only the single matching row survives.
             let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
             let expr = col("row_number").eq(lit(2i64));
             let (morselizer, file) =
@@ -5706,7 +5926,7 @@ mod test {
 
             let stream = open_file(&morselizer, file).await.unwrap();
             let (_batches, rows) = count_batches_and_rows(stream).await;
-            assert_eq!(rows, 5);
+            assert_eq!(rows, 1);
         }
     }
 }
diff --git a/datafusion/datasource-parquet/src/push_decoder.rs b/datafusion/datasource-parquet/src/push_decoder.rs
index e5c03df4860fa..a811212a3dfe3 100644
--- a/datafusion/datasource-parquet/src/push_decoder.rs
+++ b/datafusion/datasource-parquet/src/push_decoder.rs
@@ -39,6 +39,7 @@ use std::collections::VecDeque;
 use std::sync::Arc;
 
 use arrow::array::RecordBatch;
+use arrow::compute::BatchCoalescer;
 use arrow::datatypes::SchemaRef;
 use futures::StreamExt;
 use futures::stream::BoxStream;
@@ -57,12 +58,13 @@ use parquet::file::metadata::ParquetMetaData;
 
 use datafusion_common::{DataFusionError, Result, internal_err};
 use datafusion_physical_expr::expressions::DynamicFilterTracking;
+use datafusion_physical_expr::split_conjunction;
 use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
 use datafusion_physical_plan::metrics::{BaselineMetrics, Count, Gauge};
 use datafusion_pruning::{PruningPredicate, PruningPredicateBuilder};
 
 use crate::ParquetFileMetrics;
-use crate::decoder_projection::DecoderProjection;
+use crate::decoder_projection::{DecoderProjection, PostScanSelection};
 use crate::metrics::{ByteProgress, RowFilterSkippedFullyMatchedMetric};
 use crate::row_filter::{
     PrebuiltRowFilterCandidate, prebuild_row_filter_candidates, row_filter_from_prebuilt,
@@ -320,6 +322,26 @@ pub(crate) struct PushDecoderStreamState {
     /// group at a time as they are decoded or skipped, and topped up to the
     /// full range when the stream is dropped.
     pub(crate) byte_progress: ByteProgress,
+    /// Stream-level remaining row limit, enforced *after* the post-scan
+    /// filter. `Some` only when the file has a post-scan filter (which makes
+    /// the decoder-local `with_limit` unsafe — the decoder would short-circuit
+    /// before the filter rejects enough rows); `None` otherwise, in which case
+    /// the limit is enforced inside the decoder via `DecoderBuilderConfig`.
+    pub(crate) remaining_limit: Option<usize>,
+    /// Reassembles post-filter batches back to the target batch size.
+    ///
+    /// `Some` exactly when the file has a post-scan filter. A selective
+    /// predicate leaves only a handful of rows per decoded batch (TPC-H q3
+    /// yields ~41 rows from each 8192-row batch), and without this every one
+    /// of those slivers would be handed to the operator above as its own
+    /// batch. `FilterExec` coalesces for the same reason. `None` when there is
+    /// no post-scan filter: decoder batches are already full size, so routing
+    /// them through the coalescer would only add a copy.
+    pub(crate) batch_coalescer: Option<BatchCoalescer>,
+    /// Set once [`BatchCoalescer::finish_buffered_batch`] has been called, so
+    /// end-of-input flushing happens exactly once no matter which terminal
+    /// path reached it.
+    pub(crate) flushed: bool,
 }
 
 /// A reusable, `Arc`-shared list of prebuilt row-filter candidates.
@@ -359,8 +381,14 @@ pub(crate) struct RowFilterContext {
 
 impl RowFilterContext {
     /// Precompute the candidate list from the raw predicate + file schema +
-    /// metadata. Returns `None` when the predicate has no push-downable
-    /// conjuncts (mirrors the file-open path behaviour).
+    /// metadata.
+    ///
+    /// The first element is `None` when the predicate has no push-downable
+    /// conjuncts (mirrors the file-open path behaviour). The second element
+    /// holds the conjuncts the `RowFilter` machinery could not place on this
+    /// file. The caller must evaluate them elsewhere (post-scan), otherwise
+    /// the predicate is silently relaxed. On a whole-file build error every
+    /// conjunct is returned in that list rather than being dropped.
     pub(crate) fn try_new(
         predicate: &Arc<dyn PhysicalExpr>,
         physical_file_schema: &SchemaRef,
@@ -368,22 +396,31 @@ impl RowFilterContext {
         reorder_predicates: bool,
         file_metrics: ParquetFileMetrics,
         max_predicate_cache_size: Option<usize>,
-    ) -> Option<Self> {
+    ) -> (Option<Self>, Vec<Arc<dyn PhysicalExpr>>) {
         match prebuild_row_filter_candidates(
             predicate,
             physical_file_schema,
             file_metadata.as_ref(),
         ) {
-            Ok(Some(prebuilt)) => Some(Self {
-                prebuilt: PrebuiltRowFilterCandidateList::new(prebuilt),
-                reorder_predicates,
-                file_metrics,
-                max_predicate_cache_size,
-            }),
-            Ok(None) => None,
+            Ok((prebuilt, rejected)) => {
+                let context = prebuilt.map(|prebuilt| Self {
+                    prebuilt: PrebuiltRowFilterCandidateList::new(prebuilt),
+                    reorder_predicates,
+                    file_metrics,
+                    max_predicate_cache_size,
+                });
+                (context, rejected)
+            }
             Err(e) => {
-                debug!("Ignoring error prebuilding row filter candidates: {e}");
-                None
+                // Whole-file build failure: route every conjunct post-scan
+                // rather than silently dropping the predicate.
+                debug!(
+                    "Ignoring error prebuilding row filter candidates: {e}; \
+                     all conjuncts will be evaluated post-scan"
+                );
+                let rejected =
+                    split_conjunction(predicate).into_iter().cloned().collect();
+                (None, rejected)
             }
         }
     }
@@ -445,11 +482,72 @@ impl PushDecoderStreamState {
         let elapsed_compute = self.baseline_metrics.elapsed_compute().clone();
         let mut timer = elapsed_compute.timer();
         loop {
+            // Hand out anything the coalescer has already assembled into a
+            // full-size batch before doing more decoding work.
+            if self
+                .batch_coalescer
+                .as_ref()
+                .is_some_and(BatchCoalescer::has_completed_batch)
+            {
+                return self.emit_completed();
+            }
+
+            // The stream-level limit (set only when a post-scan filter made
+            // the decoder-local limit unsafe) is exhausted — stop. Anything
+            // still buffered is beyond the limit, so it is dropped rather
+            // than flushed.
+            if self.remaining_limit == Some(0) {
+                return None;
+            }
+
             // Step 1: drain a batch from the active reader if any.
             if let Some(reader) = self.active_reader.as_mut() {
                 match reader.next() {
                     Some(Ok(batch)) => {
                         self.copy_arrow_reader_metrics();
+
+                        // Apply the in-scan post-scan filter (if any). The
+                        // decoder's projection mask already covers the
+                        // predicate's columns; the filter's compact-once loop
+                        // needs them for every conjunct, but once it is done
+                        // those the projector does not also read are dropped by
+                        // `narrow` before the residual mask is applied, so we
+                        // never filter a column just to discard it. Survivors go
+                        // into the coalescer rather than straight downstream, so
+                        // a selective predicate does not fragment the stream into
+                        // slivers; the limit and the projection are applied to
+                        // the full-size batches the coalescer hands back.
+                        if let Some(filter) = self.decoder_projection.post_scan_filter() {
+                            let pushed = filter.evaluate(batch).and_then(|selection| {
+                                let (batch, mask) = match selection {
+                                    PostScanSelection::Empty => return Ok(()),
+                                    PostScanSelection::Rows { batch, mask } => {
+                                        (batch, mask)
+                                    }
+                                };
+                                let narrowed = self.decoder_projection.narrow(batch)?;
+                                let coalescer = self
+                                    .batch_coalescer
+                                    .as_mut()
+                                    .expect("coalescer present with a post-scan filter");
+                                match mask {
+                                    Some(mask) => {
+                                        coalescer
+                                            .push_batch_with_filter(narrowed, &mask)?;
+                                    }
+                                    None => coalescer.push_batch(narrowed)?,
+                                }
+                                Ok(())
+                            });
+                            if let Err(e) = pushed {
+                                return Some((Err(e), self));
+                            }
+                            continue;
+                        }
+
+                        // No post-scan filter: the decoder's batches are
+                        // already the right shape, so project and yield
+                        // directly. The limit was pushed into the decoder.
                         let result = self.project_batch(&batch);
                         return Some((result, self));
                     }
@@ -502,7 +600,7 @@ impl PushDecoderStreamState {
             if at_boundary && !self.rg_plan.is_empty() {
                 let pruned_count = self.prune_boundary_row_groups();
                 match self.rebuild_decoder_at_boundary(pruned_count) {
-                    Ok(true) => return None,
+                    Ok(true) => return self.finish(),
                     Ok(false) => {}
                     Err(e) => return Some((Err(e), self)),
                 }
@@ -549,7 +647,7 @@ impl PushDecoderStreamState {
                     }
                     self.active_reader = Some(reader);
                 }
-                Ok(DecodeResult::Finished) => return None,
+                Ok(DecodeResult::Finished) => return self.finish(),
                 Err(e) => {
                     return Some((Err(DataFusionError::from(e)), self));
                 }
@@ -716,6 +814,52 @@ impl PushDecoderStreamState {
         }
     }
 
+    /// Pop one assembled batch from the coalescer, apply the stream-level
+    /// limit, and project it onto the scan's output schema.
+    ///
+    /// Returns `None` only when the coalescer has nothing left, which ends the
+    /// stream. Called from within [`Self::transition`], whose
+    /// `elapsed_compute` timer covers this work.
+    fn emit_completed(mut self) -> Option<(Result<RecordBatch>, Self)> {
+        let batch = self.batch_coalescer.as_mut()?.next_completed_batch()?;
+
+        // Enforce the stream-level limit here rather than in the decoder: the
+        // post-scan filter rejects rows the decoder has already counted, so a
+        // decoder-local limit would stop short.
+        let batch = if let Some(remaining) = self.remaining_limit {
+            if batch.num_rows() > remaining {
+                self.remaining_limit = Some(0);
+                batch.slice(0, remaining)
+            } else {
+                self.remaining_limit = Some(remaining - batch.num_rows());
+                batch
+            }
+        } else {
+            batch
+        };
+        let result = self.project_batch(&batch);
+        Some((result, self))
+    }
+
+    /// End of input: flush the partial batch the coalescer is still holding,
+    /// then drain it one batch at a time. Idempotent — every terminal path in
+    /// `transition` routes through here, but the flush happens once.
+    fn finish(mut self) -> Option<(Result<RecordBatch>, Self)> {
+        self.batch_coalescer.as_ref()?;
+        if !self.flushed {
+            self.flushed = true;
+            if let Err(e) = self
+                .batch_coalescer
+                .as_mut()
+                .expect("coalescer checked present")
+                .finish_buffered_batch()
+            {
+                return Some((Err(DataFusionError::from(e)), self));
+            }
+        }
+        self.emit_completed()
+    }
+
     fn project_batch(&self, batch: &RecordBatch) -> Result<RecordBatch> {
         self.decoder_projection.map(batch)
     }
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 463f9f84daf5a..96c304a38f661 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -412,36 +412,41 @@ fn size_of_columns(columns: &[usize], metadata: &ParquetMetaData) -> Result<usiz
 /// * `file_metrics` - Metrics for tracking filter performance
 ///
 /// # Returns
-/// * `Ok(Some(row_filter))` if the expression can be used as a RowFilter
-/// * `Ok(None)` if the expression cannot be used as a RowFilter
-/// * `Err(e)` if an error occurs while building the filter
 ///
-/// Note: The returned `RowFilter` may not contain all conjuncts from the original
-/// expression. Conjuncts that cannot be evaluated as an `ArrowPredicate` are ignored.
+/// `Ok((row_filter, rejected))` where:
+/// * `row_filter` is `Some` if at least one conjunct can be evaluated as an
+///   `ArrowPredicate`, `None` otherwise.
+/// * `rejected` holds the conjuncts that *cannot* be evaluated as an
+///   `ArrowPredicate` (for example whole-struct references or columns missing
+///   from this file's physical schema). The caller MUST apply these elsewhere
+///   — e.g. as a post-scan filter — otherwise the predicate is relaxed and the
+///   query returns wrong results.
+///
+/// `Err(e)` if an error occurs while building the filter.
 ///
 /// For example, if the expression is `a = 1 AND b = 2 AND c = 3` and `b = 2`
-/// cannot be evaluated for some reason, the returned `RowFilter` will contain
-/// only `a = 1` and `c = 3`.
+/// cannot be evaluated as an `ArrowPredicate`, the returned `RowFilter`
+/// contains `a = 1` and `c = 3` and `rejected` contains `b = 2`.
+// The tuple captures two related but distinct outputs; a type alias would
+// obscure rather than clarify the public signature.
+#[expect(clippy::type_complexity)]
 pub fn build_row_filter(
     expr: &Arc<dyn PhysicalExpr>,
     file_schema: &SchemaRef,
     metadata: &ParquetMetaData,
     reorder_predicates: bool,
     file_metrics: &ParquetFileMetrics,
-) -> Result<Option<RowFilter>> {
+) -> Result<(Option<RowFilter>, Vec<Arc<dyn PhysicalExpr>>)> {
     // Implemented on top of the prebuild split so there is a single place
     // that splits conjuncts, orders candidates, and wires metrics — callers
     // that build once per file go through the same code as the per-row-group
     // rebuild path in `RowFilterContext`.
-    let Some(prebuilt) = prebuild_row_filter_candidates(expr, file_schema, metadata)?
-    else {
-        return Ok(None);
-    };
-    Ok(Some(row_filter_from_prebuilt(
-        &prebuilt,
-        reorder_predicates,
-        file_metrics,
-    )))
+    let (prebuilt, rejected) =
+        prebuild_row_filter_candidates(expr, file_schema, metadata)?;
+    let row_filter = prebuilt.map(|prebuilt| {
+        row_filter_from_prebuilt(&prebuilt, reorder_predicates, file_metrics)
+    });
+    Ok((row_filter, rejected))
 }
 
 /// A precomputed [`FilterCandidate`] with its expression column-reassigned to
@@ -483,29 +488,40 @@ impl PrebuiltRowFilterCandidate {
 /// walks and `Arc<PhysicalExpr>` allocations that showed up as top hot spots
 /// in TPCH profiles.
 ///
-/// Returns `Ok(None)` when the predicate has no push-downable conjuncts, in
-/// which case callers should skip installing a `RowFilter` entirely.
+/// The first element is `None` when the predicate has no push-downable
+/// conjuncts, in which case callers should skip installing a `RowFilter`
+/// entirely. The second element holds the conjuncts that cannot be evaluated
+/// as an `ArrowPredicate` on this file; see [`build_row_filter`] for why the
+/// caller must apply them elsewhere.
+#[expect(clippy::type_complexity)]
 pub(crate) fn prebuild_row_filter_candidates(
     expr: &Arc<dyn PhysicalExpr>,
     file_schema: &SchemaRef,
     metadata: &ParquetMetaData,
-) -> Result<Option<Vec<PrebuiltRowFilterCandidate>>> {
+) -> Result<(
+    Option<Vec<PrebuiltRowFilterCandidate>>,
+    Vec<Arc<dyn PhysicalExpr>>,
+)> {
     // Split into conjuncts:
     // `a = 1 AND b = 2 AND c = 3` -> [`a = 1`, `b = 2`, `c = 3`]
     let predicates = split_conjunction(expr);
-    let candidates: Vec<FilterCandidate> = predicates
-        .into_iter()
-        .map(|expr| {
-            FilterCandidateBuilder::new(Arc::clone(expr), Arc::clone(file_schema))
-                .build(metadata)
-        })
-        .collect::<Result<Vec<_>, _>>()?
-        .into_iter()
-        .flatten()
-        .collect();
+
+    // Partition conjuncts into those that can be evaluated as ArrowPredicates
+    // and those that cannot. Rejected conjuncts are returned to the caller so
+    // they are never silently dropped.
+    let mut candidates: Vec<FilterCandidate> = Vec::with_capacity(predicates.len());
+    let mut rejected: Vec<Arc<dyn PhysicalExpr>> = Vec::new();
+    for predicate in predicates {
+        match FilterCandidateBuilder::new(Arc::clone(predicate), Arc::clone(file_schema))
+            .build(metadata)?
+        {
+            Some(candidate) => candidates.push(candidate),
+            None => rejected.push(Arc::clone(predicate)),
+        }
+    }
 
     if candidates.is_empty() {
-        return Ok(None);
+        return Ok((None, rejected));
     }
 
     let prebuilt: Vec<PrebuiltRowFilterCandidate> = candidates
@@ -523,7 +539,7 @@ pub(crate) fn prebuild_row_filter_candidates(
         })
         .collect::<Result<Vec<_>>>()?;
 
-    Ok(Some(prebuilt))
+    Ok((Some(prebuilt), rejected))
 }
 
 /// Wrap a list of prebuilt candidates into a fresh [`RowFilter`], assigning
@@ -934,10 +950,14 @@ mod test {
         let file_metrics =
             ParquetFileMetrics::new(0, &format!("{func_name}.parquet"), &metrics);
 
-        let row_filter =
+        let (row_filter, rejected) =
             build_row_filter(&expr, &file_schema, &metadata, false, &file_metrics)
-                .expect("building row filter")
-                .expect("row filter should exist");
+                .expect("building row filter");
+        assert!(
+            rejected.is_empty(),
+            "expected no rejected conjuncts, got {rejected:?}"
+        );
+        let row_filter = row_filter.expect("row filter should exist");
 
         let reader = parquet_reader_builder
             .with_row_filter(row_filter)
@@ -1648,10 +1668,14 @@ mod test {
         let metrics = ExecutionPlanMetricsSet::new();
         let file_metrics = ParquetFileMetrics::new(0, "struct_e2e.parquet", &metrics);
 
-        let row_filter =
+        let (row_filter, rejected) =
             build_row_filter(&expr, &file_schema, &metadata, false, &file_metrics)
-                .expect("building row filter")
-                .expect("row filter should exist");
+                .expect("building row filter");
+        assert!(
+            rejected.is_empty(),
+            "expected no rejected conjuncts, got {rejected:?}"
+        );
+        let row_filter = row_filter.expect("row filter should exist");
 
         let reader = parquet_reader_builder
             .with_row_filter(row_filter)
@@ -2104,10 +2128,14 @@ mod test {
         let file_metrics =
             ParquetFileMetrics::new(0, "shared_prefix_e2e.parquet", &metrics);
 
-        let row_filter =
+        let (row_filter, rejected) =
             build_row_filter(&expr, &file_schema, &metadata, false, &file_metrics)
-                .expect("building row filter")
-                .expect("row filter should exist");
+                .expect("building row filter");
+        assert!(
+            rejected.is_empty(),
+            "expected no rejected conjuncts, got {rejected:?}"
+        );
+        let row_filter = row_filter.expect("row filter should exist");
 
         let reader = parquet_reader_builder
             .with_row_filter(row_filter)
@@ -2127,4 +2155,85 @@ mod test {
         assert_eq!(file_metrics.pushdown_rows_pruned.value(), 2);
         assert_eq!(file_metrics.pushdown_rows_matched.value(), 2);
     }
+
+    /// Regression test: a predicate `(s IS NOT NULL) AND (id = 1)` mixes a
+    /// conjunct that the `RowFilter` machinery cannot evaluate (whole-struct
+    /// reference — [`PushdownChecker`] flags it as non-primitive) with one
+    /// that it can. `build_row_filter` must return the rejected conjunct in
+    /// its second tuple element so the caller can re-route it to a post-scan
+    /// filter; before this was fixed the rejected conjunct was silently
+    /// dropped on the floor while the parent `FilterExec` had already been
+    /// removed, relaxing the predicate and returning wrong results.
+    #[test]
+    fn build_row_filter_surfaces_rejected_struct_conjunct() {
+        let struct_fields: Fields = vec![
+            Arc::new(Field::new("value", DataType::Int32, false)),
+            Arc::new(Field::new("label", DataType::Utf8, false)),
+        ]
+        .into();
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("id", DataType::Int32, false),
+            Field::new("s", DataType::Struct(struct_fields.clone()), false),
+        ]));
+
+        let batch = RecordBatch::try_new(
+            Arc::clone(&schema),
+            vec![
+                Arc::new(Int32Array::from(vec![1, 2, 3])),
+                Arc::new(StructArray::new(
+                    struct_fields,
+                    vec![
+                        Arc::new(Int32Array::from(vec![10, 20, 30])) as _,
+                        Arc::new(StringArray::from(vec!["a", "b", "c"])) as _,
+                    ],
+                    None,
+                )),
+            ],
+        )
+        .unwrap();
+
+        let file = NamedTempFile::new().expect("temp file");
+        let mut writer =
+            ArrowWriter::try_new(file.reopen().unwrap(), Arc::clone(&schema), None)
+                .expect("writer");
+        writer.write(&batch).expect("write batch");
+        writer.close().expect("close writer");
+
+        let reader_file = file.reopen().expect("reopen file");
+        let parquet_reader_builder =
+            ParquetRecordBatchReaderBuilder::try_new(reader_file)
+                .expect("reader builder");
+        let metadata = parquet_reader_builder.metadata().clone();
+        let file_schema = parquet_reader_builder.schema().clone();
+
+        // (s IS NOT NULL) AND (id = 1)
+        // The first conjunct references a whole struct -> RowFilter rejects.
+        // The second is a plain Int32 equality -> RowFilter accepts.
+        let predicate_expr = col("s")
+            .is_not_null()
+            .and(col("id").eq(Expr::Literal(ScalarValue::Int32(Some(1)), None)));
+        let expr = logical2physical(&predicate_expr, &file_schema);
+
+        let metrics = ExecutionPlanMetricsSet::new();
+        let file_metrics =
+            ParquetFileMetrics::new(0, "build_row_filter_rejected.parquet", &metrics);
+
+        let (row_filter, rejected) =
+            build_row_filter(&expr, &file_schema, &metadata, false, &file_metrics)
+                .expect("building row filter");
+
+        // The plain id = 1 conjunct produced a RowFilter…
+        assert!(
+            row_filter.is_some(),
+            "id = 1 should have produced a RowFilter"
+        );
+        // …and the struct IS NOT NULL conjunct must be surfaced as rejected,
+        // never silently dropped.
+        assert_eq!(
+            rejected.len(),
+            1,
+            "expected exactly one rejected conjunct (s IS NOT NULL), got {rejected:?}"
+        );
+    }
 }
diff --git a/datafusion/datasource-parquet/src/source.rs b/datafusion/datasource-parquet/src/source.rs
index 32fc8c9f3da33..c1e7e47b459e3 100644
--- a/datafusion/datasource-parquet/src/source.rs
+++ b/datafusion/datasource-parquet/src/source.rs
@@ -896,14 +896,14 @@ impl FileSource for ParquetSource {
         source.predicate = Some(predicate);
         source = source.with_pushdown_filters(pushdown_filters);
         let source = Arc::new(source);
-        // If pushdown_filters is false we tell our parents that they still have to handle the filters,
-        // even if we updated the predicate to include the filters (they will only be used for stats pruning).
-        if !pushdown_filters {
-            return Ok(FilterPushdownPropagation::with_parent_pushdown_result(
-                vec![PushedDown::No; filters.len()],
-            )
-            .with_updated_node(source));
-        }
+        // The parquet scan always accepts pushable filters: report each
+        // pushable filter as accepted (`Yes`) so the parent `FilterExec` is
+        // removed. The scan now owns these filters and guarantees they are
+        // applied — as a parquet `RowFilter` when `pushdown_filters` is
+        // enabled, or as the in-scan post-scan filter otherwise (and for any
+        // conjunct the `RowFilter` cannot evaluate on a given file). The
+        // `pushdown_filters` config is preserved because it still controls the
+        // `RowFilter` vs. post-scan placement downstream.
         Ok(FilterPushdownPropagation::with_parent_pushdown_result(
             filters.iter().map(|f| f.discriminant).collect(),
         )
diff --git a/datafusion/sqllogictest/test_files/clickbench.slt b/datafusion/sqllogictest/test_files/clickbench.slt
index 7cb5547383c38..d2ee7f8def96a 100644
--- a/datafusion/sqllogictest/test_files/clickbench.slt
+++ b/datafusion/sqllogictest/test_files/clickbench.slt
@@ -88,9 +88,8 @@ physical_plan
 02)--AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))]
 03)----CoalescePartitionsExec
 04)------AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))]
-05)--------FilterExec: AdvEngineID@0 != 0, projection=[]
-06)----------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-07)------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[AdvEngineID], file_type=parquet, predicate=AdvEngineID@40 != 0, pruning_predicate=AdvEngineID_null_count@2 != row_count@3 AND (AdvEngineID_min@0 != 0 OR 0 != AdvEngineID_max@1), required_guarantees=[AdvEngineID not in (0)]
+05)--------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+06)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, file_type=parquet, predicate=AdvEngineID@40 != 0, pruning_predicate=AdvEngineID_null_count@2 != row_count@3 AND (AdvEngineID_min@0 != 0 OR 0 != AdvEngineID_max@1), required_guarantees=[AdvEngineID not in (0)]
 
 query I
 SELECT COUNT(*) FROM hits WHERE "AdvEngineID" <> 0;
@@ -218,9 +217,8 @@ physical_plan
 04)------AggregateExec: mode=FinalPartitioned, gby=[AdvEngineID@0 as AdvEngineID], aggr=[count(Int64(1))]
 05)--------RepartitionExec: partitioning=Hash([AdvEngineID@0], 4), input_partitions=4
 06)----------AggregateExec: mode=Partial, gby=[AdvEngineID@0 as AdvEngineID], aggr=[count(Int64(1))]
-07)------------FilterExec: AdvEngineID@0 != 0
-08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[AdvEngineID], file_type=parquet, predicate=AdvEngineID@40 != 0, pruning_predicate=AdvEngineID_null_count@2 != row_count@3 AND (AdvEngineID_min@0 != 0 OR 0 != AdvEngineID_max@1), required_guarantees=[AdvEngineID not in (0)]
+07)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+08)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[AdvEngineID], file_type=parquet, predicate=AdvEngineID@40 != 0, pruning_predicate=AdvEngineID_null_count@2 != row_count@3 AND (AdvEngineID_min@0 != 0 OR 0 != AdvEngineID_max@1), required_guarantees=[AdvEngineID not in (0)]
 
 query II
 SELECT "AdvEngineID", COUNT(*) FROM hits WHERE "AdvEngineID" <> 0 GROUP BY "AdvEngineID" ORDER BY COUNT(*) DESC;
@@ -306,9 +304,8 @@ physical_plan
 07)------------AggregateExec: mode=FinalPartitioned, gby=[MobilePhoneModel@0 as MobilePhoneModel, alias1@1 as alias1], aggr=[]
 08)--------------RepartitionExec: partitioning=Hash([MobilePhoneModel@0, alias1@1], 4), input_partitions=4
 09)----------------AggregateExec: mode=Partial, gby=[MobilePhoneModel@1 as MobilePhoneModel, UserID@0 as alias1], aggr=[]
-10)------------------FilterExec: MobilePhoneModel@1 !=
-11)--------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-12)----------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[UserID, MobilePhoneModel], file_type=parquet, predicate=MobilePhoneModel@34 != , pruning_predicate=MobilePhoneModel_null_count@2 != row_count@3 AND (MobilePhoneModel_min@0 !=  OR  != MobilePhoneModel_max@1), required_guarantees=[MobilePhoneModel not in ()]
+10)------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+11)--------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[UserID, MobilePhoneModel], file_type=parquet, predicate=MobilePhoneModel@34 != , pruning_predicate=MobilePhoneModel_null_count@2 != row_count@3 AND (MobilePhoneModel_min@0 !=  OR  != MobilePhoneModel_max@1), required_guarantees=[MobilePhoneModel not in ()]
 
 query TI
 SELECT "MobilePhoneModel", COUNT(DISTINCT "UserID") AS u FROM hits WHERE "MobilePhoneModel" <> '' GROUP BY "MobilePhoneModel" ORDER BY u DESC LIMIT 10;
@@ -336,9 +333,8 @@ physical_plan
 07)------------AggregateExec: mode=FinalPartitioned, gby=[MobilePhone@0 as MobilePhone, MobilePhoneModel@1 as MobilePhoneModel, alias1@2 as alias1], aggr=[]
 08)--------------RepartitionExec: partitioning=Hash([MobilePhone@0, MobilePhoneModel@1, alias1@2], 4), input_partitions=4
 09)----------------AggregateExec: mode=Partial, gby=[MobilePhone@1 as MobilePhone, MobilePhoneModel@2 as MobilePhoneModel, UserID@0 as alias1], aggr=[]
-10)------------------FilterExec: MobilePhoneModel@2 !=
-11)--------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-12)----------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[UserID, MobilePhone, MobilePhoneModel], file_type=parquet, predicate=MobilePhoneModel@34 != , pruning_predicate=MobilePhoneModel_null_count@2 != row_count@3 AND (MobilePhoneModel_min@0 !=  OR  != MobilePhoneModel_max@1), required_guarantees=[MobilePhoneModel not in ()]
+10)------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+11)--------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[UserID, MobilePhone, MobilePhoneModel], file_type=parquet, predicate=MobilePhoneModel@34 != , pruning_predicate=MobilePhoneModel_null_count@2 != row_count@3 AND (MobilePhoneModel_min@0 !=  OR  != MobilePhoneModel_max@1), required_guarantees=[MobilePhoneModel not in ()]
 
 query ITI
 SELECT "MobilePhone", "MobilePhoneModel", COUNT(DISTINCT "UserID") AS u FROM hits WHERE "MobilePhoneModel" <> '' GROUP BY "MobilePhone", "MobilePhoneModel" ORDER BY u DESC LIMIT 10;
@@ -362,9 +358,8 @@ physical_plan
 04)------AggregateExec: mode=FinalPartitioned, gby=[SearchPhrase@0 as SearchPhrase], aggr=[count(Int64(1))]
 05)--------RepartitionExec: partitioning=Hash([SearchPhrase@0], 4), input_partitions=4
 06)----------AggregateExec: mode=Partial, gby=[SearchPhrase@0 as SearchPhrase], aggr=[count(Int64(1))]
-07)------------FilterExec: SearchPhrase@0 !=
-08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 != , pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
+07)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+08)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 != , pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
 
 query TI
 SELECT "SearchPhrase", COUNT(*) AS c FROM hits WHERE "SearchPhrase" <> '' GROUP BY "SearchPhrase" ORDER BY c DESC LIMIT 10;
@@ -392,9 +387,8 @@ physical_plan
 07)------------AggregateExec: mode=FinalPartitioned, gby=[SearchPhrase@0 as SearchPhrase, alias1@1 as alias1], aggr=[]
 08)--------------RepartitionExec: partitioning=Hash([SearchPhrase@0, alias1@1], 4), input_partitions=4
 09)----------------AggregateExec: mode=Partial, gby=[SearchPhrase@1 as SearchPhrase, UserID@0 as alias1], aggr=[]
-10)------------------FilterExec: SearchPhrase@1 !=
-11)--------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-12)----------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[UserID, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 != , pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
+10)------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+11)--------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[UserID, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 != , pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
 
 query TI
 SELECT "SearchPhrase", COUNT(DISTINCT "UserID") AS u FROM hits WHERE "SearchPhrase" <> '' GROUP BY "SearchPhrase" ORDER BY u DESC LIMIT 10;
@@ -418,9 +412,8 @@ physical_plan
 04)------AggregateExec: mode=FinalPartitioned, gby=[SearchEngineID@0 as SearchEngineID, SearchPhrase@1 as SearchPhrase], aggr=[count(Int64(1))]
 05)--------RepartitionExec: partitioning=Hash([SearchEngineID@0, SearchPhrase@1], 4), input_partitions=4
 06)----------AggregateExec: mode=Partial, gby=[SearchEngineID@0 as SearchEngineID, SearchPhrase@1 as SearchPhrase], aggr=[count(Int64(1))]
-07)------------FilterExec: SearchPhrase@1 !=
-08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[SearchEngineID, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 != , pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
+07)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+08)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[SearchEngineID, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 != , pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
 
 query ITI
 SELECT "SearchEngineID", "SearchPhrase", COUNT(*) AS c FROM hits WHERE "SearchPhrase" <> '' GROUP BY "SearchEngineID", "SearchPhrase" ORDER BY c DESC LIMIT 10;
@@ -550,10 +543,7 @@ logical_plan
 01)SubqueryAlias: hits
 02)--Filter: hits_raw.UserID = Int64(435090932899640449)
 03)----TableScan: hits_raw projection=[UserID], partial_filters=[hits_raw.UserID = Int64(435090932899640449)]
-physical_plan
-01)FilterExec: UserID@0 = 435090932899640449
-02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[UserID], file_type=parquet, predicate=UserID@9 = 435090932899640449, pruning_predicate=UserID_null_count@2 != row_count@3 AND UserID_min@0 <= 435090932899640449 AND 435090932899640449 <= UserID_max@1, required_guarantees=[UserID in (435090932899640449)]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[UserID], file_type=parquet, predicate=UserID@9 = 435090932899640449, pruning_predicate=UserID_null_count@2 != row_count@3 AND UserID_min@0 <= 435090932899640449 AND 435090932899640449 <= UserID_max@1, required_guarantees=[UserID in (435090932899640449)]
 
 query I
 SELECT "UserID" FROM hits WHERE "UserID" = 435090932899640449;
@@ -575,9 +565,8 @@ physical_plan
 02)--AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))]
 03)----CoalescePartitionsExec
 04)------AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))]
-05)--------FilterExec: URL@0 LIKE %google%, projection=[]
-06)----------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-07)------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[URL], file_type=parquet, predicate=URL@13 LIKE %google%
+05)--------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+06)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, file_type=parquet, predicate=URL@13 LIKE %google%
 
 query I
 SELECT COUNT(*) FROM hits WHERE "URL" LIKE '%google%';
@@ -602,9 +591,8 @@ physical_plan
 04)------AggregateExec: mode=FinalPartitioned, gby=[SearchPhrase@0 as SearchPhrase], aggr=[min(hits.URL), count(Int64(1))]
 05)--------RepartitionExec: partitioning=Hash([SearchPhrase@0], 4), input_partitions=4
 06)----------AggregateExec: mode=Partial, gby=[SearchPhrase@1 as SearchPhrase], aggr=[min(hits.URL), count(Int64(1))]
-07)------------FilterExec: SearchPhrase@1 !=  AND URL@0 LIKE %google%
-08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[URL, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 !=  AND URL@13 LIKE %google%, pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
+07)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+08)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[URL, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 !=  AND URL@13 LIKE %google%, pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
 
 query TTI
 SELECT "SearchPhrase", MIN("URL"), COUNT(*) AS c FROM hits WHERE "URL" LIKE '%google%' AND "SearchPhrase" <> '' GROUP BY "SearchPhrase" ORDER BY c DESC LIMIT 10;
@@ -628,9 +616,8 @@ physical_plan
 04)------AggregateExec: mode=FinalPartitioned, gby=[SearchPhrase@0 as SearchPhrase], aggr=[min(hits.URL), min(hits.Title), count(Int64(1)), count(DISTINCT hits.UserID)]
 05)--------RepartitionExec: partitioning=Hash([SearchPhrase@0], 4), input_partitions=4
 06)----------AggregateExec: mode=Partial, gby=[SearchPhrase@3 as SearchPhrase], aggr=[min(hits.URL), min(hits.Title), count(Int64(1)), count(DISTINCT hits.UserID)]
-07)------------FilterExec: SearchPhrase@3 !=  AND Title@0 LIKE %Google% AND URL@2 NOT LIKE %.google.%
-08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[Title, UserID, URL, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 !=  AND Title@2 LIKE %Google% AND URL@13 NOT LIKE %.google.%, pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
+07)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+08)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[Title, UserID, URL, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 !=  AND Title@2 LIKE %Google% AND URL@13 NOT LIKE %.google.%, pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
 
 query TTTII
 SELECT "SearchPhrase", MIN("URL"), MIN("Title"), COUNT(*) AS c, COUNT(DISTINCT "UserID") FROM hits WHERE "Title" LIKE '%Google%' AND "URL" NOT LIKE '%.google.%' AND "SearchPhrase" <> '' GROUP BY "SearchPhrase" ORDER BY c DESC LIMIT 10;
@@ -650,9 +637,8 @@ physical_plan
 01)SortPreservingMergeExec: [EventTime@4 ASC NULLS LAST], fetch=10
 02)--SortExec: TopK(fetch=10), expr=[EventTime@4 ASC NULLS LAST], preserve_partitioning=[true]
 03)----ProjectionExec: expr=[WatchID@0 as WatchID, JavaEnable@1 as JavaEnable, Title@2 as Title, GoodEvent@3 as GoodEvent, EventTime@4 as EventTime, CounterID@6 as CounterID, ClientIP@7 as ClientIP, RegionID@8 as RegionID, UserID@9 as UserID, CounterClass@10 as CounterClass, OS@11 as OS, UserAgent@12 as UserAgent, URL@13 as URL, Referer@14 as Referer, IsRefresh@15 as IsRefresh, RefererCategoryID@16 as RefererCategoryID, RefererRegionID@17 as RefererRegionID, URLCategoryID@18 as URLCategoryID, URLRegionID@19 as URLRegionID, ResolutionWidth@20 as ResolutionWidth, ResolutionHeight@21 as ResolutionHeight, ResolutionDepth@22 as ResolutionDepth, FlashMajor@23 as FlashMajor, FlashMinor@24 as FlashMinor, FlashMinor2@25 as FlashMinor2, NetMajor@26 as NetMajor, NetMinor@27 as NetMinor, UserAgentMajor@28 as UserAgentMajor, UserAgentMinor@29 as UserAgentMinor, CookieEnable@30 as CookieEnable, JavascriptEnable@31 as JavascriptEnable, IsMobile@32 as IsMobile, MobilePhone@33 as MobilePhone, MobilePhoneModel@34 as MobilePhoneModel, Params@35 as Params, IPNetworkID@36 as IPNetworkID, TraficSourceID@37 as TraficSourceID, SearchEngineID@38 as SearchEngineID, SearchPhrase@39 as SearchPhrase, AdvEngineID@40 as AdvEngineID, IsArtifical@41 as IsArtifical, WindowClientWidth@42 as WindowClientWidth, WindowClientHeight@43 as WindowClientHeight, ClientTimeZone@44 as ClientTimeZone, ClientEventTime@45 as ClientEventTime, SilverlightVersion1@46 as SilverlightVersion1, SilverlightVersion2@47 as SilverlightVersion2, SilverlightVersion3@48 as SilverlightVersion3, SilverlightVersion4@49 as SilverlightVersion4, PageCharset@50 as PageCharset, CodeVersion@51 as CodeVersion, IsLink@52 as IsLink, IsDownload@53 as IsDownload, IsNotBounce@54 as IsNotBounce, FUniqID@55 as FUniqID, OriginalURL@56 as OriginalURL, HID@57 as HID, IsOldCounter@58 as IsOldCounter, IsEvent@59 as IsEvent, IsParameter@60 as IsParameter, DontCountHits@61 as DontCountHits, WithHash@62 as WithHash, HitColor@63 as HitColor, LocalEventTime@64 as LocalEventTime, Age@65 as Age, Sex@66 as Sex, Income@67 as Income, Interests@68 as Interests, Robotness@69 as Robotness, RemoteIP@70 as RemoteIP, WindowName@71 as WindowName, OpenerName@72 as OpenerName, HistoryLength@73 as HistoryLength, BrowserLanguage@74 as BrowserLanguage, BrowserCountry@75 as BrowserCountry, SocialNetwork@76 as SocialNetwork, SocialAction@77 as SocialAction, HTTPError@78 as HTTPError, SendTiming@79 as SendTiming, DNSTiming@80 as DNSTiming, ConnectTiming@81 as ConnectTiming, ResponseStartTiming@82 as ResponseStartTiming, ResponseEndTiming@83 as ResponseEndTiming, FetchTiming@84 as FetchTiming, SocialSourceNetworkID@85 as SocialSourceNetworkID, SocialSourcePage@86 as SocialSourcePage, ParamPrice@87 as ParamPrice, ParamOrderID@88 as ParamOrderID, ParamCurrency@89 as ParamCurrency, ParamCurrencyID@90 as ParamCurrencyID, OpenstatServiceName@91 as OpenstatServiceName, OpenstatCampaignID@92 as OpenstatCampaignID, OpenstatAdID@93 as OpenstatAdID, OpenstatSourceID@94 as OpenstatSourceID, UTMSource@95 as UTMSource, UTMMedium@96 as UTMMedium, UTMCampaign@97 as UTMCampaign, UTMContent@98 as UTMContent, UTMTerm@99 as UTMTerm, FromTag@100 as FromTag, HasGCLID@101 as HasGCLID, RefererHash@102 as RefererHash, URLHash@103 as URLHash, CLID@104 as CLID, CAST(CAST(EventDate@5 AS Int32) AS Date32) as EventDate]
-04)------FilterExec: URL@13 LIKE %google%
-05)--------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-06)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[WatchID, JavaEnable, Title, GoodEvent, EventTime, EventDate, CounterID, ClientIP, RegionID, UserID, CounterClass, OS, UserAgent, URL, Referer, IsRefresh, RefererCategoryID, RefererRegionID, URLCategoryID, URLRegionID, ResolutionWidth, ResolutionHeight, ResolutionDepth, FlashMajor, FlashMinor, FlashMinor2, NetMajor, NetMinor, UserAgentMajor, UserAgentMinor, CookieEnable, JavascriptEnable, IsMobile, MobilePhone, MobilePhoneModel, Params, IPNetworkID, TraficSourceID, SearchEngineID, SearchPhrase, AdvEngineID, IsArtifical, WindowClientWidth, WindowClientHeight, ClientTimeZone, ClientEventTime, SilverlightVersion1, SilverlightVersion2, SilverlightVersion3, SilverlightVersion4, PageCharset, CodeVersion, IsLink, IsDownload, IsNotBounce, FUniqID, OriginalURL, HID, IsOldCounter, IsEvent, IsParameter, DontCountHits, WithHash, HitColor, LocalEventTime, Age, Sex, Income, Interests, Robotness, RemoteIP, WindowName, OpenerName, HistoryLength, BrowserLanguage, BrowserCountry, SocialNetwork, SocialAction, HTTPError, SendTiming, DNSTiming, ConnectTiming, ResponseStartTiming, ResponseEndTiming, FetchTiming, SocialSourceNetworkID, SocialSourcePage, ParamPrice, ParamOrderID, ParamCurrency, ParamCurrencyID, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID, UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash, URLHash, CLID], file_type=parquet, predicate=URL@13 LIKE %google% AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible
+04)------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+05)--------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[WatchID, JavaEnable, Title, GoodEvent, EventTime, EventDate, CounterID, ClientIP, RegionID, UserID, CounterClass, OS, UserAgent, URL, Referer, IsRefresh, RefererCategoryID, RefererRegionID, URLCategoryID, URLRegionID, ResolutionWidth, ResolutionHeight, ResolutionDepth, FlashMajor, FlashMinor, FlashMinor2, NetMajor, NetMinor, UserAgentMajor, UserAgentMinor, CookieEnable, JavascriptEnable, IsMobile, MobilePhone, MobilePhoneModel, Params, IPNetworkID, TraficSourceID, SearchEngineID, SearchPhrase, AdvEngineID, IsArtifical, WindowClientWidth, WindowClientHeight, ClientTimeZone, ClientEventTime, SilverlightVersion1, SilverlightVersion2, SilverlightVersion3, SilverlightVersion4, PageCharset, CodeVersion, IsLink, IsDownload, IsNotBounce, FUniqID, OriginalURL, HID, IsOldCounter, IsEvent, IsParameter, DontCountHits, WithHash, HitColor, LocalEventTime, Age, Sex, Income, Interests, Robotness, RemoteIP, WindowName, OpenerName, HistoryLength, BrowserLanguage, BrowserCountry, SocialNetwork, SocialAction, HTTPError, SendTiming, DNSTiming, ConnectTiming, ResponseStartTiming, ResponseEndTiming, FetchTiming, SocialSourceNetworkID, SocialSourcePage, ParamPrice, ParamOrderID, ParamCurrency, ParamCurrencyID, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID, UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash, URLHash, CLID], file_type=parquet, predicate=URL@13 LIKE %google% AND DynamicFilter [ empty ], sort_order_for_reorder=[EventTime@4 ASC NULLS LAST], dynamic_rg_pruning=eligible
 
 query IITIIIIIIIIITTIIIIIIIIIITIIITIIIITTIIITIIIIIIIIIITIIIIITIIIIIITIIIIIIIIIITTTTIIIIIIIITITTITTTTTTTTTTIIIID
 SELECT * FROM hits WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10;
@@ -670,13 +656,9 @@ logical_plan
 05)--------Filter: hits_raw.SearchPhrase != Utf8View("")
 06)----------TableScan: hits_raw projection=[EventTime, SearchPhrase], partial_filters=[hits_raw.SearchPhrase != Utf8View("")]
 physical_plan
-01)ProjectionExec: expr=[SearchPhrase@0 as SearchPhrase]
-02)--SortPreservingMergeExec: [EventTime@1 ASC NULLS LAST], fetch=10
-03)----ProjectionExec: expr=[SearchPhrase@1 as SearchPhrase, EventTime@0 as EventTime]
-04)------SortExec: TopK(fetch=10), expr=[EventTime@0 ASC NULLS LAST], preserve_partitioning=[true]
-05)--------FilterExec: SearchPhrase@1 !=
-06)----------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-07)------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventTime, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 !=  AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible, pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
+01)ProjectionExec: expr=[SearchPhrase@1 as SearchPhrase]
+02)--SortExec: TopK(fetch=10), expr=[EventTime@0 ASC NULLS LAST], preserve_partitioning=[false]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventTime, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 !=  AND DynamicFilter [ empty ], sort_order_for_reorder=[EventTime@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
 
 query T
 SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "EventTime" LIMIT 10;
@@ -692,11 +674,8 @@ logical_plan
 03)----Filter: hits_raw.SearchPhrase != Utf8View("")
 04)------TableScan: hits_raw projection=[SearchPhrase], partial_filters=[hits_raw.SearchPhrase != Utf8View("")]
 physical_plan
-01)SortPreservingMergeExec: [SearchPhrase@0 ASC NULLS LAST], fetch=10
-02)--SortExec: TopK(fetch=10), expr=[SearchPhrase@0 ASC NULLS LAST], preserve_partitioning=[true]
-03)----FilterExec: SearchPhrase@0 !=
-04)------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-05)--------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 !=  AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible, pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
+01)SortExec: TopK(fetch=10), expr=[SearchPhrase@0 ASC NULLS LAST], preserve_partitioning=[false]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 !=  AND DynamicFilter [ empty ], sort_order_for_reorder=[SearchPhrase@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
 
 query T
 SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10;
@@ -714,13 +693,9 @@ logical_plan
 05)--------Filter: hits_raw.SearchPhrase != Utf8View("")
 06)----------TableScan: hits_raw projection=[EventTime, SearchPhrase], partial_filters=[hits_raw.SearchPhrase != Utf8View("")]
 physical_plan
-01)ProjectionExec: expr=[SearchPhrase@0 as SearchPhrase]
-02)--SortPreservingMergeExec: [EventTime@1 ASC NULLS LAST, SearchPhrase@0 ASC NULLS LAST], fetch=10
-03)----ProjectionExec: expr=[SearchPhrase@1 as SearchPhrase, EventTime@0 as EventTime]
-04)------SortExec: TopK(fetch=10), expr=[EventTime@0 ASC NULLS LAST, SearchPhrase@1 ASC NULLS LAST], preserve_partitioning=[true]
-05)--------FilterExec: SearchPhrase@1 !=
-06)----------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-07)------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventTime, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 !=  AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible, pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
+01)ProjectionExec: expr=[SearchPhrase@1 as SearchPhrase]
+02)--SortExec: TopK(fetch=10), expr=[EventTime@0 ASC NULLS LAST, SearchPhrase@1 ASC NULLS LAST], preserve_partitioning=[false]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventTime, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 !=  AND DynamicFilter [ empty ], sort_order_for_reorder=[EventTime@0 ASC NULLS LAST, SearchPhrase@1 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
 
 query T
 SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "EventTime", "SearchPhrase" LIMIT 10;
@@ -746,9 +721,8 @@ physical_plan
 05)--------AggregateExec: mode=FinalPartitioned, gby=[CounterID@0 as CounterID], aggr=[avg(octet_length(hits.URL)), count(Int64(1))]
 06)----------RepartitionExec: partitioning=Hash([CounterID@0], 4), input_partitions=4
 07)------------AggregateExec: mode=Partial, gby=[CounterID@0 as CounterID], aggr=[avg(octet_length(hits.URL)), count(Int64(1))]
-08)--------------FilterExec: URL@1 !=
-09)----------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-10)------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[CounterID, URL], file_type=parquet, predicate=URL@13 != , pruning_predicate=URL_null_count@2 != row_count@3 AND (URL_min@0 !=  OR  != URL_max@1), required_guarantees=[URL not in ()]
+08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[CounterID, URL], file_type=parquet, predicate=URL@13 != , pruning_predicate=URL_null_count@2 != row_count@3 AND (URL_min@0 !=  OR  != URL_max@1), required_guarantees=[URL not in ()]
 
 query IRI
 SELECT "CounterID", AVG(octet_length("URL")) AS l, COUNT(*) AS c FROM hits WHERE "URL" <> '' GROUP BY "CounterID" HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
@@ -774,9 +748,8 @@ physical_plan
 05)--------AggregateExec: mode=FinalPartitioned, gby=[regexp_replace(hits.Referer,Utf8("^https?://(?:www\.)?([^/]+)/.*$"),Utf8("\1"))@0 as regexp_replace(hits.Referer,Utf8("^https?://(?:www\.)?([^/]+)/.*$"),Utf8("\1"))], aggr=[avg(octet_length(hits.Referer)), count(Int64(1)), min(hits.Referer)]
 06)----------RepartitionExec: partitioning=Hash([regexp_replace(hits.Referer,Utf8("^https?://(?:www\.)?([^/]+)/.*$"),Utf8("\1"))@0], 4), input_partitions=4
 07)------------AggregateExec: mode=Partial, gby=[regexp_replace(Referer@0, ^https?://(?:www\.)?([^/]+)/.*$, \1) as regexp_replace(hits.Referer,Utf8("^https?://(?:www\.)?([^/]+)/.*$"),Utf8("\1"))], aggr=[avg(octet_length(hits.Referer)), count(Int64(1)), min(hits.Referer)]
-08)--------------FilterExec: Referer@0 !=
-09)----------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-10)------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[Referer], file_type=parquet, predicate=Referer@14 != , pruning_predicate=Referer_null_count@2 != row_count@3 AND (Referer_min@0 !=  OR  != Referer_max@1), required_guarantees=[Referer not in ()]
+08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[Referer], file_type=parquet, predicate=Referer@14 != , pruning_predicate=Referer_null_count@2 != row_count@3 AND (Referer_min@0 !=  OR  != Referer_max@1), required_guarantees=[Referer not in ()]
 
 query TRIT
 SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(octet_length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
@@ -822,9 +795,8 @@ physical_plan
 04)------AggregateExec: mode=FinalPartitioned, gby=[SearchEngineID@0 as SearchEngineID, ClientIP@1 as ClientIP], aggr=[count(Int64(1)), sum(hits.IsRefresh), avg(hits.ResolutionWidth)]
 05)--------RepartitionExec: partitioning=Hash([SearchEngineID@0, ClientIP@1], 4), input_partitions=4
 06)----------AggregateExec: mode=Partial, gby=[SearchEngineID@3 as SearchEngineID, ClientIP@0 as ClientIP], aggr=[count(Int64(1)), sum(hits.IsRefresh), avg(hits.ResolutionWidth)]
-07)------------FilterExec: SearchPhrase@4 != , projection=[ClientIP@0, IsRefresh@1, ResolutionWidth@2, SearchEngineID@3]
-08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[ClientIP, IsRefresh, ResolutionWidth, SearchEngineID, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 != , pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
+07)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+08)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[ClientIP, IsRefresh, ResolutionWidth, SearchEngineID], file_type=parquet, predicate=SearchPhrase@39 != , pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
 
 query IIIIR
 SELECT "SearchEngineID", "ClientIP", COUNT(*) AS c, SUM("IsRefresh"), AVG("ResolutionWidth") FROM hits WHERE "SearchPhrase" <> '' GROUP BY "SearchEngineID", "ClientIP" ORDER BY c DESC LIMIT 10;
@@ -849,9 +821,8 @@ physical_plan
 04)------AggregateExec: mode=FinalPartitioned, gby=[WatchID@0 as WatchID, ClientIP@1 as ClientIP], aggr=[count(Int64(1)), sum(hits.IsRefresh), avg(hits.ResolutionWidth)]
 05)--------RepartitionExec: partitioning=Hash([WatchID@0, ClientIP@1], 4), input_partitions=4
 06)----------AggregateExec: mode=Partial, gby=[WatchID@0 as WatchID, ClientIP@1 as ClientIP], aggr=[count(Int64(1)), sum(hits.IsRefresh), avg(hits.ResolutionWidth)]
-07)------------FilterExec: SearchPhrase@4 != , projection=[WatchID@0, ClientIP@1, IsRefresh@2, ResolutionWidth@3]
-08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[WatchID, ClientIP, IsRefresh, ResolutionWidth, SearchPhrase], file_type=parquet, predicate=SearchPhrase@39 != , pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
+07)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+08)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[WatchID, ClientIP, IsRefresh, ResolutionWidth], file_type=parquet, predicate=SearchPhrase@39 != , pruning_predicate=SearchPhrase_null_count@2 != row_count@3 AND (SearchPhrase_min@0 !=  OR  != SearchPhrase_max@1), required_guarantees=[SearchPhrase not in ()]
 
 query IIIIR
 SELECT "WatchID", "ClientIP", COUNT(*) AS c, SUM("IsRefresh"), AVG("ResolutionWidth") FROM hits WHERE "SearchPhrase" <> '' GROUP BY "WatchID", "ClientIP" ORDER BY c DESC LIMIT 10;
@@ -996,9 +967,8 @@ physical_plan
 04)------AggregateExec: mode=FinalPartitioned, gby=[URL@0 as URL], aggr=[count(Int64(1))]
 05)--------RepartitionExec: partitioning=Hash([URL@0], 4), input_partitions=4
 06)----------AggregateExec: mode=Partial, gby=[URL@0 as URL], aggr=[count(Int64(1))]
-07)------------FilterExec: CounterID@1 = 62 AND EventDate@0 >= 15887 AND EventDate@0 <= 15917 AND DontCountHits@4 = 0 AND IsRefresh@3 = 0 AND URL@2 != , projection=[URL@2]
-08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventDate, CounterID, URL, IsRefresh, DontCountHits], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND DontCountHits@61 = 0 AND IsRefresh@15 = 0 AND URL@13 != , pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND DontCountHits_null_count@9 != row_count@3 AND DontCountHits_min@7 <= 0 AND 0 <= DontCountHits_max@8 AND IsRefresh_null_count@12 != row_count@3 AND IsRefresh_min@10 <= 0 AND 0 <= IsRefresh_max@11 AND URL_null_count@15 != row_count@3 AND (URL_min@13 !=  OR  != URL_max@14), required_guarantees=[CounterID in (62), DontCountHits in (0), IsRefresh in (0), URL not in ()]
+07)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+08)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[URL], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND DontCountHits@61 = 0 AND IsRefresh@15 = 0 AND URL@13 != , pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND DontCountHits_null_count@9 != row_count@3 AND DontCountHits_min@7 <= 0 AND 0 <= DontCountHits_max@8 AND IsRefresh_null_count@12 != row_count@3 AND IsRefresh_min@10 <= 0 AND 0 <= IsRefresh_max@11 AND URL_null_count@15 != row_count@3 AND (URL_min@13 !=  OR  != URL_max@14), required_guarantees=[CounterID in (62), DontCountHits in (0), IsRefresh in (0), URL not in ()]
 
 query TI
 SELECT "URL", COUNT(*) AS PageViews FROM hits WHERE "CounterID" = 62 AND "EventDate" >= '2013-07-01' AND "EventDate" <= '2013-07-31' AND "DontCountHits" = 0 AND "IsRefresh" = 0 AND "URL" <> '' GROUP BY "URL" ORDER BY PageViews DESC LIMIT 10;
@@ -1023,9 +993,8 @@ physical_plan
 04)------AggregateExec: mode=FinalPartitioned, gby=[Title@0 as Title], aggr=[count(Int64(1))]
 05)--------RepartitionExec: partitioning=Hash([Title@0], 4), input_partitions=4
 06)----------AggregateExec: mode=Partial, gby=[Title@0 as Title], aggr=[count(Int64(1))]
-07)------------FilterExec: CounterID@2 = 62 AND EventDate@1 >= 15887 AND EventDate@1 <= 15917 AND DontCountHits@4 = 0 AND IsRefresh@3 = 0 AND Title@0 != , projection=[Title@0]
-08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[Title, EventDate, CounterID, IsRefresh, DontCountHits], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND DontCountHits@61 = 0 AND IsRefresh@15 = 0 AND Title@2 != , pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND DontCountHits_null_count@9 != row_count@3 AND DontCountHits_min@7 <= 0 AND 0 <= DontCountHits_max@8 AND IsRefresh_null_count@12 != row_count@3 AND IsRefresh_min@10 <= 0 AND 0 <= IsRefresh_max@11 AND Title_null_count@15 != row_count@3 AND (Title_min@13 !=  OR  != Title_max@14), required_guarantees=[CounterID in (62), DontCountHits in (0), IsRefresh in (0), Title not in ()]
+07)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+08)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[Title], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND DontCountHits@61 = 0 AND IsRefresh@15 = 0 AND Title@2 != , pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND DontCountHits_null_count@9 != row_count@3 AND DontCountHits_min@7 <= 0 AND 0 <= DontCountHits_max@8 AND IsRefresh_null_count@12 != row_count@3 AND IsRefresh_min@10 <= 0 AND 0 <= IsRefresh_max@11 AND Title_null_count@15 != row_count@3 AND (Title_min@13 !=  OR  != Title_max@14), required_guarantees=[CounterID in (62), DontCountHits in (0), IsRefresh in (0), Title not in ()]
 
 query TI
 SELECT "Title", COUNT(*) AS PageViews FROM hits WHERE "CounterID" = 62 AND "EventDate" >= '2013-07-01' AND "EventDate" <= '2013-07-31' AND "DontCountHits" = 0 AND "IsRefresh" = 0 AND "Title" <> '' GROUP BY "Title" ORDER BY PageViews DESC LIMIT 10;
@@ -1052,9 +1021,8 @@ physical_plan
 05)--------AggregateExec: mode=FinalPartitioned, gby=[URL@0 as URL], aggr=[count(Int64(1))]
 06)----------RepartitionExec: partitioning=Hash([URL@0], 4), input_partitions=4
 07)------------AggregateExec: mode=Partial, gby=[URL@0 as URL], aggr=[count(Int64(1))]
-08)--------------FilterExec: CounterID@1 = 62 AND EventDate@0 >= 15887 AND EventDate@0 <= 15917 AND IsRefresh@3 = 0 AND IsLink@4 != 0 AND IsDownload@5 = 0, projection=[URL@2]
-09)----------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-10)------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventDate, CounterID, URL, IsRefresh, IsLink, IsDownload], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND IsRefresh@15 = 0 AND IsLink@52 != 0 AND IsDownload@53 = 0, pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND IsRefresh_null_count@9 != row_count@3 AND IsRefresh_min@7 <= 0 AND 0 <= IsRefresh_max@8 AND IsLink_null_count@12 != row_count@3 AND (IsLink_min@10 != 0 OR 0 != IsLink_max@11) AND IsDownload_null_count@15 != row_count@3 AND IsDownload_min@13 <= 0 AND 0 <= IsDownload_max@14, required_guarantees=[CounterID in (62), IsDownload in (0), IsLink not in (0), IsRefresh in (0)]
+08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[URL], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND IsRefresh@15 = 0 AND IsLink@52 != 0 AND IsDownload@53 = 0, pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND IsRefresh_null_count@9 != row_count@3 AND IsRefresh_min@7 <= 0 AND 0 <= IsRefresh_max@8 AND IsLink_null_count@12 != row_count@3 AND (IsLink_min@10 != 0 OR 0 != IsLink_max@11) AND IsDownload_null_count@15 != row_count@3 AND IsDownload_min@13 <= 0 AND 0 <= IsDownload_max@14, required_guarantees=[CounterID in (62), IsDownload in (0), IsLink not in (0), IsRefresh in (0)]
 
 query TI
 SELECT "URL", COUNT(*) AS PageViews FROM hits WHERE "CounterID" = 62 AND "EventDate" >= '2013-07-01' AND "EventDate" <= '2013-07-31' AND "IsRefresh" = 0 AND "IsLink" <> 0 AND "IsDownload" = 0 GROUP BY "URL" ORDER BY PageViews DESC LIMIT 10 OFFSET 1000;
@@ -1081,9 +1049,8 @@ physical_plan
 05)--------AggregateExec: mode=FinalPartitioned, gby=[TraficSourceID@0 as TraficSourceID, SearchEngineID@1 as SearchEngineID, AdvEngineID@2 as AdvEngineID, CASE WHEN hits.SearchEngineID = Int64(0) AND hits.AdvEngineID = Int64(0) THEN hits.Referer ELSE Utf8("") END@3 as CASE WHEN hits.SearchEngineID = Int64(0) AND hits.AdvEngineID = Int64(0) THEN hits.Referer ELSE Utf8("") END, URL@4 as URL], aggr=[count(Int64(1))]
 06)----------RepartitionExec: partitioning=Hash([TraficSourceID@0, SearchEngineID@1, AdvEngineID@2, CASE WHEN hits.SearchEngineID = Int64(0) AND hits.AdvEngineID = Int64(0) THEN hits.Referer ELSE Utf8("") END@3, URL@4], 4), input_partitions=4
 07)------------AggregateExec: mode=Partial, gby=[TraficSourceID@2 as TraficSourceID, SearchEngineID@3 as SearchEngineID, AdvEngineID@4 as AdvEngineID, CASE WHEN SearchEngineID@3 = 0 AND AdvEngineID@4 = 0 THEN Referer@1 ELSE  END as CASE WHEN hits.SearchEngineID = Int64(0) AND hits.AdvEngineID = Int64(0) THEN hits.Referer ELSE Utf8("") END, URL@0 as URL], aggr=[count(Int64(1))]
-08)--------------FilterExec: CounterID@1 = 62 AND EventDate@0 >= 15887 AND EventDate@0 <= 15917 AND IsRefresh@4 = 0, projection=[URL@2, Referer@3, TraficSourceID@5, SearchEngineID@6, AdvEngineID@7]
-09)----------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-10)------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventDate, CounterID, URL, Referer, IsRefresh, TraficSourceID, SearchEngineID, AdvEngineID], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND IsRefresh@15 = 0, pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND IsRefresh_null_count@9 != row_count@3 AND IsRefresh_min@7 <= 0 AND 0 <= IsRefresh_max@8, required_guarantees=[CounterID in (62), IsRefresh in (0)]
+08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[URL, Referer, TraficSourceID, SearchEngineID, AdvEngineID], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND IsRefresh@15 = 0, pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND IsRefresh_null_count@9 != row_count@3 AND IsRefresh_min@7 <= 0 AND 0 <= IsRefresh_max@8, required_guarantees=[CounterID in (62), IsRefresh in (0)]
 
 query IIITTI
 SELECT "TraficSourceID", "SearchEngineID", "AdvEngineID", CASE WHEN ("SearchEngineID" = 0 AND "AdvEngineID" = 0) THEN "Referer" ELSE '' END AS Src, "URL" AS Dst, COUNT(*) AS PageViews FROM hits WHERE "CounterID" = 62 AND "EventDate" >= '2013-07-01' AND "EventDate" <= '2013-07-31' AND "IsRefresh" = 0 GROUP BY "TraficSourceID", "SearchEngineID", "AdvEngineID", Src, Dst ORDER BY PageViews DESC LIMIT 10 OFFSET 1000;
@@ -1110,10 +1077,9 @@ physical_plan
 05)--------AggregateExec: mode=FinalPartitioned, gby=[URLHash@0 as URLHash, EventDate@1 as EventDate], aggr=[count(Int64(1))]
 06)----------RepartitionExec: partitioning=Hash([URLHash@0, EventDate@1], 4), input_partitions=4
 07)------------AggregateExec: mode=Partial, gby=[URLHash@0 as URLHash, EventDate@1 as EventDate], aggr=[count(Int64(1))]
-08)--------------ProjectionExec: expr=[URLHash@0 as URLHash, CAST(CAST(EventDate@1 AS Int32) AS Date32) as EventDate]
-09)----------------FilterExec: CounterID@1 = 62 AND EventDate@0 >= 15887 AND EventDate@0 <= 15917 AND IsRefresh@2 = 0 AND (TraficSourceID@3 = -1 OR TraficSourceID@3 = 6) AND RefererHash@4 = 3594120000172545465, projection=[URLHash@5, EventDate@0]
-10)------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-11)--------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventDate, CounterID, IsRefresh, TraficSourceID, RefererHash, URLHash], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND IsRefresh@15 = 0 AND (TraficSourceID@37 = -1 OR TraficSourceID@37 = 6) AND RefererHash@102 = 3594120000172545465, pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND IsRefresh_null_count@9 != row_count@3 AND IsRefresh_min@7 <= 0 AND 0 <= IsRefresh_max@8 AND (TraficSourceID_null_count@12 != row_count@3 AND TraficSourceID_min@10 <= -1 AND -1 <= TraficSourceID_max@11 OR TraficSourceID_null_count@12 != row_count@3 AND TraficSourceID_min@10 <= 6 AND 6 <= TraficSourceID_max@11) AND RefererHash_null_count@15 != row_count@3 AND RefererHash_min@13 <= 3594120000172545465 AND 3594120000172545465 <= RefererHash_max@14, required_guarantees=[CounterID in (62), IsRefresh in (0), RefererHash in (3594120000172545465), TraficSourceID in (-1, 6)]
+08)--------------ProjectionExec: expr=[URLHash@5 as URLHash, CAST(CAST(EventDate@0 AS Int32) AS Date32) as EventDate]
+09)----------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+10)------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventDate, CounterID, IsRefresh, TraficSourceID, RefererHash, URLHash], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND IsRefresh@15 = 0 AND (TraficSourceID@37 = -1 OR TraficSourceID@37 = 6) AND RefererHash@102 = 3594120000172545465, pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND IsRefresh_null_count@9 != row_count@3 AND IsRefresh_min@7 <= 0 AND 0 <= IsRefresh_max@8 AND (TraficSourceID_null_count@12 != row_count@3 AND TraficSourceID_min@10 <= -1 AND -1 <= TraficSourceID_max@11 OR TraficSourceID_null_count@12 != row_count@3 AND TraficSourceID_min@10 <= 6 AND 6 <= TraficSourceID_max@11) AND RefererHash_null_count@15 != row_count@3 AND RefererHash_min@13 <= 3594120000172545465 AND 3594120000172545465 <= RefererHash_max@14, required_guarantees=[CounterID in (62), IsRefresh in (0), RefererHash in (3594120000172545465), TraficSourceID in (-1, 6)]
 
 query IDI
 SELECT "URLHash", "EventDate", COUNT(*) AS PageViews FROM hits WHERE "CounterID" = 62 AND "EventDate" >= '2013-07-01' AND "EventDate" <= '2013-07-31' AND "IsRefresh" = 0 AND "TraficSourceID" IN (-1, 6) AND "RefererHash" = 3594120000172545465 GROUP BY "URLHash", "EventDate" ORDER BY PageViews DESC LIMIT 10 OFFSET 100;
@@ -1140,9 +1106,8 @@ physical_plan
 05)--------AggregateExec: mode=FinalPartitioned, gby=[WindowClientWidth@0 as WindowClientWidth, WindowClientHeight@1 as WindowClientHeight], aggr=[count(Int64(1))]
 06)----------RepartitionExec: partitioning=Hash([WindowClientWidth@0, WindowClientHeight@1], 4), input_partitions=4
 07)------------AggregateExec: mode=Partial, gby=[WindowClientWidth@0 as WindowClientWidth, WindowClientHeight@1 as WindowClientHeight], aggr=[count(Int64(1))]
-08)--------------FilterExec: CounterID@1 = 62 AND EventDate@0 >= 15887 AND EventDate@0 <= 15917 AND IsRefresh@2 = 0 AND DontCountHits@5 = 0 AND URLHash@6 = 2868770270353813622, projection=[WindowClientWidth@3, WindowClientHeight@4]
-09)----------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-10)------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventDate, CounterID, IsRefresh, WindowClientWidth, WindowClientHeight, DontCountHits, URLHash], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND IsRefresh@15 = 0 AND DontCountHits@61 = 0 AND URLHash@103 = 2868770270353813622, pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND IsRefresh_null_count@9 != row_count@3 AND IsRefresh_min@7 <= 0 AND 0 <= IsRefresh_max@8 AND DontCountHits_null_count@12 != row_count@3 AND DontCountHits_min@10 <= 0 AND 0 <= DontCountHits_max@11 AND URLHash_null_count@15 != row_count@3 AND URLHash_min@13 <= 2868770270353813622 AND 2868770270353813622 <= URLHash_max@14, required_guarantees=[CounterID in (62), DontCountHits in (0), IsRefresh in (0), URLHash in (2868770270353813622)]
+08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[WindowClientWidth, WindowClientHeight], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15887 AND EventDate@5 <= 15917 AND IsRefresh@15 = 0 AND DontCountHits@61 = 0 AND URLHash@103 = 2868770270353813622, pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15887 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15917 AND IsRefresh_null_count@9 != row_count@3 AND IsRefresh_min@7 <= 0 AND 0 <= IsRefresh_max@8 AND DontCountHits_null_count@12 != row_count@3 AND DontCountHits_min@10 <= 0 AND 0 <= DontCountHits_max@11 AND URLHash_null_count@15 != row_count@3 AND URLHash_min@13 <= 2868770270353813622 AND 2868770270353813622 <= URLHash_max@14, required_guarantees=[CounterID in (62), DontCountHits in (0), IsRefresh in (0), URLHash in (2868770270353813622)]
 
 query III
 SELECT "WindowClientWidth", "WindowClientHeight", COUNT(*) AS PageViews FROM hits WHERE "CounterID" = 62 AND "EventDate" >= '2013-07-01' AND "EventDate" <= '2013-07-31' AND "IsRefresh" = 0 AND "DontCountHits" = 0 AND "URLHash" = 2868770270353813622 GROUP BY "WindowClientWidth", "WindowClientHeight" ORDER BY PageViews DESC LIMIT 10 OFFSET 10000;
@@ -1169,9 +1134,8 @@ physical_plan
 05)--------AggregateExec: mode=FinalPartitioned, gby=[date_trunc(Utf8("minute"),to_timestamp_seconds(hits.EventTime))@0 as date_trunc(Utf8("minute"),to_timestamp_seconds(hits.EventTime))], aggr=[count(Int64(1))]
 06)----------RepartitionExec: partitioning=Hash([date_trunc(Utf8("minute"),to_timestamp_seconds(hits.EventTime))@0], 4), input_partitions=4
 07)------------AggregateExec: mode=Partial, gby=[date_trunc(minute, to_timestamp_seconds(EventTime@0)) as date_trunc(Utf8("minute"),to_timestamp_seconds(hits.EventTime))], aggr=[count(Int64(1))]
-08)--------------FilterExec: CounterID@2 = 62 AND EventDate@1 >= 15900 AND EventDate@1 <= 15901 AND IsRefresh@3 = 0 AND DontCountHits@4 = 0, projection=[EventTime@0]
-09)----------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-10)------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventTime, EventDate, CounterID, IsRefresh, DontCountHits], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15900 AND EventDate@5 <= 15901 AND IsRefresh@15 = 0 AND DontCountHits@61 = 0, pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15900 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15901 AND IsRefresh_null_count@9 != row_count@3 AND IsRefresh_min@7 <= 0 AND 0 <= IsRefresh_max@8 AND DontCountHits_null_count@12 != row_count@3 AND DontCountHits_min@10 <= 0 AND 0 <= DontCountHits_max@11, required_guarantees=[CounterID in (62), DontCountHits in (0), IsRefresh in (0)]
+08)--------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
+09)----------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/clickbench_hits_10.parquet]]}, projection=[EventTime], file_type=parquet, predicate=CounterID@6 = 62 AND EventDate@5 >= 15900 AND EventDate@5 <= 15901 AND IsRefresh@15 = 0 AND DontCountHits@61 = 0, pruning_predicate=CounterID_null_count@2 != row_count@3 AND CounterID_min@0 <= 62 AND 62 <= CounterID_max@1 AND EventDate_null_count@5 != row_count@3 AND EventDate_max@4 >= 15900 AND EventDate_null_count@5 != row_count@3 AND EventDate_min@6 <= 15901 AND IsRefresh_null_count@9 != row_count@3 AND IsRefresh_min@7 <= 0 AND 0 <= IsRefresh_max@8 AND DontCountHits_null_count@12 != row_count@3 AND DontCountHits_min@10 <= 0 AND 0 <= DontCountHits_max@11, required_guarantees=[CounterID in (62), DontCountHits in (0), IsRefresh in (0)]
 
 query PI
 SELECT DATE_TRUNC('minute', to_timestamp_seconds("EventTime")) AS M, COUNT(*) AS PageViews FROM hits WHERE "CounterID" = 62 AND "EventDate" >= '2013-07-14' AND "EventDate" <= '2013-07-15' AND "IsRefresh" = 0 AND "DontCountHits" = 0 GROUP BY DATE_TRUNC('minute', to_timestamp_seconds("EventTime")) ORDER BY DATE_TRUNC('minute', M) LIMIT 10 OFFSET 1000;
diff --git a/datafusion/sqllogictest/test_files/cte.slt b/datafusion/sqllogictest/test_files/cte.slt
index 289134d9f140b..c23501d65af42 100644
--- a/datafusion/sqllogictest/test_files/cte.slt
+++ b/datafusion/sqllogictest/test_files/cte.slt
@@ -1699,9 +1699,7 @@ physical_plan
 07)----------FilterExec: k@0 < scalar_subquery(<pending>)
 08)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
 09)--------------WorkTableExec: name=r
-10)--FilterExec: k@0 = 2, projection=[v2@1]
-11)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-12)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/cte/test.parquet]]}, projection=[k, v2], output_ordering=[k@0 ASC NULLS LAST], file_type=parquet, predicate=k@0 = 2, pruning_predicate=k_null_count@2 != row_count@3 AND k_min@0 <= 2 AND 2 <= k_max@1, required_guarantees=[k in (2)]
+10)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/cte/test.parquet]]}, projection=[v2], file_type=parquet, predicate=k@0 = 2, pruning_predicate=k_null_count@2 != row_count@3 AND k_min@0 <= 2 AND 2 <= k_max@1, required_guarantees=[k in (2)]
 
 query II
 with recursive r as (
diff --git a/datafusion/sqllogictest/test_files/explain_tree.slt b/datafusion/sqllogictest/test_files/explain_tree.slt
index fcac86aa21a2e..17dc5d26c99ea 100644
--- a/datafusion/sqllogictest/test_files/explain_tree.slt
+++ b/datafusion/sqllogictest/test_files/explain_tree.slt
@@ -530,29 +530,14 @@ explain SELECT int_col FROM table2 WHERE string_col != 'foo';
 ----
 physical_plan
 01)┌───────────────────────────┐
-02)│         FilterExec        │
+02)│       DataSourceExec      │
 03)│    --------------------   │
-04)│         predicate:        │
-05)│     string_col != foo     │
-06)└─────────────┬─────────────┘
-07)┌─────────────┴─────────────┐
-08)│      RepartitionExec      │
-09)│    --------------------   │
-10)│ partition_count(in->out): │
-11)│           1 -> 4          │
-12)│                           │
-13)│    partitioning_scheme:   │
-14)│     RoundRobinBatch(4)    │
-15)└─────────────┬─────────────┘
-16)┌─────────────┴─────────────┐
-17)│       DataSourceExec      │
-18)│    --------------------   │
-19)│          files: 1         │
-20)│      format: parquet      │
-21)│                           │
-22)│         predicate:        │
-23)│     string_col != foo     │
-24)└───────────────────────────┘
+04)│          files: 1         │
+05)│      format: parquet      │
+06)│                           │
+07)│         predicate:        │
+08)│     string_col != foo     │
+09)└───────────────────────────┘
 
 # Query with filter on memory
 query TT
diff --git a/datafusion/sqllogictest/test_files/grouping_set_repartition.slt b/datafusion/sqllogictest/test_files/grouping_set_repartition.slt
index 16ab90651c8b3..78122b1dd2ed0 100644
--- a/datafusion/sqllogictest/test_files/grouping_set_repartition.slt
+++ b/datafusion/sqllogictest/test_files/grouping_set_repartition.slt
@@ -150,20 +150,17 @@ physical_plan
 09)----------------AggregateExec: mode=FinalPartitioned, gby=[brand@0 as brand], aggr=[sum(sales.amount)]
 10)------------------RepartitionExec: partitioning=Hash([brand@0], 4), input_partitions=4
 11)--------------------AggregateExec: mode=Partial, gby=[brand@0 as brand], aggr=[sum(sales.amount)]
-12)----------------------FilterExec: channel@0 = store, projection=[brand@1, amount@2]
-13)------------------------DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=1/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=2/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=3/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=4/data.parquet]]}, projection=[channel, brand, amount], file_type=parquet, predicate=channel@0 = store, pruning_predicate=channel_null_count@2 != row_count@3 AND channel_min@0 <= store AND store <= channel_max@1, required_guarantees=[channel in (store)]
-14)--------------ProjectionExec: expr=[web as channel, brand@0 as brand, sum(sales.amount)@1 as total]
-15)----------------AggregateExec: mode=FinalPartitioned, gby=[brand@0 as brand], aggr=[sum(sales.amount)]
-16)------------------RepartitionExec: partitioning=Hash([brand@0], 4), input_partitions=4
-17)--------------------AggregateExec: mode=Partial, gby=[brand@0 as brand], aggr=[sum(sales.amount)]
-18)----------------------FilterExec: channel@0 = web, projection=[brand@1, amount@2]
-19)------------------------DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=1/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=2/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=3/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=4/data.parquet]]}, projection=[channel, brand, amount], file_type=parquet, predicate=channel@0 = web, pruning_predicate=channel_null_count@2 != row_count@3 AND channel_min@0 <= web AND web <= channel_max@1, required_guarantees=[channel in (web)]
-20)--------------ProjectionExec: expr=[catalog as channel, brand@0 as brand, sum(sales.amount)@1 as total]
-21)----------------AggregateExec: mode=FinalPartitioned, gby=[brand@0 as brand], aggr=[sum(sales.amount)]
-22)------------------RepartitionExec: partitioning=Hash([brand@0], 4), input_partitions=4
-23)--------------------AggregateExec: mode=Partial, gby=[brand@0 as brand], aggr=[sum(sales.amount)]
-24)----------------------FilterExec: channel@0 = catalog, projection=[brand@1, amount@2]
-25)------------------------DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=1/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=2/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=3/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=4/data.parquet]]}, projection=[channel, brand, amount], file_type=parquet, predicate=channel@0 = catalog, pruning_predicate=channel_null_count@2 != row_count@3 AND channel_min@0 <= catalog AND catalog <= channel_max@1, required_guarantees=[channel in (catalog)]
+12)----------------------DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=1/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=2/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=3/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=4/data.parquet]]}, projection=[brand, amount], file_type=parquet, predicate=channel@0 = store, pruning_predicate=channel_null_count@2 != row_count@3 AND channel_min@0 <= store AND store <= channel_max@1, required_guarantees=[channel in (store)]
+13)--------------ProjectionExec: expr=[web as channel, brand@0 as brand, sum(sales.amount)@1 as total]
+14)----------------AggregateExec: mode=FinalPartitioned, gby=[brand@0 as brand], aggr=[sum(sales.amount)]
+15)------------------RepartitionExec: partitioning=Hash([brand@0], 4), input_partitions=4
+16)--------------------AggregateExec: mode=Partial, gby=[brand@0 as brand], aggr=[sum(sales.amount)]
+17)----------------------DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=1/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=2/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=3/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=4/data.parquet]]}, projection=[brand, amount], file_type=parquet, predicate=channel@0 = web, pruning_predicate=channel_null_count@2 != row_count@3 AND channel_min@0 <= web AND web <= channel_max@1, required_guarantees=[channel in (web)]
+18)--------------ProjectionExec: expr=[catalog as channel, brand@0 as brand, sum(sales.amount)@1 as total]
+19)----------------AggregateExec: mode=FinalPartitioned, gby=[brand@0 as brand], aggr=[sum(sales.amount)]
+20)------------------RepartitionExec: partitioning=Hash([brand@0], 4), input_partitions=4
+21)--------------------AggregateExec: mode=Partial, gby=[brand@0 as brand], aggr=[sum(sales.amount)]
+22)----------------------DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=1/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=2/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=3/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/grouping_set_repartition/part=4/data.parquet]]}, projection=[brand, amount], file_type=parquet, predicate=channel@0 = catalog, pruning_predicate=channel_null_count@2 != row_count@3 AND channel_min@0 <= catalog AND catalog <= channel_max@1, required_guarantees=[channel in (catalog)]
 
 query TTI rowsort
 SELECT channel, brand, SUM(total) as grand_total
diff --git a/datafusion/sqllogictest/test_files/monotonic_projection_test.slt b/datafusion/sqllogictest/test_files/monotonic_projection_test.slt
index 4ecdbca24a116..944eac1ce025a 100644
--- a/datafusion/sqllogictest/test_files/monotonic_projection_test.slt
+++ b/datafusion/sqllogictest/test_files/monotonic_projection_test.slt
@@ -238,11 +238,8 @@ logical_plan
 03)----Filter: concat_equality_ordered.c = concat(concat_equality_ordered.a, concat_equality_ordered.b)
 04)------TableScan: concat_equality_ordered projection=[c, a, b], partial_filters=[concat_equality_ordered.c = concat(concat_equality_ordered.a, concat_equality_ordered.b)]
 physical_plan
-01)SortPreservingMergeExec: [a@0 ASC NULLS LAST, b@1 ASC NULLS LAST]
-02)--SortExec: expr=[a@0 ASC NULLS LAST, b@1 ASC NULLS LAST], preserve_partitioning=[true]
-03)----FilterExec: c@0 = concat(a@1, b@2), projection=[a@1, b@2]
-04)------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, maintains_sort_order=true
-05)--------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/monotonic_projection_test/concat_equality_ordered.parquet]]}, projection=[c, a, b], output_ordering=[c@0 ASC NULLS LAST, a@1 ASC NULLS LAST, b@2 ASC NULLS LAST], file_type=parquet, predicate=c@0 = concat(a@1, b@2)
+01)SortExec: expr=[a@0 ASC NULLS LAST, b@1 ASC NULLS LAST], preserve_partitioning=[false]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/monotonic_projection_test/concat_equality_ordered.parquet]]}, projection=[a, b], file_type=parquet, predicate=c@0 = concat(a@1, b@2), sort_order_for_reorder=[a@0 ASC NULLS LAST, b@1 ASC NULLS LAST]
 
 query TT
 SELECT a, b
diff --git a/datafusion/sqllogictest/test_files/parquet.slt b/datafusion/sqllogictest/test_files/parquet.slt
index 9750038fed73f..7708370c0425a 100644
--- a/datafusion/sqllogictest/test_files/parquet.slt
+++ b/datafusion/sqllogictest/test_files/parquet.slt
@@ -457,10 +457,7 @@ EXPLAIN
 logical_plan
 01)Filter: CAST(binary_as_string_default.binary_col AS Utf8View) LIKE Utf8View("%a%") AND CAST(binary_as_string_default.largebinary_col AS Utf8View) LIKE Utf8View("%a%") AND CAST(binary_as_string_default.binaryview_col AS Utf8View) LIKE Utf8View("%a%")
 02)--TableScan: binary_as_string_default projection=[binary_col, largebinary_col, binaryview_col], partial_filters=[CAST(binary_as_string_default.binary_col AS Utf8View) LIKE Utf8View("%a%"), CAST(binary_as_string_default.largebinary_col AS Utf8View) LIKE Utf8View("%a%"), CAST(binary_as_string_default.binaryview_col AS Utf8View) LIKE Utf8View("%a%")]
-physical_plan
-01)FilterExec: CAST(binary_col@0 AS Utf8View) LIKE %a% AND CAST(largebinary_col@1 AS Utf8View) LIKE %a% AND CAST(binaryview_col@2 AS Utf8View) LIKE %a%
-02)--RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet/binary_as_string.parquet]]}, projection=[binary_col, largebinary_col, binaryview_col], file_type=parquet, predicate=CAST(binary_col@0 AS Utf8View) LIKE %a% AND CAST(largebinary_col@1 AS Utf8View) LIKE %a% AND CAST(binaryview_col@2 AS Utf8View) LIKE %a%
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet/binary_as_string.parquet]]}, projection=[binary_col, largebinary_col, binaryview_col], file_type=parquet, predicate=CAST(binary_col@0 AS Utf8View) LIKE %a% AND CAST(largebinary_col@1 AS Utf8View) LIKE %a% AND CAST(binaryview_col@2 AS Utf8View) LIKE %a%
 
 
 statement ok
@@ -504,10 +501,7 @@ EXPLAIN
 logical_plan
 01)Filter: binary_as_string_option.binary_col LIKE Utf8View("%a%") AND binary_as_string_option.largebinary_col LIKE Utf8View("%a%") AND binary_as_string_option.binaryview_col LIKE Utf8View("%a%")
 02)--TableScan: binary_as_string_option projection=[binary_col, largebinary_col, binaryview_col], partial_filters=[binary_as_string_option.binary_col LIKE Utf8View("%a%"), binary_as_string_option.largebinary_col LIKE Utf8View("%a%"), binary_as_string_option.binaryview_col LIKE Utf8View("%a%")]
-physical_plan
-01)FilterExec: binary_col@0 LIKE %a% AND largebinary_col@1 LIKE %a% AND binaryview_col@2 LIKE %a%
-02)--RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet/binary_as_string.parquet]]}, projection=[binary_col, largebinary_col, binaryview_col], file_type=parquet, predicate=binary_col@0 LIKE %a% AND largebinary_col@1 LIKE %a% AND binaryview_col@2 LIKE %a%
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet/binary_as_string.parquet]]}, projection=[binary_col, largebinary_col, binaryview_col], file_type=parquet, predicate=binary_col@0 LIKE %a% AND largebinary_col@1 LIKE %a% AND binaryview_col@2 LIKE %a%
 
 
 statement ok
@@ -554,10 +548,7 @@ EXPLAIN
 logical_plan
 01)Filter: binary_as_string_both.binary_col LIKE Utf8View("%a%") AND binary_as_string_both.largebinary_col LIKE Utf8View("%a%") AND binary_as_string_both.binaryview_col LIKE Utf8View("%a%")
 02)--TableScan: binary_as_string_both projection=[binary_col, largebinary_col, binaryview_col], partial_filters=[binary_as_string_both.binary_col LIKE Utf8View("%a%"), binary_as_string_both.largebinary_col LIKE Utf8View("%a%"), binary_as_string_both.binaryview_col LIKE Utf8View("%a%")]
-physical_plan
-01)FilterExec: binary_col@0 LIKE %a% AND largebinary_col@1 LIKE %a% AND binaryview_col@2 LIKE %a%
-02)--RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet/binary_as_string.parquet]]}, projection=[binary_col, largebinary_col, binaryview_col], file_type=parquet, predicate=binary_col@0 LIKE %a% AND largebinary_col@1 LIKE %a% AND binaryview_col@2 LIKE %a%
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet/binary_as_string.parquet]]}, projection=[binary_col, largebinary_col, binaryview_col], file_type=parquet, predicate=binary_col@0 LIKE %a% AND largebinary_col@1 LIKE %a% AND binaryview_col@2 LIKE %a%
 
 
 statement ok
@@ -668,10 +659,7 @@ explain select * from foo where starts_with(column1, 'f');
 logical_plan
 01)Filter: foo.column1 LIKE Utf8View("f%")
 02)--TableScan: foo projection=[column1], partial_filters=[foo.column1 LIKE Utf8View("f%")]
-physical_plan
-01)FilterExec: column1@0 LIKE f%
-02)--RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet/foo.parquet]]}, projection=[column1], file_type=parquet, predicate=column1@0 LIKE f%, pruning_predicate=column1_null_count@2 != row_count@3 AND column1_min@0 <= g AND f <= column1_max@1, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet/foo.parquet]]}, projection=[column1], file_type=parquet, predicate=column1@0 LIKE f%, pruning_predicate=column1_null_count@2 != row_count@3 AND column1_min@0 <= g AND f <= column1_max@1, required_guarantees=[]
 
 statement ok
 drop table foo
diff --git a/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt b/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt
index 2bdb99acfe3d3..224640cab0997 100644
--- a/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt
+++ b/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt
@@ -94,9 +94,7 @@ logical_plan
 physical_plan
 01)SortPreservingMergeExec: [a@0 ASC NULLS LAST]
 02)--SortExec: expr=[a@0 ASC NULLS LAST], preserve_partitioning=[true]
-03)----FilterExec: b@1 > 2, projection=[a@0]
-04)------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2
-05)--------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a, b], file_type=parquet, predicate=b@1 > 2, pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 > 2, required_guarantees=[]
+03)----DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a], file_type=parquet, predicate=b@1 > 2, sort_order_for_reorder=[a@0 ASC NULLS LAST], pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 > 2, required_guarantees=[]
 
 query TT
 EXPLAIN select a from t_pushdown where b > 2 ORDER BY a;
@@ -131,9 +129,7 @@ logical_plan
 04)------TableScan: t projection=[a, b], partial_filters=[t.b = Int32(2)]
 physical_plan
 01)CoalescePartitionsExec
-02)--FilterExec: b@1 = 2, projection=[a@0]
-03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2
-04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a, b], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)]
+02)--DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)]
 
 query TT
 EXPLAIN select a from t_pushdown where b = 2 ORDER BY b;
@@ -262,9 +258,7 @@ logical_plan
 physical_plan
 01)SortPreservingMergeExec: [a@0 ASC NULLS LAST]
 02)--SortExec: expr=[a@0 ASC NULLS LAST], preserve_partitioning=[true]
-03)----FilterExec: b@1 > 2, projection=[a@0]
-04)------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2
-05)--------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a, b], file_type=parquet, predicate=b@1 > 2, pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 > 2, required_guarantees=[]
+03)----DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a], file_type=parquet, predicate=b@1 > 2, sort_order_for_reorder=[a@0 ASC NULLS LAST], pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 > 2, required_guarantees=[]
 
 query TT
 EXPLAIN select a from t_pushdown where b > 2 ORDER BY a;
@@ -299,9 +293,7 @@ logical_plan
 04)------TableScan: t projection=[a, b], partial_filters=[t.b = Int32(2)]
 physical_plan
 01)CoalescePartitionsExec
-02)--FilterExec: b@1 = 2, projection=[a@0]
-03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2
-04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a, b], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)]
+02)--DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)]
 
 query TT
 EXPLAIN select a from t_pushdown where b = 2 ORDER BY b;
@@ -337,9 +329,7 @@ logical_plan
 physical_plan
 01)SortPreservingMergeExec: [a@0 ASC NULLS LAST]
 02)--SortExec: expr=[a@0 ASC NULLS LAST], preserve_partitioning=[true]
-03)----FilterExec: b@1 > 2, projection=[a@0]
-04)------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2
-05)--------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a, b], file_type=parquet, predicate=b@1 > 2, pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 > 2, required_guarantees=[]
+03)----DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a], file_type=parquet, predicate=b@1 > 2, sort_order_for_reorder=[a@0 ASC NULLS LAST], pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 > 2, required_guarantees=[]
 
 query T
 select a from t_pushdown where b = 2 ORDER BY b;
diff --git a/datafusion/sqllogictest/test_files/parquet_max_row_group_bytes.slt b/datafusion/sqllogictest/test_files/parquet_max_row_group_bytes.slt
index 21c5ef22c9c52..5223822e79f26 100644
--- a/datafusion/sqllogictest/test_files/parquet_max_row_group_bytes.slt
+++ b/datafusion/sqllogictest/test_files/parquet_max_row_group_bytes.slt
@@ -127,9 +127,7 @@ LOCATION 'test_files/scratch/parquet_max_row_group_bytes/rg_count_size_only/';
 query TT
 EXPLAIN ANALYZE SELECT * FROM rg_count_size_only WHERE id >= 0;
 ----
-Plan with Metrics
-01)FilterExec: id@0 >= 0<slt:ignore>
-02)--DataSourceExec: <slt:ignore>row_groups_pruned_statistics=5 total<slt:ignore>
+Plan with Metrics DataSourceExec: <slt:ignore>row_groups_pruned_statistics=5 total<slt:ignore>
 
 # Both limits set: the byte limit also flushes the sub-1000-row remainders that
 # the row-count limit would otherwise carry into the next batch, so the file is
@@ -148,9 +146,7 @@ LOCATION 'test_files/scratch/parquet_max_row_group_bytes/rg_count_size_and_bytes
 query TT
 EXPLAIN ANALYZE SELECT * FROM rg_count_size_and_bytes WHERE id >= 0;
 ----
-Plan with Metrics
-01)FilterExec: id@0 >= 0<slt:ignore>
-02)--DataSourceExec: <slt:ignore>row_groups_pruned_statistics=8 total<slt:ignore>
+Plan with Metrics DataSourceExec: <slt:ignore>row_groups_pruned_statistics=8 total<slt:ignore>
 
 # Byte limit drives alone: the row-count limit is far larger than the data, so
 # only the byte limit splits. Each 1024-row batch fills a fresh (empty) row
@@ -169,9 +165,7 @@ LOCATION 'test_files/scratch/parquet_max_row_group_bytes/rg_count_bytes_only/';
 query TT
 EXPLAIN ANALYZE SELECT * FROM rg_count_bytes_only WHERE id >= 0;
 ----
-Plan with Metrics
-01)FilterExec: id@0 >= 0<slt:ignore>
-02)--DataSourceExec: <slt:ignore>row_groups_pruned_statistics=4 total<slt:ignore>
+Plan with Metrics DataSourceExec: <slt:ignore>row_groups_pruned_statistics=4 total<slt:ignore>
 
 statement ok
 reset datafusion.execution.parquet.allow_single_file_parallelism;
diff --git a/datafusion/sqllogictest/test_files/parquet_statistics.slt b/datafusion/sqllogictest/test_files/parquet_statistics.slt
index 2c9a29c74b3c1..1eea640a17cb0 100644
--- a/datafusion/sqllogictest/test_files/parquet_statistics.slt
+++ b/datafusion/sqllogictest/test_files/parquet_statistics.slt
@@ -58,10 +58,7 @@ LOCATION 'test_files/scratch/parquet_statistics/test_table';
 query TT
 EXPLAIN SELECT * FROM test_table WHERE column1 = 1;
 ----
-physical_plan
-01)FilterExec: column1@0 = 1, statistics=[Rows=Inexact(2), Bytes=Inexact(10), [(Col[0]: Min=Exact(Int64(1)) Max=Exact(Int64(1)) Null=Exact(0) Distinct=Exact(1) ScanBytes=Inexact(10))]]
-02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2, statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Int64(1)) Max=Inexact(Int64(4)) Null=Inexact(0) ScanBytes=Inexact(40))]]
-03)----DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/1.parquet]]}, projection=[column1], file_type=parquet, predicate=column1@0 = 1, pruning_predicate=column1_null_count@2 != row_count@3 AND column1_min@0 <= 1 AND 1 <= column1_max@1, required_guarantees=[column1 in (1)], statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Int64(1)) Max=Inexact(Int64(4)) Null=Inexact(0) ScanBytes=Inexact(40))]]
+physical_plan DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/1.parquet]]}, projection=[column1], file_type=parquet, predicate=column1@0 = 1, pruning_predicate=column1_null_count@2 != row_count@3 AND column1_min@0 <= 1 AND 1 <= column1_max@1, required_guarantees=[column1 in (1)], statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Int64(1)) Max=Inexact(Int64(4)) Null=Inexact(0) ScanBytes=Inexact(40))]]
 
 # cleanup
 statement ok
@@ -83,10 +80,7 @@ LOCATION 'test_files/scratch/parquet_statistics/test_table';
 query TT
 EXPLAIN SELECT * FROM test_table WHERE column1 = 1;
 ----
-physical_plan
-01)FilterExec: column1@0 = 1, statistics=[Rows=Inexact(2), Bytes=Inexact(10), [(Col[0]: Min=Exact(Int64(1)) Max=Exact(Int64(1)) Null=Exact(0) Distinct=Exact(1) ScanBytes=Inexact(10))]]
-02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2, statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Int64(1)) Max=Inexact(Int64(4)) Null=Inexact(0) ScanBytes=Inexact(40))]]
-03)----DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/1.parquet]]}, projection=[column1], file_type=parquet, predicate=column1@0 = 1, pruning_predicate=column1_null_count@2 != row_count@3 AND column1_min@0 <= 1 AND 1 <= column1_max@1, required_guarantees=[column1 in (1)], statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Int64(1)) Max=Inexact(Int64(4)) Null=Inexact(0) ScanBytes=Inexact(40))]]
+physical_plan DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/1.parquet]]}, projection=[column1], file_type=parquet, predicate=column1@0 = 1, pruning_predicate=column1_null_count@2 != row_count@3 AND column1_min@0 <= 1 AND 1 <= column1_max@1, required_guarantees=[column1 in (1)], statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Int64(1)) Max=Inexact(Int64(4)) Null=Inexact(0) ScanBytes=Inexact(40))]]
 
 # cleanup
 statement ok
@@ -108,10 +102,7 @@ LOCATION 'test_files/scratch/parquet_statistics/test_table';
 query TT
 EXPLAIN SELECT * FROM test_table WHERE column1 = 1;
 ----
-physical_plan
-01)FilterExec: column1@0 = 1, statistics=[Rows=Absent, Bytes=Absent, [(Col[0]: Min=Exact(Int64(1)) Max=Exact(Int64(1)) Null=Exact(0) Distinct=Inexact(1))]]
-02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2, statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:)]]
-03)----DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/1.parquet]]}, projection=[column1], file_type=parquet, predicate=column1@0 = 1, pruning_predicate=column1_null_count@2 != row_count@3 AND column1_min@0 <= 1 AND 1 <= column1_max@1, required_guarantees=[column1 in (1)], statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:)]]
+physical_plan DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/1.parquet]]}, projection=[column1], file_type=parquet, predicate=column1@0 = 1, pruning_predicate=column1_null_count@2 != row_count@3 AND column1_min@0 <= 1 AND 1 <= column1_max@1, required_guarantees=[column1 in (1)], statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:)]]
 
 # cleanup
 statement ok
@@ -151,37 +142,25 @@ LOCATION 'test_files/scratch/parquet_statistics/typed_table.parquet';
 query TT
 EXPLAIN SELECT i8 FROM typed_table WHERE i8 = 2;
 ----
-physical_plan
-01)FilterExec: i8@0 = 2, statistics=[Rows=Inexact(1), Bytes=Inexact(1), [(Col[0]: Min=Exact(Int8(2)) Max=Exact(Int8(2)) Null=Exact(0) Distinct=Exact(1) ScanBytes=Inexact(1))]]
-02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, statistics=[Rows=Inexact(5), Bytes=Inexact(5), [(Col[0]: Min=Inexact(Int8(1)) Max=Inexact(Int8(5)) Null=Inexact(0) ScanBytes=Inexact(5))]]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/typed_table.parquet]]}, projection=[i8], file_type=parquet, predicate=i8@0 = 2, pruning_predicate=i8_null_count@2 != row_count@3 AND i8_min@0 <= 2 AND 2 <= i8_max@1, required_guarantees=[i8 in (2)], statistics=[Rows=Inexact(5), Bytes=Inexact(5), [(Col[0]: Min=Inexact(Int8(1)) Max=Inexact(Int8(5)) Null=Inexact(0) ScanBytes=Inexact(5))]]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/typed_table.parquet]]}, projection=[i8], file_type=parquet, predicate=i8@0 = 2, pruning_predicate=i8_null_count@2 != row_count@3 AND i8_min@0 <= 2 AND 2 <= i8_max@1, required_guarantees=[i8 in (2)], statistics=[Rows=Inexact(5), Bytes=Inexact(5), [(Col[0]: Min=Inexact(Int8(1)) Max=Inexact(Int8(5)) Null=Inexact(0) ScanBytes=Inexact(5))]]
 
 # Int64 equality
 query TT
 EXPLAIN SELECT i64 FROM typed_table WHERE i64 = 2;
 ----
-physical_plan
-01)FilterExec: i64@0 = 2, statistics=[Rows=Inexact(1), Bytes=Inexact(8), [(Col[0]: Min=Exact(Int64(2)) Max=Exact(Int64(2)) Null=Exact(0) Distinct=Exact(1) ScanBytes=Inexact(8))]]
-02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Int64(1)) Max=Inexact(Int64(5)) Null=Inexact(0) ScanBytes=Inexact(40))]]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/typed_table.parquet]]}, projection=[i64], file_type=parquet, predicate=i64@1 = 2, pruning_predicate=i64_null_count@2 != row_count@3 AND i64_min@0 <= 2 AND 2 <= i64_max@1, required_guarantees=[i64 in (2)], statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Int64(1)) Max=Inexact(Int64(5)) Null=Inexact(0) ScanBytes=Inexact(40))]]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/typed_table.parquet]]}, projection=[i64], file_type=parquet, predicate=i64@1 = 2, pruning_predicate=i64_null_count@2 != row_count@3 AND i64_min@0 <= 2 AND 2 <= i64_max@1, required_guarantees=[i64 in (2)], statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Int64(1)) Max=Inexact(Int64(5)) Null=Inexact(0) ScanBytes=Inexact(40))]]
 
 # Float32 equality
 query TT
 EXPLAIN SELECT f32 FROM typed_table WHERE f32 = 2.5;
 ----
-physical_plan
-01)FilterExec: CAST(f32@0 AS Float64) = 2.5, statistics=[Rows=Inexact(1), Bytes=Inexact(1), [(Col[0]: Min=Exact(Float32(2.5)) Max=Exact(Float32(2.5)) Null=Inexact(0) Distinct=Exact(1) ScanBytes=Inexact(1))]]
-02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, statistics=[Rows=Inexact(5), Bytes=Inexact(20), [(Col[0]: Min=Inexact(Float32(1.5)) Max=Inexact(Float32(5.5)) Null=Inexact(0) ScanBytes=Inexact(20))]]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/typed_table.parquet]]}, projection=[f32], file_type=parquet, predicate=CAST(f32@2 AS Float64) = 2.5, pruning_predicate=f32_null_count@2 != row_count@3 AND CAST(f32_min@0 AS Float64) <= 2.5 AND 2.5 <= CAST(f32_max@1 AS Float64), required_guarantees=[], statistics=[Rows=Inexact(5), Bytes=Inexact(20), [(Col[0]: Min=Inexact(Float32(1.5)) Max=Inexact(Float32(5.5)) Null=Inexact(0) ScanBytes=Inexact(20))]]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/typed_table.parquet]]}, projection=[f32], file_type=parquet, predicate=CAST(f32@2 AS Float64) = 2.5, pruning_predicate=f32_null_count@2 != row_count@3 AND CAST(f32_min@0 AS Float64) <= 2.5 AND 2.5 <= CAST(f32_max@1 AS Float64), required_guarantees=[], statistics=[Rows=Inexact(5), Bytes=Inexact(20), [(Col[0]: Min=Inexact(Float32(1.5)) Max=Inexact(Float32(5.5)) Null=Inexact(0) ScanBytes=Inexact(20))]]
 
 # Reversed operand order: literal = column (Float64)
 query TT
 EXPLAIN SELECT f64 FROM typed_table WHERE 2.5 = f64;
 ----
-physical_plan
-01)FilterExec: f64@0 = 2.5, statistics=[Rows=Inexact(1), Bytes=Inexact(1), [(Col[0]: Min=Exact(Float64(2.5)) Max=Exact(Float64(2.5)) Null=Exact(0) Distinct=Exact(1) ScanBytes=Inexact(1))]]
-02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Float64(1.5)) Max=Inexact(Float64(5.5)) Null=Inexact(0) ScanBytes=Inexact(40))]]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/typed_table.parquet]]}, projection=[f64], file_type=parquet, predicate=f64@3 = 2.5, pruning_predicate=f64_null_count@2 != row_count@3 AND f64_min@0 <= 2.5 AND 2.5 <= f64_max@1, required_guarantees=[f64 in (2.5)], statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Float64(1.5)) Max=Inexact(Float64(5.5)) Null=Inexact(0) ScanBytes=Inexact(40))]]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/typed_table.parquet]]}, projection=[f64], file_type=parquet, predicate=f64@3 = 2.5, pruning_predicate=f64_null_count@2 != row_count@3 AND f64_min@0 <= 2.5 AND 2.5 <= f64_max@1, required_guarantees=[f64 in (2.5)], statistics=[Rows=Inexact(5), Bytes=Inexact(40), [(Col[0]: Min=Inexact(Float64(1.5)) Max=Inexact(Float64(5.5)) Null=Inexact(0) ScanBytes=Inexact(40))]]
 
 statement ok
 DROP TABLE typed_table;
diff --git a/datafusion/sqllogictest/test_files/preserve_file_partitioning.slt b/datafusion/sqllogictest/test_files/preserve_file_partitioning.slt
index e2dd22cc82bba..2eaeca9827139 100644
--- a/datafusion/sqllogictest/test_files/preserve_file_partitioning.slt
+++ b/datafusion/sqllogictest/test_files/preserve_file_partitioning.slt
@@ -363,11 +363,8 @@ physical_plan
 04)------RepartitionExec: partitioning=Hash([f_dkey@0], 3), input_partitions=3, preserve_order=true, sort_exprs=f_dkey@0 ASC NULLS LAST
 05)--------AggregateExec: mode=Partial, gby=[f_dkey@1 as f_dkey], aggr=[max(d.env), max(d.service), count(Int64(1)), sum(f.value)], ordering_mode=Sorted
 06)----------HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(d_dkey@0, f_dkey@1)], projection=[value@3, f_dkey@4, env@1, service@2]
-07)------------CoalescePartitionsExec
-08)--------------FilterExec: service@2 = log
-09)----------------RepartitionExec: partitioning=RoundRobinBatch(3), input_partitions=1
-10)------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/dimension/data.parquet]]}, projection=[d_dkey, env, service], file_type=parquet, predicate=service@2 = log, pruning_predicate=service_null_count@2 != row_count@3 AND service_min@0 <= log AND log <= service_max@1, required_guarantees=[service in (log)]
-11)------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=C/data.parquet]]}, projection=[value, f_dkey], output_ordering=[f_dkey@1 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
+07)------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/dimension/data.parquet]]}, projection=[d_dkey, env, service], file_type=parquet, predicate=service@2 = log, pruning_predicate=service_null_count@2 != row_count@3 AND service_min@0 <= log AND log <= service_max@1, required_guarantees=[service in (log)]
+08)------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=C/data.parquet]]}, projection=[value, f_dkey], output_ordering=[f_dkey@1 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
 
 # Verify results without optimization
 query TTTIR rowsort
@@ -414,11 +411,8 @@ physical_plan
 02)--ProjectionExec: expr=[f_dkey@0 as f_dkey, max(d.env)@1 as max(d.env), max(d.service)@2 as max(d.service), count(Int64(1))@3 as count(*), sum(f.value)@4 as sum(f.value)]
 03)----AggregateExec: mode=SinglePartitioned, gby=[f_dkey@1 as f_dkey], aggr=[max(d.env), max(d.service), count(Int64(1)), sum(f.value)], ordering_mode=Sorted
 04)------HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(d_dkey@0, f_dkey@1)], projection=[value@3, f_dkey@4, env@1, service@2]
-05)--------CoalescePartitionsExec
-06)----------FilterExec: service@2 = log
-07)------------RepartitionExec: partitioning=RoundRobinBatch(3), input_partitions=1
-08)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/dimension/data.parquet]]}, projection=[d_dkey, env, service], file_type=parquet, predicate=service@2 = log, pruning_predicate=service_null_count@2 != row_count@3 AND service_min@0 <= log AND log <= service_max@1, required_guarantees=[service in (log)]
-09)--------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=C/data.parquet]]}, projection=[value, f_dkey], output_ordering=[f_dkey@1 ASC NULLS LAST], output_partitioning=Hash([f_dkey@1], 3), file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
+05)--------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/dimension/data.parquet]]}, projection=[d_dkey, env, service], file_type=parquet, predicate=service@2 = log, pruning_predicate=service_null_count@2 != row_count@3 AND service_min@0 <= log AND log <= service_max@1, required_guarantees=[service in (log)]
+06)--------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=C/data.parquet]]}, projection=[value, f_dkey], output_ordering=[f_dkey@1 ASC NULLS LAST], output_partitioning=Hash([f_dkey@1], 3), file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
 
 query TTTIR rowsort
 SELECT f.f_dkey, MAX(d.env), MAX(d.service), count(*), sum(f.value)
diff --git a/datafusion/sqllogictest/test_files/projection.slt b/datafusion/sqllogictest/test_files/projection.slt
index 5cfe4ba76bef3..f3844cc59f4a1 100644
--- a/datafusion/sqllogictest/test_files/projection.slt
+++ b/datafusion/sqllogictest/test_files/projection.slt
@@ -342,7 +342,4 @@ logical_plan
 01)Projection:
 02)--Filter: t1.a > Int64(1)
 03)----TableScan: t1 projection=[a], partial_filters=[t1.a > Int64(1)]
-physical_plan
-01)FilterExec: a@0 > 1, projection=[]
-02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection/17513.parquet]]}, projection=[a], file_type=parquet, predicate=a@0 > 1, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 > 1, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection/17513.parquet]]}, file_type=parquet, predicate=a@0 > 1, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 > 1, required_guarantees=[]
diff --git a/datafusion/sqllogictest/test_files/projection_pushdown.slt b/datafusion/sqllogictest/test_files/projection_pushdown.slt
index c92c95fdfbc59..5b6af161adc24 100644
--- a/datafusion/sqllogictest/test_files/projection_pushdown.slt
+++ b/datafusion/sqllogictest/test_files/projection_pushdown.slt
@@ -239,10 +239,7 @@ logical_plan
 02)--Filter: simple_struct.id > Int64(2)
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
-physical_plan
-01)ProjectionExec: expr=[id@1 as id, __datafusion_extracted_1@0 as simple_struct.s[value]]
-02)--FilterExec: id@1 > 2
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id, get_field(s@1, value) as simple_struct.s[value]], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
 
 # Verify correctness
 query II
@@ -264,10 +261,7 @@ logical_plan
 02)--Filter: simple_struct.id > Int64(2)
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
-physical_plan
-01)ProjectionExec: expr=[id@1 as id, __datafusion_extracted_1@0 + 1 as simple_struct.s[value] + Int64(1)]
-02)--FilterExec: id@1 > 2
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id, get_field(s@1, value) + 1 as simple_struct.s[value] + Int64(1)], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
 
 # Verify correctness
 query II
@@ -289,10 +283,7 @@ logical_plan
 02)--Filter: __datafusion_extracted_1 > Int64(150)
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id, get_field(simple_struct.s, Utf8("label")) AS __datafusion_extracted_2
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[get_field(simple_struct.s, Utf8("value")) > Int64(150)]
-physical_plan
-01)ProjectionExec: expr=[id@0 as id, __datafusion_extracted_2@1 as simple_struct.s[label]]
-02)--FilterExec: __datafusion_extracted_1@0 > 150, projection=[id@1, __datafusion_extracted_2@2]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id, get_field(s@1, label) as __datafusion_extracted_2], file_type=parquet, predicate=get_field(s@1, value) > 150
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id, get_field(s@1, label) as simple_struct.s[label]], file_type=parquet, predicate=get_field(s@1, value) > 150
 
 # Verify correctness
 query IT
@@ -567,8 +558,7 @@ logical_plan
 physical_plan
 01)ProjectionExec: expr=[id@1 as id, __datafusion_extracted_1@0 as simple_struct.s[value]]
 02)--SortExec: expr=[__datafusion_extracted_1@0 ASC NULLS LAST], preserve_partitioning=[false]
-03)----FilterExec: id@1 > 1
-04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 1, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 1, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
 
 # Verify correctness
 query II
@@ -595,8 +585,7 @@ logical_plan
 physical_plan
 01)ProjectionExec: expr=[id@1 as id, __datafusion_extracted_1@0 as simple_struct.s[value]]
 02)--SortExec: TopK(fetch=2), expr=[__datafusion_extracted_1@0 ASC NULLS LAST], preserve_partitioning=[false]
-03)----FilterExec: id@1 > 1
-04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 1 AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 1 AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
 
 # Verify correctness
 query II
@@ -620,9 +609,7 @@ logical_plan
 05)--------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(1)]
 physical_plan
 01)SortExec: TopK(fetch=2), expr=[id@0 ASC NULLS LAST], preserve_partitioning=[false]
-02)--ProjectionExec: expr=[id@1 as id, __datafusion_extracted_1@0 + 1 as simple_struct.s[value] + Int64(1)]
-03)----FilterExec: id@1 > 1
-04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 1 AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id, get_field(s@1, value) + 1 as simple_struct.s[value] + Int64(1)], file_type=parquet, predicate=id@0 > 1 AND DynamicFilter [ empty ], sort_order_for_reorder=[id@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
 
 # Verify correctness
 query II
@@ -765,9 +752,7 @@ physical_plan
 01)SortPreservingMergeExec: [id@0 ASC NULLS LAST]
 02)--ProjectionExec: expr=[id@1 as id, __datafusion_extracted_1@0 as multi_struct.s[value]]
 03)----SortExec: expr=[id@1 ASC NULLS LAST], preserve_partitioning=[true]
-04)------FilterExec: id@1 > 2
-05)--------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=3
-06)----------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/multi/part1.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/multi/part2.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/multi/part3.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/multi/part4.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/multi/part5.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+04)------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/multi/part1.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/multi/part2.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/multi/part3.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/multi/part4.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/multi/part5.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 2, sort_order_for_reorder=[id@1 ASC NULLS LAST], pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
 
 # Verify correctness
 query II
@@ -849,10 +834,7 @@ logical_plan
 02)--Filter: __datafusion_extracted_1 IS NOT NULL
 03)----Projection: get_field(nullable_struct.s, Utf8("value")) AS __datafusion_extracted_1, nullable_struct.id, get_field(nullable_struct.s, Utf8("label")) AS __datafusion_extracted_2
 04)------TableScan: nullable_struct projection=[id, s], partial_filters=[get_field(nullable_struct.s, Utf8("value")) IS NOT NULL]
-physical_plan
-01)ProjectionExec: expr=[id@0 as id, __datafusion_extracted_2@1 as nullable_struct.s[label]]
-02)--FilterExec: __datafusion_extracted_1@0 IS NOT NULL, projection=[id@1, __datafusion_extracted_2@2]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/nullable.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id, get_field(s@1, label) as __datafusion_extracted_2], file_type=parquet, predicate=get_field(s@1, value) IS NOT NULL
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/nullable.parquet]]}, projection=[id, get_field(s@1, label) as nullable_struct.s[label]], file_type=parquet, predicate=get_field(s@1, value) IS NOT NULL
 
 # Verify correctness
 query IT
@@ -975,9 +957,7 @@ logical_plan
 05)--------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
 physical_plan
 01)ProjectionExec: expr=[__common_expr_1@0 * __common_expr_1@0 as id_and_value]
-02)--ProjectionExec: expr=[id@1 + __datafusion_extracted_2@0 as __common_expr_1]
-03)----FilterExec: id@1 > 2
-04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_2, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id@0 + get_field(s@1, value) as __common_expr_1], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
 
 
 query TT
@@ -988,10 +968,7 @@ logical_plan
 02)--Filter: simple_struct.id > Int64(2)
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
-physical_plan
-01)ProjectionExec: expr=[__datafusion_extracted_1@0 + __datafusion_extracted_1@0 as doubled]
-02)--FilterExec: id@1 > 2, projection=[__datafusion_extracted_1@0]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) + get_field(s@1, value) as doubled], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
 
 # Verify correctness
 query I
@@ -1013,10 +990,7 @@ logical_plan
 02)--Filter: simple_struct.id > Int64(2)
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, get_field(simple_struct.s, Utf8("label")) AS __datafusion_extracted_2, simple_struct.id
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
-physical_plan
-01)ProjectionExec: expr=[__datafusion_extracted_1@0 as simple_struct.s[value], __datafusion_extracted_2@1 as simple_struct.s[label]]
-02)--FilterExec: id@2 > 2, projection=[__datafusion_extracted_1@0, __datafusion_extracted_2@1]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, get_field(s@1, label) as __datafusion_extracted_2, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as simple_struct.s[value], get_field(s@1, label) as simple_struct.s[label]], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
 
 # Verify correctness
 query IT
@@ -1063,10 +1037,7 @@ logical_plan
 02)--Filter: simple_struct.id > Int64(1)
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, get_field(simple_struct.s, Utf8("label")) AS __datafusion_extracted_2, simple_struct.id
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(1)]
-physical_plan
-01)ProjectionExec: expr=[__datafusion_extracted_1@0 * 2 + CAST(character_length(__datafusion_extracted_2@1) AS Int64) as score]
-02)--FilterExec: id@2 > 1, projection=[__datafusion_extracted_1@0, __datafusion_extracted_2@1]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, get_field(s@1, label) as __datafusion_extracted_2, id], file_type=parquet, predicate=id@0 > 1, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) * 2 + CAST(character_length(get_field(s@1, label)) AS Int64) as score], file_type=parquet, predicate=id@0 > 1, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
 
 # Verify correctness
 query I
@@ -1140,10 +1111,7 @@ logical_plan
 02)--Filter: simple_struct.id > Int64(1)
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(1)]
-physical_plan
-01)ProjectionExec: expr=[id@1 as id, __datafusion_extracted_1@0 as simple_struct.s[value]]
-02)--FilterExec: id@1 > 1
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 1, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id, get_field(s@1, value) as simple_struct.s[value]], file_type=parquet, predicate=id@0 > 1, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
 
 # Verify correctness
 query II
@@ -1160,10 +1128,7 @@ logical_plan
 02)--Filter: simple_struct.id > Int64(1) AND (simple_struct.id < Int64(4) OR simple_struct.id = Int64(5))
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(1), simple_struct.id < Int64(4) OR simple_struct.id = Int64(5)]
-physical_plan
-01)ProjectionExec: expr=[__datafusion_extracted_1@0 as simple_struct.s[value]]
-02)--FilterExec: id@1 > 1 AND (id@1 < 4 OR id@1 = 5), projection=[__datafusion_extracted_1@0]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 1 AND (id@0 < 4 OR id@0 = 5), pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1 AND (id_null_count@1 != row_count@2 AND id_min@3 < 4 OR id_null_count@1 != row_count@2 AND id_min@3 <= 5 AND 5 <= id_max@0), required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as simple_struct.s[value]], file_type=parquet, predicate=id@0 > 1 AND (id@0 < 4 OR id@0 = 5), pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1 AND (id_null_count@1 != row_count@2 AND id_min@3 < 4 OR id_null_count@1 != row_count@2 AND id_min@3 <= 5 AND 5 <= id_max@0), required_guarantees=[]
 
 # Verify correctness - should return rows where (id > 1) AND ((id < 4) OR (id = 5))
 # That's: id=2,3 (1<id<4) and id=5 (id=5 from second branch)
@@ -1182,10 +1147,7 @@ logical_plan
 02)--Filter: simple_struct.id > Int64(1) AND simple_struct.id < Int64(5)
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(1), simple_struct.id < Int64(5)]
-physical_plan
-01)ProjectionExec: expr=[__datafusion_extracted_1@0 as simple_struct.s[value]]
-02)--FilterExec: id@1 > 1 AND id@1 < 5, projection=[__datafusion_extracted_1@0]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 1 AND id@0 < 5, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1 AND id_null_count@1 != row_count@2 AND id_min@3 < 5, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as simple_struct.s[value]], file_type=parquet, predicate=id@0 > 1 AND id@0 < 5, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1 AND id_null_count@1 != row_count@2 AND id_min@3 < 5, required_guarantees=[]
 
 # Verify correctness - should return rows where 1 < id < 5 (id=2,3,4)
 query I
@@ -1203,10 +1165,7 @@ logical_plan
 02)--Filter: simple_struct.id > Int64(1)
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, get_field(simple_struct.s, Utf8("label")) AS __datafusion_extracted_2, simple_struct.id
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(1)]
-physical_plan
-01)ProjectionExec: expr=[__datafusion_extracted_1@0 as simple_struct.s[value], __datafusion_extracted_2@1 as simple_struct.s[label], id@2 as id]
-02)--FilterExec: id@2 > 1
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, get_field(s@1, label) as __datafusion_extracted_2, id], file_type=parquet, predicate=id@0 > 1, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as simple_struct.s[value], get_field(s@1, label) as simple_struct.s[label], id], file_type=parquet, predicate=id@0 > 1, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 1, required_guarantees=[]
 
 # Verify correctness - note that id is now at index 2 in the augmented projection
 query ITI
@@ -1224,10 +1183,7 @@ logical_plan
 02)--Filter: character_length(__datafusion_extracted_1) > Int32(4)
 03)----Projection: get_field(simple_struct.s, Utf8("label")) AS __datafusion_extracted_1, get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_2
 04)------TableScan: simple_struct projection=[s], partial_filters=[character_length(get_field(simple_struct.s, Utf8("label"))) > Int32(4)]
-physical_plan
-01)ProjectionExec: expr=[__datafusion_extracted_2@0 as simple_struct.s[value]]
-02)--FilterExec: character_length(__datafusion_extracted_1@0) > 4, projection=[__datafusion_extracted_2@1]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, label) as __datafusion_extracted_1, get_field(s@1, value) as __datafusion_extracted_2], file_type=parquet, predicate=character_length(get_field(s@1, label)) > 4
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as simple_struct.s[value]], file_type=parquet, predicate=character_length(get_field(s@1, label)) > 4
 
 # Verify correctness - filter on rows where label length > 4 (all have length 5, except 'one' has 3)
 # Wait, from the data: alpha(5), beta(4), gamma(5), delta(5), epsilon(7)
@@ -1459,9 +1415,8 @@ logical_plan
 06)--TableScan: join_right projection=[id]
 physical_plan
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(id@0, id@0)]
-02)--FilterExec: __datafusion_extracted_1@0 > 150, projection=[id@1]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=get_field(s@1, value) > 150
-04)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/join_right.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id], file_type=parquet, predicate=get_field(s@1, value) > 150
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/join_right.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
 
 # Verify correctness - id matches and value > 150
 query II
@@ -1498,10 +1453,8 @@ logical_plan
 09)--------TableScan: join_right projection=[id, s], partial_filters=[get_field(join_right.s, Utf8("level")) > Int64(3)]
 physical_plan
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(id@0, id@0)]
-02)--FilterExec: __datafusion_extracted_1@0 > 100, projection=[id@1]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=get_field(s@1, value) > 100
-04)--FilterExec: __datafusion_extracted_2@0 > 3, projection=[id@1]
-05)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/join_right.parquet]]}, projection=[get_field(s@1, level) as __datafusion_extracted_2, id], file_type=parquet, predicate=get_field(s@1, level) > 3 AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id], file_type=parquet, predicate=get_field(s@1, value) > 100
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/join_right.parquet]]}, projection=[id], file_type=parquet, predicate=get_field(s@1, level) > 3 AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible
 
 # Verify correctness - id matches, value > 100, and level > 3
 # Matching ids where value > 100: 2(200), 3(150), 4(300), 5(250)
@@ -1607,8 +1560,7 @@ physical_plan
 01)ProjectionExec: expr=[id@0 as id, __datafusion_extracted_2@1 as simple_struct.s[value], __datafusion_extracted_3@2 as join_right.s[level]]
 02)--HashJoinExec: mode=CollectLeft, join_type=Left, on=[(id@1, id@0)], projection=[id@1, __datafusion_extracted_2@0, __datafusion_extracted_3@3]
 03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_2, id], file_type=parquet
-04)----FilterExec: __datafusion_extracted_1@0 > 5, projection=[id@1, __datafusion_extracted_3@2]
-05)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/join_right.parquet]]}, projection=[get_field(s@1, level) as __datafusion_extracted_1, id, get_field(s@1, level) as __datafusion_extracted_3], file_type=parquet, predicate=get_field(s@1, level) > 5 AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/join_right.parquet]]}, projection=[id, get_field(s@1, level) as __datafusion_extracted_3], file_type=parquet, predicate=get_field(s@1, level) > 5 AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible
 
 # Verify correctness - left join with level > 5 condition
 # Only join_right rows with level > 5 are matched: id=1 (level=10), id=4 (level=8)
@@ -1640,11 +1592,7 @@ logical_plan
 02)--Filter: simple_struct.id > Int64(2)
 03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
 04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
-physical_plan
-01)ProjectionExec: expr=[__datafusion_extracted_1@0 as simple_struct.s[value]]
-02)--FilterExec: id@1 > 2, projection=[__datafusion_extracted_1@0]
-03)----RepartitionExec: partitioning=RoundRobinBatch(32), input_partitions=1
-04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as simple_struct.s[value]], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
 
 #####################
 # Section 14: SubqueryAlias tests
@@ -1665,10 +1613,7 @@ logical_plan
 04)------Filter: simple_struct.id > Int64(2)
 05)--------Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
 06)----------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
-physical_plan
-01)ProjectionExec: expr=[__datafusion_extracted_1@0 as t.s[value]]
-02)--FilterExec: id@1 > 2, projection=[__datafusion_extracted_1@0]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as t.s[value]], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
 
 # Verify correctness
 query I
@@ -1715,10 +1660,7 @@ logical_plan
 05)--------Filter: simple_struct.id > Int64(2)
 06)----------Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
 07)------------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
-physical_plan
-01)ProjectionExec: expr=[__datafusion_extracted_1@0 as u.s[value]]
-02)--FilterExec: id@1 > 2, projection=[__datafusion_extracted_1@0]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as u.s[value]], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
 
 # Verify correctness
 query I
@@ -1738,9 +1680,7 @@ logical_plan
 03)----Filter: __datafusion_extracted_1 > Int64(200)
 04)------Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
 05)--------TableScan: simple_struct projection=[id, s], partial_filters=[get_field(simple_struct.s, Utf8("value")) > Int64(200)]
-physical_plan
-01)FilterExec: __datafusion_extracted_1@0 > 200, projection=[id@1]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=get_field(s@1, value) > 200
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id], file_type=parquet, predicate=get_field(s@1, value) > 200
 
 # Verify correctness
 query I
@@ -1776,10 +1716,8 @@ logical_plan
 physical_plan
 01)ProjectionExec: expr=[__datafusion_extracted_1@0 as t.s[value]]
 02)--UnionExec
-03)----FilterExec: id@1 <= 3, projection=[__datafusion_extracted_1@0]
-04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 <= 3, pruning_predicate=id_null_count@1 != row_count@2 AND id_min@0 <= 3, required_guarantees=[]
-05)----FilterExec: id@1 > 3, projection=[__datafusion_extracted_1@0]
-06)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 3, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 3, required_guarantees=[]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1], file_type=parquet, predicate=id@0 <= 3, pruning_predicate=id_null_count@1 != row_count@2 AND id_min@0 <= 3, required_guarantees=[]
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1], file_type=parquet, predicate=id@0 > 3, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 3, required_guarantees=[]
 
 # Verify correctness
 query I
@@ -1821,11 +1759,9 @@ physical_plan
 02)--ProjectionExec: expr=[__datafusion_extracted_1@0 as t.s[value], __datafusion_extracted_2@1 as t.s[label]]
 03)----UnionExec
 04)------SortExec: expr=[__datafusion_extracted_1@0 ASC NULLS LAST], preserve_partitioning=[false]
-05)--------FilterExec: id@2 <= 3, projection=[__datafusion_extracted_1@0, __datafusion_extracted_2@1]
-06)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, get_field(s@1, label) as __datafusion_extracted_2, id], file_type=parquet, predicate=id@0 <= 3, pruning_predicate=id_null_count@1 != row_count@2 AND id_min@0 <= 3, required_guarantees=[]
-07)------SortExec: expr=[__datafusion_extracted_1@0 ASC NULLS LAST], preserve_partitioning=[false]
-08)--------FilterExec: id@2 > 3, projection=[__datafusion_extracted_1@0, __datafusion_extracted_2@1]
-09)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, get_field(s@1, label) as __datafusion_extracted_2, id], file_type=parquet, predicate=id@0 > 3, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 3, required_guarantees=[]
+05)--------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, get_field(s@1, label) as __datafusion_extracted_2], file_type=parquet, predicate=id@0 <= 3, pruning_predicate=id_null_count@1 != row_count@2 AND id_min@0 <= 3, required_guarantees=[]
+06)------SortExec: expr=[__datafusion_extracted_1@0 ASC NULLS LAST], preserve_partitioning=[false]
+07)--------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, get_field(s@1, label) as __datafusion_extracted_2], file_type=parquet, predicate=id@0 > 3, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 3, required_guarantees=[]
 
 # Verify correctness
 query IT
@@ -1985,9 +1921,7 @@ logical_plan
 02)--Filter: CASE WHEN __datafusion_extracted_3 IS NOT NULL THEN __datafusion_extracted_3 ELSE __datafusion_extracted_4 END = Int64(1)
 03)----Projection: get_field(t.s, Utf8("f1")) AS __datafusion_extracted_3, get_field(t.s, Utf8("f2")) AS __datafusion_extracted_4, get_field(t.s, Utf8("f1")) AS __datafusion_extracted_2
 04)------TableScan: t projection=[s], partial_filters=[CASE WHEN get_field(t.s, Utf8("f1")) IS NOT NULL THEN get_field(t.s, Utf8("f1")) ELSE get_field(t.s, Utf8("f2")) END = Int64(1)]
-physical_plan
-01)FilterExec: CASE WHEN __datafusion_extracted_3@0 IS NOT NULL THEN __datafusion_extracted_3@0 ELSE __datafusion_extracted_4@1 END = 1, projection=[__datafusion_extracted_2@2]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/test.parquet]]}, projection=[get_field(s@0, f1) as __datafusion_extracted_3, get_field(s@0, f2) as __datafusion_extracted_4, get_field(s@0, f1) as __datafusion_extracted_2], file_type=parquet, predicate=CASE WHEN get_field(s@0, f1) IS NOT NULL THEN get_field(s@0, f1) ELSE get_field(s@0, f2) END = 1
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/test.parquet]]}, projection=[get_field(s@0, f1) as __datafusion_extracted_2], file_type=parquet, predicate=CASE WHEN get_field(s@0, f1) IS NOT NULL THEN get_field(s@0, f1) ELSE get_field(s@0, f2) END = 1
 
 query I
 SELECT
diff --git a/datafusion/sqllogictest/test_files/repartition_scan.slt b/datafusion/sqllogictest/test_files/repartition_scan.slt
index fca1223cb0b2b..ce82229929166 100644
--- a/datafusion/sqllogictest/test_files/repartition_scan.slt
+++ b/datafusion/sqllogictest/test_files/repartition_scan.slt
@@ -62,9 +62,7 @@ EXPLAIN SELECT column1 FROM parquet_table WHERE column1 <> 42;
 logical_plan
 01)Filter: parquet_table.column1 != Int32(42)
 02)--TableScan: parquet_table projection=[column1], partial_filters=[parquet_table.column1 != Int32(42)]
-physical_plan
-01)FilterExec: column1@0 != 42
-02)--DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..131], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:131..262], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:262..393], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:393..521]]}, projection=[column1], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)]
+physical_plan DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..131], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:131..262], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:262..393], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:393..521]]}, projection=[column1], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)]
 
 # disable round robin repartitioning
 statement ok
@@ -77,9 +75,7 @@ EXPLAIN SELECT column1 FROM parquet_table WHERE column1 <> 42;
 logical_plan
 01)Filter: parquet_table.column1 != Int32(42)
 02)--TableScan: parquet_table projection=[column1], partial_filters=[parquet_table.column1 != Int32(42)]
-physical_plan
-01)FilterExec: column1@0 != 42
-02)--DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..131], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:131..262], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:262..393], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:393..521]]}, projection=[column1], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)]
+physical_plan DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..131], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:131..262], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:262..393], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:393..521]]}, projection=[column1], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)]
 
 # enable round robin repartitioning again
 statement ok
@@ -102,8 +98,7 @@ logical_plan
 physical_plan
 01)SortPreservingMergeExec: [column1@0 ASC NULLS LAST]
 02)--SortExec: expr=[column1@0 ASC NULLS LAST], preserve_partitioning=[true]
-03)----FilterExec: column1@0 != 42
-04)------DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..258], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:258..510, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..6], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:6..264], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:264..521]]}, projection=[column1], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)]
+03)----DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..258], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:258..510, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..6], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:6..264], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:264..521]]}, projection=[column1], file_type=parquet, predicate=column1@0 != 42, sort_order_for_reorder=[column1@0 ASC NULLS LAST], pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)]
 
 
 ## Read the files as though they are ordered
@@ -137,8 +132,7 @@ logical_plan
 03)----TableScan: parquet_table_with_order projection=[column1], partial_filters=[parquet_table_with_order.column1 != Int32(42)]
 physical_plan
 01)SortPreservingMergeExec: [column1@0 ASC NULLS LAST]
-02)--FilterExec: column1@0 != 42
-03)----DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..255], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..260], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:260..521], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:255..510]]}, projection=[column1], output_ordering=[column1@0 ASC NULLS LAST], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)]
+02)--DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:0..255], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..260], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:260..521], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/1.parquet:255..510]]}, projection=[column1], output_ordering=[column1@0 ASC NULLS LAST], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)]
 
 # Cleanup
 statement ok
diff --git a/datafusion/sqllogictest/test_files/repartition_subset_satisfaction.slt b/datafusion/sqllogictest/test_files/repartition_subset_satisfaction.slt
index 5371ca59beea1..6517fef40436b 100644
--- a/datafusion/sqllogictest/test_files/repartition_subset_satisfaction.slt
+++ b/datafusion/sqllogictest/test_files/repartition_subset_satisfaction.slt
@@ -378,9 +378,8 @@ physical_plan
 10)------------------AggregateExec: mode=Partial, gby=[f_dkey@0 as f_dkey, date_bin(IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 30000000000 }, timestamp@2) as date_bin(IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 30000000000 }"),j.timestamp), env@1 as env], aggr=[max(j.value)], ordering_mode=PartiallySorted([0, 1])
 11)--------------------HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(d_dkey@1, f_dkey@2)], projection=[f_dkey@4, env@0, timestamp@2, value@3]
 12)----------------------CoalescePartitionsExec
-13)------------------------FilterExec: service@1 = log, projection=[env@0, d_dkey@2]
-14)--------------------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=A/data.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=D/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=C/data.parquet]]}, projection=[env, service, d_dkey], output_partitioning=Hash([d_dkey@2], 3), file_type=parquet, predicate=service@1 = log, pruning_predicate=service_null_count@2 != row_count@3 AND service_min@0 <= log AND log <= service_max@1, required_guarantees=[service in (log)]
-15)----------------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=C/data.parquet]]}, projection=[timestamp, value, f_dkey], output_ordering=[f_dkey@2 ASC NULLS LAST, timestamp@0 ASC NULLS LAST], output_partitioning=Hash([f_dkey@2], 3), file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
+13)------------------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=A/data.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=D/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=C/data.parquet]]}, projection=[env, d_dkey], output_partitioning=Hash([d_dkey@1], 3), file_type=parquet, predicate=service@1 = log, pruning_predicate=service_null_count@2 != row_count@3 AND service_min@0 <= log AND log <= service_max@1, required_guarantees=[service in (log)]
+14)----------------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=C/data.parquet]]}, projection=[timestamp, value, f_dkey], output_ordering=[f_dkey@2 ASC NULLS LAST, timestamp@0 ASC NULLS LAST], output_partitioning=Hash([f_dkey@2], 3), file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
 
 # Verify results without subset satisfaction
 query TPR rowsort
@@ -473,9 +472,8 @@ physical_plan
 08)--------------AggregateExec: mode=SinglePartitioned, gby=[f_dkey@0 as f_dkey, date_bin(IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 30000000000 }, timestamp@2) as date_bin(IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 30000000000 }"),j.timestamp), env@1 as env], aggr=[max(j.value)], ordering_mode=PartiallySorted([0, 1])
 09)----------------HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(d_dkey@1, f_dkey@2)], projection=[f_dkey@4, env@0, timestamp@2, value@3]
 10)------------------CoalescePartitionsExec
-11)--------------------FilterExec: service@1 = log, projection=[env@0, d_dkey@2]
-12)----------------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=A/data.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=D/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=C/data.parquet]]}, projection=[env, service, d_dkey], output_partitioning=Hash([d_dkey@2], 3), file_type=parquet, predicate=service@1 = log, pruning_predicate=service_null_count@2 != row_count@3 AND service_min@0 <= log AND log <= service_max@1, required_guarantees=[service in (log)]
-13)------------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=C/data.parquet]]}, projection=[timestamp, value, f_dkey], output_ordering=[f_dkey@2 ASC NULLS LAST, timestamp@0 ASC NULLS LAST], output_partitioning=Hash([f_dkey@2], 3), file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
+11)--------------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=A/data.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=D/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/dimension/d_dkey=C/data.parquet]]}, projection=[env, d_dkey], output_partitioning=Hash([d_dkey@1], 3), file_type=parquet, predicate=service@1 = log, pruning_predicate=service_null_count@2 != row_count@3 AND service_min@0 <= log AND log <= service_max@1, required_guarantees=[service in (log)]
+12)------------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_subset_satisfaction/fact/f_dkey=C/data.parquet]]}, projection=[timestamp, value, f_dkey], output_ordering=[f_dkey@2 ASC NULLS LAST, timestamp@0 ASC NULLS LAST], output_partitioning=Hash([f_dkey@2], 3), file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
 
 # Verify results match with subset satisfaction
 query TPR rowsort

From 22ce8bfbef222c6dd69a0dcb678dae44826c8798 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Wed, 23 Sep 2026 20:19:25 -0500
Subject: [PATCH 02/42] Update tests added on main for always-accepted parquet
 filters

Tests and sqllogictest plans that landed on main after #22384 was
opened still expect the old "scan only uses the predicate for
pruning" behaviour. With the scan now applying every accepted filter:

- Two opener tests (`test_prune_all_null_column_equality_from_file_statistics`,
  `test_no_prune_when_missing_column_collapses_mixed_predicate`) now
  expect only the matching rows. The missing-column test also checks
  `post_scan_rows_pruned` so it still proves the file was read, not pruned.
- `string_in_list_pruning.rs` measured unpruned rows with the scan's
  `output_rows`. It now uses the post-scan matched + pruned counters,
  which count the rows that the scan decoded.
- Regenerated plans in `dynamic_filter_pushdown_config.slt`,
  `filter_without_sort_exec.slt`, `push_down_filter_parquet.slt`,
  `range_partitioning.slt` and `range_sorted_time_bin_agg.slt`: the
  `FilterExec` above parquet scans is gone and the new
  `post_scan_rows_*` metrics appear.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../tests/parquet/string_in_list_pruning.rs   | 28 +++++---
 .../datasource-parquet/src/opener/mod.rs      | 22 ++++--
 .../dynamic_filter_pushdown_config.slt        |  8 +--
 .../test_files/filter_without_sort_exec.slt   |  6 +-
 .../test_files/push_down_filter_parquet.slt   | 68 +++++++++----------
 .../test_files/range_partitioning.slt         | 47 +++++--------
 .../test_files/range_sorted_time_bin_agg.slt  |  3 +-
 7 files changed, 92 insertions(+), 90 deletions(-)

diff --git a/datafusion/core/tests/parquet/string_in_list_pruning.rs b/datafusion/core/tests/parquet/string_in_list_pruning.rs
index 1f9d6e31cbd92..1238238bb8a34 100644
--- a/datafusion/core/tests/parquet/string_in_list_pruning.rs
+++ b/datafusion/core/tests/parquet/string_in_list_pruning.rs
@@ -159,6 +159,16 @@ impl ScanOutput {
             .as_usize()
     }
 
+    /// Rows the scan decoded after row-group and page pruning.
+    ///
+    /// The scan runs with `pushdown_filters = false`, so it applies the whole
+    /// predicate in its post-scan filter: every decoded row is either matched
+    /// or pruned there. The scan's `output_rows` only counts the matched rows,
+    /// so it cannot show how much pruning skipped.
+    fn rows_decoded(&self) -> usize {
+        self.counter("post_scan_rows_matched") + self.counter("post_scan_rows_pruned")
+    }
+
     fn pruned(&self, name: &str) -> usize {
         let value = self
             .metrics
@@ -372,7 +382,7 @@ async fn check_string_in_list_pruning(page_pruning: bool) {
         assert!(!unpruned.plan.contains("IN_SET_INTERSECTS"));
         assert_eq!(unpruned.pruned("row_groups_pruned_statistics"), 0);
         assert_eq!(unpruned.pruned("page_index_rows_pruned"), 0);
-        assert_eq!(unpruned.counter("output_rows"), TOTAL_ROWS);
+        assert_eq!(unpruned.rows_decoded(), TOTAL_ROWS);
 
         let output = scan(&file, list_size, Some(list_size), page_pruning, false).await;
         output.assert_results();
@@ -400,7 +410,7 @@ async fn check_string_in_list_pruning(page_pruning: bool) {
             "list_size={list_size}, metrics={}",
             output.metrics
         );
-        assert_eq!(output.counter("output_rows"), MATCHING_ROWS);
+        assert_eq!(output.rows_decoded(), MATCHING_ROWS);
     }
 
     // The default remains 20: enabling the compact representation must not
@@ -410,7 +420,7 @@ async fn check_string_in_list_pruning(page_pruning: bool) {
     assert!(!default.plan.contains("IN_SET_INTERSECTS"));
     assert_eq!(default.pruned("row_groups_pruned_statistics"), 0);
     assert_eq!(default.pruned("page_index_rows_pruned"), 0);
-    assert_eq!(default.counter("output_rows"), TOTAL_ROWS);
+    assert_eq!(default.rows_decoded(), TOTAL_ROWS);
 }
 
 /// The compact `NOT IN` form must prune exactly the units whose single repeated
@@ -424,7 +434,7 @@ async fn check_string_not_in_list_pruning(page_pruning: bool) {
         assert!(!unpruned.plan.contains("NOT_IN_SET_MAY_MATCH"));
         assert_eq!(unpruned.pruned("row_groups_pruned_statistics"), 0);
         assert_eq!(unpruned.pruned("page_index_rows_pruned"), 0);
-        assert_eq!(unpruned.counter("output_rows"), TOTAL_ROWS);
+        assert_eq!(unpruned.rows_decoded(), TOTAL_ROWS);
 
         let output = scan(&file, list_size, Some(list_size), page_pruning, true).await;
         output.assert_negated_results();
@@ -454,7 +464,7 @@ async fn check_string_not_in_list_pruning(page_pruning: bool) {
             "list_size={list_size}, metrics={}",
             output.metrics
         );
-        assert_eq!(output.counter("output_rows"), MATCHING_ROWS);
+        assert_eq!(output.rows_decoded(), MATCHING_ROWS);
     }
 
     let default = scan(&file, 21, None, page_pruning, true).await;
@@ -462,7 +472,7 @@ async fn check_string_not_in_list_pruning(page_pruning: bool) {
     assert!(!default.plan.contains("NOT_IN_SET_MAY_MATCH"));
     assert_eq!(default.pruned("row_groups_pruned_statistics"), 0);
     assert_eq!(default.pruned("page_index_rows_pruned"), 0);
-    assert_eq!(default.counter("output_rows"), TOTAL_ROWS);
+    assert_eq!(default.rows_decoded(), TOTAL_ROWS);
 
     // A mixed interval that overlaps a list member can still contain matching
     // rows. Only the adjacent single-valued unit can be excluded.
@@ -481,7 +491,7 @@ async fn check_string_not_in_list_pruning(page_pruning: bool) {
     unpruned.assert_no_filter_interference();
     assert_eq!(unpruned.pruned("row_groups_pruned_statistics"), 0);
     assert_eq!(unpruned.pruned("page_index_rows_pruned"), 0);
-    assert_eq!(unpruned.counter("output_rows"), ROWS_PER_UNIT * 2);
+    assert_eq!(unpruned.rows_decoded(), ROWS_PER_UNIT * 2);
 
     let output = scan(&mixed_file, 21, Some(21), page_pruning, true).await;
     assert_eq!(
@@ -500,7 +510,7 @@ async fn check_string_not_in_list_pruning(page_pruning: bool) {
         output.pruned("page_index_rows_pruned"),
         if page_pruning { ROWS_PER_UNIT } else { 0 }
     );
-    assert_eq!(output.counter("output_rows"), ROWS_PER_UNIT);
+    assert_eq!(output.rows_decoded(), ROWS_PER_UNIT);
 }
 
 async fn check_string_not_in_list_with_truncated_bounds(page_pruning: bool) {
@@ -515,7 +525,7 @@ async fn check_string_not_in_list_with_truncated_bounds(page_pruning: bool) {
     assert!(output.plan.contains("NOT_IN_SET_MAY_MATCH"));
     assert_eq!(output.pruned("row_groups_pruned_statistics"), 0);
     assert_eq!(output.pruned("page_index_rows_pruned"), 0);
-    assert_eq!(output.counter("output_rows"), ROWS_PER_UNIT);
+    assert_eq!(output.rows_decoded(), ROWS_PER_UNIT);
 }
 
 #[tokio::test]
diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index 4b97fcd66e657..9f28d1ef94e4a 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -3464,7 +3464,9 @@ mod test {
         );
 
         // A predicate the statistics cannot disprove still reads the file,
-        // so the skip above is not simply "prune everything".
+        // so the skip above is not simply "prune everything". The scan
+        // accepts the filter and applies it post-scan, so only the matching
+        // row survives.
         let metrics = ExecutionPlanMetricsSet::new();
         let expr = col("a").eq(lit(2));
         let predicate = logical2physical(&expr, &table_schema);
@@ -3472,7 +3474,7 @@ mod test {
         let stream = open_file(&opener, file).await.unwrap();
         let (num_batches, num_rows) = count_batches_and_rows(stream).await;
         assert_eq!(num_batches, 1);
-        assert_eq!(num_rows, 3);
+        assert_eq!(num_rows, 1);
         assert_eq!(pruned_row_groups_statistics(&metrics), 0);
     }
 
@@ -3590,11 +3592,19 @@ mod test {
             .build();
         let stream = open_file(&opener, file).await.unwrap();
         let (_, num_rows) = count_batches_and_rows(stream).await;
-        // The row group is read and its rows handed up for the filter above
-        // to apply, which is the missing-column path this test protects.
+        // The row group is read (not pruned), which is the missing-column
+        // path this test protects. The scan then applies the predicate
+        // post-scan: `b` is NULL in every row of this file, so `b = 2` rejects
+        // them all.
         assert_eq!(
-            num_rows, 3,
-            "the file must be scanned, not pruned, on the missing-column path"
+            num_rows, 0,
+            "the file must be scanned and filtered, not pruned, on the \
+             missing-column path"
+        );
+        assert_eq!(
+            counter_metric_value(&metrics, "post_scan_rows_pruned"),
+            3,
+            "every row must be read and then rejected by the post-scan filter"
         );
         assert_eq!(
             pruned_row_groups_statistics(&metrics),
diff --git a/datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt b/datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt
index 6a6bad99f0840..cf835b5dbd5de 100644
--- a/datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt
+++ b/datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt
@@ -102,9 +102,8 @@ Plan with Metrics
 01)SortPreservingMergeExec: [v@1 DESC], fetch=3, metrics=[output_rows=3, <slt:ignore>]
 02)--SortExec: TopK(fetch=3), expr=[v@1 DESC], preserve_partitioning=[true], filter=[v@1 IS NULL OR v@1 > 800], metrics=[output_rows=3, <slt:ignore>]
 03)----ProjectionExec: expr=[id@0 as id, value@1 as v, value@1 + id@0 as name], metrics=[output_rows=10, <slt:ignore>]
-04)------FilterExec: value@1 > 3, metrics=[output_rows=10, <slt:ignore>, selectivity=100% (10/10)]
-05)--------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, metrics=[output_rows=10, <slt:ignore>]
-06)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/test_data.parquet]]}, projection=[id, value], file_type=parquet, predicate=value@1 > 3 AND DynamicFilter [ value@1 IS NULL OR value@1 > 800 ], dynamic_rg_pruning=eligible, pruning_predicate=value_null_count@1 != row_count@2 AND value_max@0 > 3 AND (value_null_count@1 > 0 OR value_null_count@1 != row_count@2 AND value_max@0 > 800), required_guarantees=[], metrics=[output_rows=10, elapsed_compute=<slt:ignore>, output_bytes=80.0 B, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched -> 1 fully matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_pages_skipped_by_fully_matched=1, limit_pruned_row_groups=0 total → 0 matched, bytes_processed=1147.0 B, bytes_scanned=210.0 B, page_index_load_skipped=1, row_groups_pruned_dynamic_filter=0, metadata_load_time=<slt:ignore>, scan_efficiency_ratio=18.31% (210/1.15 K)]
+04)------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, metrics=[output_rows=10, <slt:ignore>]
+05)--------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/test_data.parquet]]}, projection=[id, value], file_type=parquet, predicate=value@1 > 3 AND DynamicFilter [ value@1 IS NULL OR value@1 > 800 ], sort_order_for_reorder=[value@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=value_null_count@1 != row_count@2 AND value_max@0 > 3 AND (value_null_count@1 > 0 OR value_null_count@1 != row_count@2 AND value_max@0 > 800), required_guarantees=[], metrics=[output_rows=10, elapsed_compute=<slt:ignore>, output_bytes=64.0 KB, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched -> 1 fully matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_pages_skipped_by_fully_matched=1, limit_pruned_row_groups=0 total → 0 matched, bytes_processed=1147.0 B, bytes_scanned=210.0 B, page_index_load_skipped=1, row_groups_pruned_dynamic_filter=0, metadata_load_time=<slt:ignore>, scan_efficiency_ratio=18.31% (210/1.15 K)]
 
 statement ok
 set datafusion.explain.analyze_level = dev;
@@ -337,8 +336,7 @@ physical_plan
 01)HashJoinExec: mode=CollectLeft, join_type=RightSemi, on=[(id@0, id@0)]
 02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_right.parquet]]}, projection=[id], file_type=parquet
 03)--SortExec: expr=[data@1 DESC], preserve_partitioning=[false]
-04)----FilterExec: DynamicFilter [ empty ]
-05)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_left.parquet]]}, projection=[id, data], file_type=parquet, predicate=DynamicFilter [ empty ], sort_order_for_reorder=[data@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_left.parquet]]}, projection=[id, data], file_type=parquet, predicate=DynamicFilter [ empty ], sort_order_for_reorder=[data@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible
 
 statement count 0
 SET datafusion.execution.parquet.pushdown_filters = true;
diff --git a/datafusion/sqllogictest/test_files/filter_without_sort_exec.slt b/datafusion/sqllogictest/test_files/filter_without_sort_exec.slt
index df23390c64dfd..f28d2b017f38d 100644
--- a/datafusion/sqllogictest/test_files/filter_without_sort_exec.slt
+++ b/datafusion/sqllogictest/test_files/filter_without_sort_exec.slt
@@ -184,11 +184,7 @@ logical_plan
 01)Sort: cast_ordered.b ASC NULLS LAST
 02)--Filter: cast_ordered.b > Int64(1)
 03)----TableScan: cast_ordered projection=[b], partial_filters=[cast_ordered.b > Int64(1)]
-physical_plan
-01)SortPreservingMergeExec: [b@0 ASC NULLS LAST]
-02)--FilterExec: b@0 > 1
-03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, maintains_sort_order=true
-04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/filter_without_sort_exec/cast_ordering.parquet]]}, projection=[b], output_ordering=[b@0 ASC NULLS LAST], file_type=parquet, predicate=b@0 > 1, pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 > 1, required_guarantees=[]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/filter_without_sort_exec/cast_ordering.parquet]]}, projection=[b], output_ordering=[b@0 ASC NULLS LAST], file_type=parquet, predicate=b@0 > 1, pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 > 1, required_guarantees=[]
 
 statement ok
 SET datafusion.execution.target_partitions = 1;
diff --git a/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt b/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
index 8dd763205f6c2..79c80cce37d2a 100644
--- a/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
+++ b/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
@@ -206,7 +206,7 @@ EXPLAIN ANALYZE SELECT t FROM topk_pushdown ORDER BY t * t LIMIT 10;
 ----
 Plan with Metrics
 01)SortExec: TopK(fetch=10), expr=[t@0 * t@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[t@0 * t@0 < 1884329474306198481], metrics=[output_rows=10, output_batches=1, row_replacements=10]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_pushdown.parquet]]}, projection=[t], output_ordering=[t@0 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ t@0 * t@0 < 1884329474306198481 ], dynamic_rg_pruning=eligible, metrics=[output_rows=128, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=782 total → 782 matched, row_groups_pruned_bloom_filter=782 total → 782 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=128, pushdown_rows_pruned=99.87 K, predicate_cache_inner_records=128, predicate_cache_records=128, scan_efficiency_ratio=64.87% (258.7 K/398.8 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_pushdown.parquet]]}, projection=[t], output_ordering=[t@0 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ t@0 * t@0 < 1884329474306198481 ], dynamic_rg_pruning=eligible, metrics=[output_rows=128, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=782 total → 782 matched, row_groups_pruned_bloom_filter=782 total → 782 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=128, pushdown_rows_pruned=99.87 K, predicate_cache_inner_records=128, predicate_cache_records=128, scan_efficiency_ratio=64.87% (258.7 K/398.8 K)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -268,7 +268,7 @@ EXPLAIN ANALYZE SELECT * FROM topk_single_col ORDER BY b DESC LIMIT 1;
 ----
 Plan with Metrics
 01)SortExec: TopK(fetch=1), expr=[b@1 DESC], preserve_partitioning=[false], filter=[b@1 IS NULL OR b@1 > bd], metrics=[output_rows=1, output_batches=1, row_replacements=1]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_single_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 IS NULL OR b@1 > bd ], sort_order_for_reorder=[b@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@0 > 0 OR b_null_count@0 != row_count@2 AND b_max@1 > bd, required_guarantees=[], metrics=[output_rows=4, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=0, predicate_cache_inner_records=4, predicate_cache_records=4, scan_efficiency_ratio=21.94% (222/1.01 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_single_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 IS NULL OR b@1 > bd ], sort_order_for_reorder=[b@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@0 > 0 OR b_null_count@0 != row_count@2 AND b_max@1 > bd, required_guarantees=[], metrics=[output_rows=4, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=0, predicate_cache_inner_records=4, predicate_cache_records=4, scan_efficiency_ratio=21.94% (222/1.01 K)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -319,7 +319,7 @@ EXPLAIN ANALYZE SELECT * FROM topk_multi_col ORDER BY b ASC NULLS LAST, a DESC L
 ----
 Plan with Metrics
 01)SortExec: TopK(fetch=2), expr=[b@1 ASC NULLS LAST, a@0 DESC], preserve_partitioning=[false], filter=[b@1 < bb OR b@1 = bb AND (a@0 IS NULL OR a@0 > ac)], metrics=[output_rows=2, output_batches=1, row_replacements=2]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_multi_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 < bb OR b@1 = bb AND (a@0 IS NULL OR a@0 > ac) ], sort_order_for_reorder=[b@1 ASC NULLS LAST, a@0 DESC], dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@1 != row_count@2 AND b_min@0 < bb OR b_null_count@1 != row_count@2 AND b_min@0 <= bb AND bb <= b_max@3 AND (a_null_count@4 > 0 OR a_null_count@4 != row_count@2 AND a_max@5 > ac), required_guarantees=[], metrics=[output_rows=4, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=0, predicate_cache_inner_records=8, predicate_cache_records=8, scan_efficiency_ratio=21.94% (222/1.01 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_multi_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 < bb OR b@1 = bb AND (a@0 IS NULL OR a@0 > ac) ], sort_order_for_reorder=[b@1 ASC NULLS LAST, a@0 DESC], dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@1 != row_count@2 AND b_min@0 < bb OR b_null_count@1 != row_count@2 AND b_min@0 <= bb AND bb <= b_max@3 AND (a_null_count@4 > 0 OR a_null_count@4 != row_count@2 AND a_max@5 > ac), required_guarantees=[], metrics=[output_rows=4, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=0, predicate_cache_inner_records=8, predicate_cache_records=8, scan_efficiency_ratio=21.94% (222/1.01 K)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -388,8 +388,8 @@ FROM join_probe p INNER JOIN join_build AS build
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, a@0), (b@1, b@1)], projection=[a@3, b@4, c@2, e@5], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=19.88% (196/986)]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= ba AND b_null_count@5 != row_count@2 AND b_min@6 <= bb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=22.37% (228/1.02 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=19.88% (196/986)]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= ba AND b_null_count@5 != row_count@2 AND b_min@6 <= bb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=22.37% (228/1.02 K)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -474,9 +474,9 @@ INNER JOIN nested_t3 ON nested_t2.c = nested_t3.d;
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(c@3, d@0)], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
 02)--HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, b@0)], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t1.parquet]]}, projection=[a, x], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=17.72% (132/745)]
-04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t2.parquet]]}, projection=[b, c, y], file_type=parquet, predicate=DynamicFilter [ b@0 >= aa AND b@0 <= ab AND b@0 IN (SET) ([aa, ab]) ], dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 >= aa AND b_null_count@1 != row_count@2 AND b_min@3 <= ab AND (b_null_count@1 != row_count@2 AND b_min@3 <= aa AND aa <= b_max@0 OR b_null_count@1 != row_count@2 AND b_min@3 <= ab AND ab <= b_max@0), required_guarantees=[b in (aa, ab)], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=5 total → 5 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=3, predicate_cache_inner_records=5, predicate_cache_records=2, scan_efficiency_ratio=22.78% (234/1.03 K)]
-05)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t3.parquet]]}, projection=[d, z], file_type=parquet, predicate=DynamicFilter [ d@0 >= ca AND d@0 <= cb AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=d_null_count@1 != row_count@2 AND d_max@0 >= ca AND d_null_count@1 != row_count@2 AND d_min@3 <= cb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=8 total → 8 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=6, predicate_cache_inner_records=8, predicate_cache_records=2, scan_efficiency_ratio=21.86% (172/787)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t1.parquet]]}, projection=[a, x], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=17.72% (132/745)]
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t2.parquet]]}, projection=[b, c, y], file_type=parquet, predicate=DynamicFilter [ b@0 >= aa AND b@0 <= ab AND b@0 IN (SET) ([aa, ab]) ], dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 >= aa AND b_null_count@1 != row_count@2 AND b_min@3 <= ab AND (b_null_count@1 != row_count@2 AND b_min@3 <= aa AND aa <= b_max@0 OR b_null_count@1 != row_count@2 AND b_min@3 <= ab AND ab <= b_max@0), required_guarantees=[b in (aa, ab)], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=5 total → 5 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=3, predicate_cache_inner_records=5, predicate_cache_records=2, scan_efficiency_ratio=22.78% (234/1.03 K)]
+05)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t3.parquet]]}, projection=[d, z], file_type=parquet, predicate=DynamicFilter [ d@0 >= ca AND d@0 <= cb AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=d_null_count@1 != row_count@2 AND d_max@0 >= ca AND d_null_count@1 != row_count@2 AND d_min@3 <= cb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=8 total → 8 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=6, predicate_cache_inner_records=8, predicate_cache_records=2, scan_efficiency_ratio=21.86% (172/787)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -605,8 +605,8 @@ LIMIT 2;
 Plan with Metrics
 01)SortExec: TopK(fetch=2), expr=[e@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[e@0 < bb], metrics=[output_rows=2, output_batches=1, row_replacements=2]
 02)--HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, d@0)], projection=[e@2], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_join_build.parquet]]}, projection=[a], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=6.49% (64/986)]
-04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_join_probe.parquet]]}, projection=[d, e], file_type=parquet, predicate=DynamicFilter [ d@0 >= aa AND d@0 <= ab AND d@0 IN (SET) ([aa, ab]) ] AND DynamicFilter [ e@1 < bb ], dynamic_rg_pruning=eligible, pruning_predicate=d_null_count@1 != row_count@2 AND d_max@0 >= aa AND d_null_count@1 != row_count@2 AND d_min@3 <= ab AND (d_null_count@1 != row_count@2 AND d_min@3 <= aa AND aa <= d_max@0 OR d_null_count@1 != row_count@2 AND d_min@3 <= ab AND ab <= d_max@0) AND e_null_count@5 != row_count@2 AND e_min@4 < bb, required_guarantees=[d in (aa, ab)], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=4 total → 4 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=15.11% (154/1.02 K)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_join_build.parquet]]}, projection=[a], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=6.49% (64/986)]
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_join_probe.parquet]]}, projection=[d, e], file_type=parquet, predicate=DynamicFilter [ d@0 >= aa AND d@0 <= ab AND d@0 IN (SET) ([aa, ab]) ] AND DynamicFilter [ e@1 < bb ], dynamic_rg_pruning=eligible, pruning_predicate=d_null_count@1 != row_count@2 AND d_max@0 >= aa AND d_null_count@1 != row_count@2 AND d_min@3 <= ab AND (d_null_count@1 != row_count@2 AND d_min@3 <= aa AND aa <= d_max@0 OR d_null_count@1 != row_count@2 AND d_min@3 <= ab AND ab <= d_max@0) AND e_null_count@5 != row_count@2 AND e_min@4 < bb, required_guarantees=[d in (aa, ab)], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=4 total → 4 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=15.11% (154/1.02 K)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -656,7 +656,7 @@ EXPLAIN ANALYZE SELECT b, a FROM topk_proj ORDER BY a LIMIT 2;
 Plan with Metrics
 01)ProjectionExec: expr=[b@1 as b, a@0 as a], metrics=[output_rows=2, output_batches=1]
 02)--SortExec: TopK(fetch=2), expr=[a@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[a@0 < 2], metrics=[output_rows=2, output_batches=1, row_replacements=2]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 < 2 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_min@0 < 2, required_guarantees=[], metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=3, pushdown_rows_pruned=0, predicate_cache_inner_records=3, predicate_cache_records=3, scan_efficiency_ratio=13.4% (141/1.05 K)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 < 2 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_min@0 < 2, required_guarantees=[], metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=3, pushdown_rows_pruned=0, predicate_cache_inner_records=3, predicate_cache_records=3, scan_efficiency_ratio=13.4% (141/1.05 K)]
 
 # Case 2: prune — `SELECT a` — filter stays as `a < 2` on the scan.
 query TT
@@ -664,7 +664,7 @@ EXPLAIN ANALYZE SELECT a FROM topk_proj ORDER BY a LIMIT 2;
 ----
 Plan with Metrics
 01)SortExec: TopK(fetch=2), expr=[a@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[a@0 < 2], metrics=[output_rows=2, output_batches=1, row_replacements=2]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[a], file_type=parquet, predicate=DynamicFilter [ a@0 < 2 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_min@0 < 2, required_guarantees=[], metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=3, pushdown_rows_pruned=0, predicate_cache_inner_records=3, predicate_cache_records=3, scan_efficiency_ratio=6.94% (73/1.05 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[a], file_type=parquet, predicate=DynamicFilter [ a@0 < 2 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_min@0 < 2, required_guarantees=[], metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=3, pushdown_rows_pruned=0, predicate_cache_inner_records=3, predicate_cache_records=3, scan_efficiency_ratio=6.94% (73/1.05 K)]
 
 # Case 3: expression — `SELECT a+1 AS a_plus_1` — the TopK filter is on
 # `a_plus_1`, the scan predicate must read `a@0 + 1`.
@@ -673,7 +673,7 @@ EXPLAIN ANALYZE SELECT a + 1 AS a_plus_1, b FROM topk_proj ORDER BY a_plus_1 LIM
 ----
 Plan with Metrics
 01)SortExec: TopK(fetch=2), expr=[a_plus_1@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[a_plus_1@0 < 3], metrics=[output_rows=2, output_batches=1, row_replacements=2]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[CAST(a@0 AS Int64) + 1 as a_plus_1, b], file_type=parquet, predicate=DynamicFilter [ CAST(a@0 AS Int64) + 1 < 3 ], dynamic_rg_pruning=eligible, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=3, pushdown_rows_pruned=0, predicate_cache_inner_records=3, predicate_cache_records=3, scan_efficiency_ratio=13.4% (141/1.05 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[CAST(a@0 AS Int64) + 1 as a_plus_1, b], file_type=parquet, predicate=DynamicFilter [ CAST(a@0 AS Int64) + 1 < 3 ], dynamic_rg_pruning=eligible, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=3, pushdown_rows_pruned=0, predicate_cache_inner_records=3, predicate_cache_records=3, scan_efficiency_ratio=13.4% (141/1.05 K)]
 
 # Case 4: alias shadowing — `SELECT a+1 AS a` — the projection renames
 # `a+1` to `a`, so the TopK's `a < 3` must still be rewritten to
@@ -683,7 +683,7 @@ EXPLAIN ANALYZE SELECT a + 1 AS a, b FROM topk_proj ORDER BY a LIMIT 2;
 ----
 Plan with Metrics
 01)SortExec: TopK(fetch=2), expr=[a@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[a@0 < 3], metrics=[output_rows=2, output_batches=1, row_replacements=2]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[CAST(a@0 AS Int64) + 1 as a, b], file_type=parquet, predicate=DynamicFilter [ CAST(a@0 AS Int64) + 1 < 3 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=3, pushdown_rows_pruned=0, predicate_cache_inner_records=3, predicate_cache_records=3, scan_efficiency_ratio=13.4% (141/1.05 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[CAST(a@0 AS Int64) + 1 as a, b], file_type=parquet, predicate=DynamicFilter [ CAST(a@0 AS Int64) + 1 < 3 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=3, pushdown_rows_pruned=0, predicate_cache_inner_records=3, predicate_cache_records=3, scan_efficiency_ratio=13.4% (141/1.05 K)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -740,12 +740,12 @@ INNER JOIN (
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, a@0)], projection=[a@0, min_value@2], metrics=[output_rows=2, output_batches=2, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=2, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_agg_build.parquet]]}, projection=[a], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=14.45% (64/443)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_agg_build.parquet]]}, projection=[a], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=14.45% (64/443)]
 03)--ProjectionExec: expr=[a@0 as a, min(join_agg_probe.value)@1 as min_value], metrics=[output_rows=2, output_batches=2]
 04)----AggregateExec: mode=FinalPartitioned, gby=[a@0 as a], aggr=[min(join_agg_probe.value)], metrics=[output_rows=2, output_batches=2, spill_count=0, spilled_rows=0]
 05)------RepartitionExec: partitioning=Hash([a@0], 4), input_partitions=1, metrics=[output_rows=2, output_batches=2, spill_count=0, spilled_rows=0]
 06)--------AggregateExec: mode=Partial, gby=[a@0 as a], aggr=[min(join_agg_probe.value)], metrics=[output_rows=2, output_batches=1, spill_count=0, spilled_rows=0, skipped_aggregation_rows=0, reduction_factor=100% (2/2)]
-07)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_agg_probe.parquet]]}, projection=[a, value], file_type=parquet, predicate=DynamicFilter [ a@0 >= h1 AND a@0 <= h2 AND a@0 IN (SET) ([h1, h2]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= h1 AND a_null_count@1 != row_count@2 AND a_min@3 <= h2 AND (a_null_count@1 != row_count@2 AND a_min@3 <= h1 AND h1 <= a_max@0 OR a_null_count@1 != row_count@2 AND a_min@3 <= h2 AND h2 <= a_max@0), required_guarantees=[a in (h1, h2)], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=4 total → 4 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=4, predicate_cache_records=2, scan_efficiency_ratio=19.43% (151/777)]
+07)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_agg_probe.parquet]]}, projection=[a, value], file_type=parquet, predicate=DynamicFilter [ a@0 >= h1 AND a@0 <= h2 AND a@0 IN (SET) ([h1, h2]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= h1 AND a_null_count@1 != row_count@2 AND a_min@3 <= h2 AND (a_null_count@1 != row_count@2 AND a_min@3 <= h1 AND h1 <= a_max@0 OR a_null_count@1 != row_count@2 AND a_min@3 <= h2 AND h2 <= a_max@0), required_guarantees=[a in (h1, h2)], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=4 total → 4 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=4, predicate_cache_records=2, scan_efficiency_ratio=19.43% (151/777)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -807,8 +807,8 @@ ON nulls_build.a = nulls_probe.a AND nulls_build.b = nulls_probe.b;
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, a@0), (b@1, b@1)], metrics=[output_rows=1, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=3, input_batches=1, input_rows=1, avg_fanout=100% (1/1), probe_hit_rate=100% (1/1)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nulls_build.parquet]]}, projection=[a, b], file_type=parquet, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=18.6% (144/774)]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nulls_probe.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= 1 AND b@1 <= 2 AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:1}, {c0:,c1:2}, {c0:ab,c1:}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= 1 AND b_null_count@5 != row_count@2 AND b_min@6 <= 2, required_guarantees=[], metrics=[output_rows=1, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=1, pushdown_rows_pruned=3, predicate_cache_inner_records=8, predicate_cache_records=2, scan_efficiency_ratio=20.45% (225/1.10 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nulls_build.parquet]]}, projection=[a, b], file_type=parquet, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=18.6% (144/774)]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nulls_probe.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= 1 AND b@1 <= 2 AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:1}, {c0:,c1:2}, {c0:ab,c1:}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= 1 AND b_null_count@5 != row_count@2 AND b_min@6 <= 2, required_guarantees=[], metrics=[output_rows=1, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=1, pushdown_rows_pruned=3, predicate_cache_inner_records=8, predicate_cache_records=2, scan_efficiency_ratio=20.45% (225/1.10 K)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -873,8 +873,8 @@ ON lj_build.a = lj_probe.a AND lj_build.b = lj_probe.b;
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Left, on=[(a@0, a@0), (b@1, b@1)], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=2, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=19.88% (196/986)]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= ba AND b_null_count@5 != row_count@2 AND b_min@6 <= bb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=22.37% (228/1.02 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=19.88% (196/986)]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= ba AND b_null_count@5 != row_count@2 AND b_min@6 <= bb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=22.37% (228/1.02 K)]
 
 # LEFT SEMI JOIN: only matching build rows are returned; probe scan still
 # receives the dynamic filter.
@@ -889,8 +889,8 @@ WHERE EXISTS (
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=LeftSemi, on=[(a@0, a@0), (b@1, b@1)], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=2, input_rows=4, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=19.88% (196/986)]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_probe.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= ba AND b_null_count@5 != row_count@2 AND b_min@6 <= bb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=15.11% (154/1.02 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=19.88% (196/986)]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_probe.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= ba AND b_null_count@5 != row_count@2 AND b_min@6 <= bb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=15.11% (154/1.02 K)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -959,8 +959,8 @@ FROM hl_probe p INNER JOIN hl_build AS build
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, a@0), (b@1, b@1)], projection=[a@3, b@4, c@2, e@5], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/hl_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=19.88% (196/986)]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/hl_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= ba AND b_null_count@5 != row_count@2 AND b_min@6 <= bb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=22.37% (228/1.02 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/hl_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=19.88% (196/986)]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/hl_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= ba AND b_null_count@5 != row_count@2 AND b_min@6 <= bb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=22.37% (228/1.02 K)]
 
 statement ok
 drop table hl_build;
@@ -1008,8 +1008,8 @@ FROM int_build b INNER JOIN int_probe p
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(id1@0, id1@0), (id2@1, id2@1)], projection=[id1@0, id2@1, value@2, data@5], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/int_build.parquet]]}, projection=[id1, id2, value], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=18.48% (204/1.10 K)]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/int_probe.parquet]]}, projection=[id1, id2, data], file_type=parquet, predicate=DynamicFilter [ id1@0 >= 1 AND id1@0 <= 2 AND id2@1 >= 10 AND id2@1 <= 20 AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=id1_null_count@1 != row_count@2 AND id1_max@0 >= 1 AND id1_null_count@1 != row_count@2 AND id1_min@3 <= 2 AND id2_null_count@5 != row_count@2 AND id2_max@4 >= 10 AND id2_null_count@5 != row_count@2 AND id2_min@6 <= 20, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=20.67% (221/1.07 K)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/int_build.parquet]]}, projection=[id1, id2, value], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=18.48% (204/1.10 K)]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/int_probe.parquet]]}, projection=[id1, id2, data], file_type=parquet, predicate=DynamicFilter [ id1@0 >= 1 AND id1@0 <= 2 AND id2@1 >= 10 AND id2@1 <= 20 AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=id1_null_count@1 != row_count@2 AND id1_max@0 >= 1 AND id1_null_count@1 != row_count@2 AND id1_min@3 <= 2 AND id2_null_count@5 != row_count@2 AND id2_max@4 >= 10 AND id2_null_count@5 != row_count@2 AND id2_min@6 <= 20, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=2, predicate_cache_inner_records=8, predicate_cache_records=4, scan_efficiency_ratio=20.67% (221/1.07 K)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -1060,8 +1060,8 @@ EXPLAIN ANALYZE SELECT nej_build.id, nej_probe.id FROM nej_build JOIN nej_probe
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(id@0, id@0)], NullsEqual: true, metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nej_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=12.92% (65/503)]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nej_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 IS NULL OR id@0 >= 11 AND id@0 <= 11 AND id@0 IN (SET) ([11, NULL]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@0 > 0 OR id_null_count@0 != row_count@2 AND id_max@1 >= 11 AND id_null_count@0 != row_count@2 AND id_min@3 <= 11 AND (id_null_count@0 != row_count@2 AND id_min@3 <= 11 AND 11 <= id_max@1 OR id_null_count@0 != row_count@2 AND id_min@3 <= NULL AND NULL <= id_max@1), required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=3 total → 3 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=1, predicate_cache_inner_records=3, predicate_cache_records=3, scan_efficiency_ratio=14.45% (74/512)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nej_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=12.92% (65/503)]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nej_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 IS NULL OR id@0 >= 11 AND id@0 <= 11 AND id@0 IN (SET) ([11, NULL]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@0 > 0 OR id_null_count@0 != row_count@2 AND id_max@1 >= 11 AND id_null_count@0 != row_count@2 AND id_min@3 <= 11 AND (id_null_count@0 != row_count@2 AND id_min@3 <= 11 AND 11 <= id_max@1 OR id_null_count@0 != row_count@2 AND id_min@3 <= NULL AND NULL <= id_max@1), required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=3 total → 3 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=2, pushdown_rows_pruned=1, predicate_cache_inner_records=3, predicate_cache_records=3, scan_efficiency_ratio=14.45% (74/512)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -1103,8 +1103,8 @@ EXPLAIN ANALYZE SELECT mnej_build.a, mnej_build.b, mnej_probe.a, mnej_probe.b FR
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, a@0), (b@1, b@1)], NullsEqual: true, metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=3, avg_fanout=100% (2/2), probe_hit_rate=66.67% (2/3)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/mnej_build.parquet]]}, projection=[a, b], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=16.42% (133/810)]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/mnej_probe.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 IS NULL OR b@1 IS NULL OR a@0 >= 1 AND a@0 <= 2 AND b@1 >= 10 AND b@1 <= 10 AND struct(a@0, b@1) IN (SET) ([{c0:1,c1:10}, {c0:2,c1:}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@0 > 0 OR b_null_count@1 > 0 OR a_null_count@0 != row_count@3 AND a_max@2 >= 1 AND a_null_count@0 != row_count@3 AND a_min@4 <= 2 AND b_null_count@1 != row_count@3 AND b_max@5 >= 10 AND b_null_count@1 != row_count@3 AND b_min@6 <= 10, required_guarantees=[], metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=3, pushdown_rows_pruned=0, predicate_cache_inner_records=6, predicate_cache_records=6, scan_efficiency_ratio=18.16% (148/815)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/mnej_build.parquet]]}, projection=[a, b], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=16.42% (133/810)]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/mnej_probe.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 IS NULL OR b@1 IS NULL OR a@0 >= 1 AND a@0 <= 2 AND b@1 >= 10 AND b@1 <= 10 AND struct(a@0, b@1) IN (SET) ([{c0:1,c1:10}, {c0:2,c1:}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@0 > 0 OR b_null_count@1 > 0 OR a_null_count@0 != row_count@3 AND a_max@2 >= 1 AND a_null_count@0 != row_count@3 AND a_min@4 <= 2 AND b_null_count@1 != row_count@3 AND b_max@5 >= 10 AND b_null_count@1 != row_count@3 AND b_min@6 <= 10, required_guarantees=[], metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=3, pushdown_rows_pruned=0, predicate_cache_inner_records=6, predicate_cache_records=6, scan_efficiency_ratio=18.16% (148/815)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -1144,8 +1144,8 @@ EXPLAIN ANALYZE SELECT nnb_build.id, nnb_probe.id FROM nnb_build JOIN nnb_probe
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(id@0, id@0)], NullsEqual: true, metrics=[output_rows=1, output_batches=1, array_map_created_count=1, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=1, avg_fanout=100% (1/1), probe_hit_rate=100% (1/1)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnb_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=13.71% (68/496)]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnb_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 >= 11 AND id@0 <= 22 AND id@0 IN (SET) ([11, 22]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 >= 11 AND id_null_count@1 != row_count@2 AND id_min@3 <= 22 AND (id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND 11 <= id_max@0 OR id_null_count@1 != row_count@2 AND id_min@3 <= 22 AND 22 <= id_max@0), required_guarantees=[id in (11, 22)], metrics=[output_rows=1, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=3 total → 3 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=1, pushdown_rows_pruned=2, predicate_cache_inner_records=3, predicate_cache_records=1, scan_efficiency_ratio=14.45% (74/512)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnb_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=13.71% (68/496)]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnb_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 >= 11 AND id@0 <= 22 AND id@0 IN (SET) ([11, 22]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 >= 11 AND id_null_count@1 != row_count@2 AND id_min@3 <= 22 AND (id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND 11 <= id_max@0 OR id_null_count@1 != row_count@2 AND id_min@3 <= 22 AND 22 <= id_max@0), required_guarantees=[id in (11, 22)], metrics=[output_rows=1, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=3 total → 3 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=1, pushdown_rows_pruned=2, predicate_cache_inner_records=3, predicate_cache_records=1, scan_efficiency_ratio=14.45% (74/512)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
@@ -1185,8 +1185,8 @@ EXPLAIN ANALYZE SELECT nnp_build.id, nnp_probe.id FROM nnp_build JOIN nnp_probe
 ----
 Plan with Metrics
 01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(id@0, id@0)], NullsEqual: true, metrics=[output_rows=1, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=1, avg_fanout=100% (1/1), probe_hit_rate=100% (1/1)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnp_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=12.92% (65/503)]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnp_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 >= 11 AND id@0 <= 11 AND id@0 IN (SET) ([11, NULL]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 >= 11 AND id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND (id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND 11 <= id_max@0 OR id_null_count@1 != row_count@2 AND id_min@3 <= NULL AND NULL <= id_max@0), required_guarantees=[id in (11, NULL)], metrics=[output_rows=1, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=2 total → 2 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=1, pushdown_rows_pruned=1, predicate_cache_inner_records=2, predicate_cache_records=1, scan_efficiency_ratio=13.71% (68/496)]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnp_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, predicate_cache_inner_records=0, predicate_cache_records=0, scan_efficiency_ratio=12.92% (65/503)]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnp_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 >= 11 AND id@0 <= 11 AND id@0 IN (SET) ([11, NULL]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 >= 11 AND id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND (id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND 11 <= id_max@0 OR id_null_count@1 != row_count@2 AND id_min@3 <= NULL AND NULL <= id_max@0), required_guarantees=[id in (11, NULL)], metrics=[output_rows=1, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=1 total → 1 matched, page_index_rows_pruned=2 total → 2 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, post_scan_rows_matched=0, post_scan_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=1, pushdown_rows_pruned=1, predicate_cache_inner_records=2, predicate_cache_records=1, scan_efficiency_ratio=13.71% (68/496)]
 
 statement ok
 reset datafusion.explain.analyze_categories;
diff --git a/datafusion/sqllogictest/test_files/range_partitioning.slt b/datafusion/sqllogictest/test_files/range_partitioning.slt
index ec374b3d62a28..82d64f7df5d5f 100644
--- a/datafusion/sqllogictest/test_files/range_partitioning.slt
+++ b/datafusion/sqllogictest/test_files/range_partitioning.slt
@@ -342,8 +342,7 @@ ON l.range_key = r.range_key;
 physical_plan
 01)HashJoinExec: mode=Partitioned, join_type=Left, on=[(range_key@0, range_key@0)], projection=[range_key@0, value@1, value@3]
 02)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
-03)--FilterExec: value@1 <= 150
-04)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
+03)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
 
 query III
 SELECT l.range_key, l.value, r.value
@@ -370,8 +369,7 @@ ON l.range_key = r.range_key;
 physical_plan
 01)HashJoinExec: mode=Partitioned, join_type=LeftSemi, on=[(range_key@0, range_key@0)]
 02)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
-03)--FilterExec: value@1 <= 150, projection=[range_key@0]
-04)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
+03)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
 
 query II
 SELECT l.range_key, l.value
@@ -394,8 +392,7 @@ ON l.range_key = r.range_key;
 physical_plan
 01)HashJoinExec: mode=Partitioned, join_type=LeftAnti, on=[(range_key@0, range_key@0)]
 02)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
-03)--FilterExec: value@1 <= 150, projection=[range_key@0]
-04)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
+03)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
 
 query II
 SELECT l.range_key, l.value
@@ -428,8 +425,7 @@ physical_plan
 02)--RepartitionExec: partitioning=Hash([range_key@0, non_range_key@1], 4), input_partitions=4
 03)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, non_range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
 04)--RepartitionExec: partitioning=Hash([range_key@0, non_range_key@1], 4), input_partitions=4
-05)----FilterExec: value@2 <= 150
-06)------DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, non_range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
+05)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, non_range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
 
 # Range([range_key]) does not satisfy a join keyed on non_range_key.
 query TT
@@ -508,8 +504,7 @@ physical_plan
 01)FilterExec: non_range_key@1 = 2 OR mark@3, projection=[range_key@0, value@2]
 02)--HashJoinExec: mode=Partitioned, join_type=LeftMark, on=[(range_key@0, range_key@0)]
 03)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, non_range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
-04)----FilterExec: value@1 <= 150, projection=[range_key@0]
-05)------DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
+04)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
 
 query II
 SELECT l.range_key, l.value
@@ -755,9 +750,8 @@ RIGHT JOIN range_partitioned r ON l.range_key = r.range_key;
 ----
 physical_plan
 01)HashJoinExec: mode=Partitioned, join_type=Right, on=[(range_key@0, range_key@0)], projection=[value@1, range_key@2, value@3]
-02)--FilterExec: value@1 <= 150
-03)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
-04)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
+02)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
+03)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
 
 query III
 SELECT l.value, r.range_key, r.value
@@ -787,9 +781,8 @@ RIGHT SEMI JOIN range_partitioned r ON l.range_key = r.range_key;
 ----
 physical_plan
 01)HashJoinExec: mode=Partitioned, join_type=RightSemi, on=[(range_key@0, range_key@0)]
-02)--FilterExec: value@1 <= 150, projection=[range_key@0]
-03)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
-04)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
+02)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
+03)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
 
 query II
 SELECT r.range_key, r.value
@@ -815,9 +808,8 @@ RIGHT ANTI JOIN range_partitioned r ON l.range_key = r.range_key;
 ----
 physical_plan
 01)HashJoinExec: mode=Partitioned, join_type=RightAnti, on=[(range_key@0, range_key@0)]
-02)--FilterExec: value@1 <= 150, projection=[range_key@0]
-03)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
-04)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
+02)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
+03)--DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
 
 query II
 SELECT r.range_key, r.value
@@ -845,10 +837,9 @@ RIGHT JOIN range_partitioned_shifted r ON l.range_key = r.range_key;
 physical_plan
 01)HashJoinExec: mode=Partitioned, join_type=Right, on=[(range_key@0, range_key@0)], projection=[value@1, range_key@2, value@3]
 02)--RepartitionExec: partitioning=Hash([range_key@0], 4), input_partitions=4
-03)----FilterExec: value@1 <= 150
-04)------DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
-05)--RepartitionExec: partitioning=Hash([range_key@0], 4), input_partitions=4
-06)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(15), (20), (30)], 4), file_type=parquet
+03)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
+04)--RepartitionExec: partitioning=Hash([range_key@0], 4), input_partitions=4
+05)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(15), (20), (30)], 4), file_type=parquet
 
 query III
 SELECT l.value, r.range_key, r.value
@@ -952,10 +943,9 @@ RIGHT JOIN range_partitioned r ON l.non_range_key = r.non_range_key;
 physical_plan
 01)HashJoinExec: mode=Partitioned, join_type=Right, on=[(non_range_key@0, non_range_key@1)], projection=[value@1, range_key@2, value@4]
 02)--RepartitionExec: partitioning=Hash([non_range_key@0], 4), input_partitions=4
-03)----FilterExec: range_key@0 < 10, projection=[non_range_key@1, value@2]
-04)------DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, non_range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=range_key@0 < 10, pruning_predicate=range_key_null_count@1 != row_count@2 AND range_key_min@0 < 10, required_guarantees=[]
-05)--RepartitionExec: partitioning=Hash([non_range_key@1], 4), input_partitions=4
-06)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, non_range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
+03)----DataSourceExec: file_groups=<slt:ignore>, projection=[non_range_key, value], output_partitioning=UnknownPartitioning(4), file_type=parquet, predicate=range_key@0 < 10, pruning_predicate=range_key_null_count@1 != row_count@2 AND range_key_min@0 < 10, required_guarantees=[]
+04)--RepartitionExec: partitioning=Hash([non_range_key@1], 4), input_partitions=4
+05)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, non_range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
 
 query III
 SELECT l.value, r.range_key, r.value
@@ -988,8 +978,7 @@ physical_plan
 01)FilterExec: non_range_key@1 = 2 OR mark@3, projection=[range_key@0, value@2]
 02)--HashJoinExec: mode=Partitioned, join_type=LeftMark, on=[(range_key@0, range_key@0)]
 03)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, non_range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
-04)----FilterExec: value@1 <= 150, projection=[range_key@0]
-05)------DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet
+04)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=parquet, predicate=value@2 <= 150, pruning_predicate=value_null_count@1 != row_count@2 AND value_min@0 <= 150, required_guarantees=[]
 
 # Matched rows have mark=true and are returned; unmatched rows have
 # mark=false and are only returned when non_range_key = 2.
diff --git a/datafusion/sqllogictest/test_files/range_sorted_time_bin_agg.slt b/datafusion/sqllogictest/test_files/range_sorted_time_bin_agg.slt
index 18123a492dbd6..0a3ea2da5f353 100644
--- a/datafusion/sqllogictest/test_files/range_sorted_time_bin_agg.slt
+++ b/datafusion/sqllogictest/test_files/range_sorted_time_bin_agg.slt
@@ -100,8 +100,7 @@ physical_plan
 02)--AggregateExec: mode=FinalPartitioned, gby=[key@0 as key, date_bin(IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 60000000000 }"),range_sorted_time_bin.timestamp)@1 as date_bin(IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 60000000000 }"),range_sorted_time_bin.timestamp)], aggr=[sum(range_sorted_time_bin.value)], ordering_mode=Sorted
 03)----RepartitionExec: partitioning=Hash([key@0, date_bin(IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 60000000000 }"),range_sorted_time_bin.timestamp)@1], 2), input_partitions=2, preserve_order=true, sort_exprs=key@0 ASC, date_bin(IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 60000000000 }"),range_sorted_time_bin.timestamp)@1 ASC
 04)------AggregateExec: mode=Partial, gby=[key@0 as key, date_bin(IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 60000000000 }, timestamp@1) as date_bin(IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 60000000000 }"),range_sorted_time_bin.timestamp)], aggr=[sum(range_sorted_time_bin.value)], ordering_mode=Sorted
-05)--------FilterExec: col4@1 = a, projection=[key@0, timestamp@2, value@3]
-06)----------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch_range_partitioning/range_sorted_time_bin/part-0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch_range_partitioning/range_sorted_time_bin/part-1.parquet]]}, projection=[key, col4, timestamp, value], output_ordering=[key@0 ASC, timestamp@2 ASC], output_partitioning=Range([timestamp@2 ASC], [(1704070800000000000)], 2), file_type=parquet, predicate=col4@4 = a, pruning_predicate=col4_null_count@2 != row_count@3 AND col4_min@0 <= a AND a <= col4_max@1, required_guarantees=[col4 in (a)]
+05)--------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch_range_partitioning/range_sorted_time_bin/part-0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch_range_partitioning/range_sorted_time_bin/part-1.parquet]]}, projection=[key, timestamp, value], output_ordering=[key@0 ASC, timestamp@1 ASC], output_partitioning=Range([timestamp@1 ASC], [(1704070800000000000)], 2), file_type=parquet, predicate=col4@4 = a, pruning_predicate=col4_null_count@2 != row_count@3 AND col4_min@0 <= a AND a <= col4_max@1, required_guarantees=[col4 in (a)]
 
 query TPI
 SELECT key, date_bin(INTERVAL '60 seconds', timestamp) AS time_bin, sum(value)

From 31199fcee93a3fb96117854ec5ad3beb554f3bad Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 23:39:42 -0500
Subject: [PATCH 03/42] fix: stop the Parquet push decoder stream after the
 final flush

When the TopK dynamic filter prunes every remaining row group at a row
group boundary, `rebuild_decoder_at_boundary` returns `Ok(true)` and the
stream calls `finish()`. With a batch coalescer (a post-scan filter is
present, for example with the default `pushdown_filters = false`),
`finish()` flushes the coalescer and returns a batch. The next poll then
went back to the decoder, which still pointed at a row group that the
plan had dropped, and `sync_rg_plan_to_decoder_frontier` failed with
"push decoder frontier RG N is not in rg_plan; decoder and plan have
diverged". ClickBench Q23, Q24 and Q26 fail with this error.

After the flush, the stream now only drains the coalescer.

The new sqllogictest in `dynamic_row_group_pruning.slt` fails without
the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../datasource-parquet/src/push_decoder.rs    | 10 ++++
 .../test_files/dynamic_row_group_pruning.slt  | 50 +++++++++++++++++++
 2 files changed, 60 insertions(+)

diff --git a/datafusion/datasource-parquet/src/push_decoder.rs b/datafusion/datasource-parquet/src/push_decoder.rs
index a811212a3dfe3..e38cec26da65a 100644
--- a/datafusion/datasource-parquet/src/push_decoder.rs
+++ b/datafusion/datasource-parquet/src/push_decoder.rs
@@ -481,6 +481,16 @@ impl PushDecoderStreamState {
         // every return.
         let elapsed_compute = self.baseline_metrics.elapsed_compute().clone();
         let mut timer = elapsed_compute.timer();
+        // Once `finish` has flushed the coalescer, the stream only drains it.
+        // The decoder can still point at row groups that the plan dropped
+        // (for example, when a dynamic filter pruned every remaining row
+        // group at a boundary), so it must not be driven again.
+        if self.flushed {
+            if self.remaining_limit == Some(0) {
+                return None;
+            }
+            return self.emit_completed();
+        }
         loop {
             // Hand out anything the coalescer has already assembled into a
             // full-size batch before doing more decoding work.
diff --git a/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt b/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt
index c674dede75706..fec9add854b3a 100644
--- a/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt
+++ b/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt
@@ -329,3 +329,53 @@ set datafusion.execution.target_partitions = 4;
 
 statement ok
 RESET datafusion.execution.parquet.pushdown_filters;
+
+# Regression test: the TopK dynamic filter prunes every remaining row group at a
+# row group boundary while the in-scan post-scan filter (`pushdown_filters =
+# false`, the default) holds rows in its batch coalescer. The stream flushes the
+# coalescer and must then only drain it. Before the fix the next poll went back
+# to the decoder, which still pointed at a row group that the plan had dropped,
+# and failed with "decoder and plan have diverged".
+# Layout: 10 row groups of 1000 rows, row group k holds `ts` k*1000..k*1000+999
+# in a scrambled order, so the first row group fills the TopK heap and its
+# threshold prunes all other row groups.
+statement ok
+set datafusion.execution.target_partitions = 1;
+
+statement ok
+COPY (
+  SELECT (value / 1000) * 1000 + ((value * 7919) % 1000) AS ts, 'x' AS s
+  FROM generate_series(0, 9999)
+)
+TO 'test_files/scratch/dynamic_row_group_pruning/flush.parquet'
+STORED AS PARQUET
+OPTIONS ('format.max_row_group_size' '1000');
+
+statement ok
+CREATE EXTERNAL TABLE flush_t
+STORED AS PARQUET
+LOCATION 'test_files/scratch/dynamic_row_group_pruning/flush.parquet';
+
+query IIII
+SELECT count(*), count(DISTINCT ts), min(ts), max(ts)
+FROM (SELECT * FROM flush_t WHERE s <> '' ORDER BY ts LIMIT 500);
+----
+500 500 0 499
+
+statement ok
+set datafusion.execution.parquet.pushdown_filters = true;
+
+query IIII
+SELECT count(*), count(DISTINCT ts), min(ts), max(ts)
+FROM (SELECT * FROM flush_t WHERE s <> '' ORDER BY ts LIMIT 500);
+----
+500 500 0 499
+
+statement ok
+drop table flush_t;
+
+statement ok
+set datafusion.execution.target_partitions = 4;
+
+statement ok
+RESET datafusion.execution.parquet.pushdown_filters;

From c2d8f85980716908689922b349bce16bd9ea11bc Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 20:04:41 -0500
Subject: [PATCH 04/42] refactor(physical-plan): share the round-robin row
 check of EnforceDistribution

Move the check "a round-robin repartition of this input is useful for its
row count" from `EnforceDistribution` into
`repartition::round_robin_beneficial_for_rows`. The behavior does not
change. The next commit uses the same check in the file scan, so that the
scan and the optimizer make the same decision.

PR: #22384

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../enforce_distribution.rs                   | 34 +++++++------------
 .../physical-plan/src/repartition/mod.rs      | 24 +++++++++++++
 2 files changed, 36 insertions(+), 22 deletions(-)

diff --git a/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs b/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs
index 776e3c3bf14bd..fb62ea0e1b8b3 100644
--- a/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs
+++ b/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs
@@ -41,7 +41,6 @@ use crate::utils::{
 use arrow::compute::SortOptions;
 use datafusion_common::config::ConfigOptions;
 use datafusion_common::error::Result;
-use datafusion_common::stats::Precision;
 use datafusion_common::tree_node::Transformed;
 use datafusion_expr::logical_plan::{Aggregate, JoinType};
 use datafusion_physical_expr::expressions::{Column, NoOp};
@@ -62,7 +61,9 @@ use datafusion_physical_plan::joins::{
     CrossJoinExec, HashJoinExec, PartitionMode, SortMergeJoinExec,
 };
 use datafusion_physical_plan::projection::{ProjectionExec, ProjectionExpr};
-use datafusion_physical_plan::repartition::RepartitionExec;
+use datafusion_physical_plan::repartition::{
+    RepartitionExec, round_robin_beneficial_for_rows,
+};
 use datafusion_physical_plan::sorts::sort::SortExec;
 use datafusion_physical_plan::sorts::sort_preserving_merge::SortPreservingMergeExec;
 use datafusion_physical_plan::statistics::{StatisticsArgs, StatisticsContext};
@@ -995,8 +996,7 @@ struct DistributionChildState {
 )]
 fn get_repartition_requirement_status(
     plan: &Arc<dyn ExecutionPlan>,
-    batch_size: usize,
-    should_use_estimates: bool,
+    config: &ConfigOptions,
     stats_ctx: &StatisticsContext,
 ) -> Result<Vec<RepartitionRequirementStatus>> {
     let mut needs_alignment = false;
@@ -1009,14 +1009,12 @@ fn get_repartition_requirement_status(
     {
         // Decide whether adding a round robin is beneficial depending on
         // the statistical information we have on the number of rows:
-        let roundrobin_beneficial_stats = match stats_ctx
-            .compute(child.as_ref(), &StatisticsArgs::new())?
-            .num_rows
-        {
-            Precision::Exact(n_rows) => n_rows > batch_size,
-            Precision::Inexact(n_rows) => !should_use_estimates || (n_rows > batch_size),
-            Precision::Absent => true,
-        };
+        let roundrobin_beneficial_stats = round_robin_beneficial_for_rows(
+            &stats_ctx
+                .compute(child.as_ref(), &StatisticsArgs::new())?
+                .num_rows,
+            config,
+        );
         let is_hash = matches!(
             requirement,
             Distribution::HashPartitioned(_) | Distribution::KeyPartitioned(_)
@@ -1375,10 +1373,6 @@ pub fn ensure_distribution_with_stats(
     // When `false`, round robin repartition will not be added to increase parallelism
     let enable_round_robin = config.optimizer.enable_round_robin_repartition;
     let repartition_file_scans = config.optimizer.repartition_file_scans;
-    let batch_size = config.execution.batch_size.get();
-    let should_use_estimates = config
-        .execution
-        .use_row_number_estimates_to_optimize_partitioning;
     let subset_satisfaction_threshold = config.optimizer.subset_repartition_threshold;
     let unbounded_and_pipeline_friendly = dist_context.plan.boundedness().is_unbounded()
         && matches!(
@@ -1455,12 +1449,8 @@ pub fn ensure_distribution_with_stats(
         || plan.is::<SortMergeJoinExec>();
 
     let input_distributions = plan.input_distribution_requirements();
-    let repartition_status_flags = get_repartition_requirement_status(
-        &plan,
-        batch_size,
-        should_use_estimates,
-        stats_ctx,
-    )?;
+    let repartition_status_flags =
+        get_repartition_requirement_status(&plan, config, stats_ctx)?;
     // This loop iterates over all the children to:
     // - Increase parallelism for every child if it is beneficial.
     // - Satisfy the distribution requirements of every child, if it is not
diff --git a/datafusion/physical-plan/src/repartition/mod.rs b/datafusion/physical-plan/src/repartition/mod.rs
index 794888b4ea827..773fe6869eac1 100644
--- a/datafusion/physical-plan/src/repartition/mod.rs
+++ b/datafusion/physical-plan/src/repartition/mod.rs
@@ -1384,6 +1384,30 @@ impl BatchPartitioner {
     }
 }
 
+/// Returns `true` if the row count of an input makes a round-robin
+/// repartition of that input useful: the input has more rows than one batch,
+/// or its row count is unknown.
+///
+/// The optimizer uses an inexact row count only when
+/// `datafusion.execution.use_row_number_estimates_to_optimize_partitioning`
+/// is set. Otherwise it treats an inexact row count as unknown.
+pub fn round_robin_beneficial_for_rows(
+    num_rows: &Precision<usize>,
+    config: &ConfigOptions,
+) -> bool {
+    let batch_size = config.execution.batch_size.get();
+    match num_rows {
+        Precision::Exact(n_rows) => *n_rows > batch_size,
+        Precision::Inexact(n_rows) => {
+            !config
+                .execution
+                .use_row_number_estimates_to_optimize_partitioning
+                || *n_rows > batch_size
+        }
+        Precision::Absent => true,
+    }
+}
+
 /// Maps `N` input partitions to `M` output partitions based on a
 /// [`Partitioning`] scheme.
 ///

From 9ff1146909c23ff7fec54fc00fff73ceefe40cab Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 21:38:26 -0500
Subject: [PATCH 05/42] fix(datasource): keep a filter above a scan that cannot
 give the target partitions

The Parquet scan accepts all pushable filters, thus `FilterPushdown`
removes the `FilterExec`. For a scan of one small file (one partition,
too small to split into byte ranges), main puts a round-robin
`RepartitionExec` between the scan and the `FilterExec`, and a
`CoalescePartitionsExec` above them. Without the `FilterExec`, the
optimizer adds neither. The filter then runs in one partition, and the
scans of the build sides of the hash joins run one after the other in the
task of the probe side, not in parallel tasks. On TPC-DS SF1 this made
short queries 5% to 30% slower than main (for example Q37 1.26x).

`FileScanConfig::try_pushdown_filters` now makes the same decision as
`EnforceDistribution`: if the scan has fewer than `target_partitions`
partitions, `repartitioned` cannot give more, and a round-robin
repartition is useful for the rows that the scan reads, the filters stay
above the scan (`PushedDown::No`). The scan still gets them, through the
new `FileSource::try_pushdown_pruning_filters`, and uses them only to
prune files, row groups and pages. This is what main does with all
filters when `pushdown_filters` is false. The plan is then the plan of
main for these scans.

- Only the filters of a `FilterExec` stay above the scan. A dynamic filter
  (of a join, a TopK or an aggregate) has no `FilterExec` above the scan,
  thus the scan applies it as before.
- The default of `try_pushdown_pruning_filters` returns `None`: other file
  sources get their filters as before.
- The Parquet scan applies all conjuncts of its predicate or none of them.
  A scan with a pruning-only predicate uses later filters only to prune
  too.
- `ParquetScanExecNode` gets `pruning_only_predicate`, thus a decoded scan
  does not apply its predicate again.
- An exact row count of at most one batch keeps the filter in the scan: a
  round-robin repartition cannot split one batch.

Tests:
- unit tests for the decision in `file_scan_config` and for the
  pruning-only predicate of `ParquetSource`;
- a proto round trip of the pruning-only predicate;
- a sqllogictest plan pin in `parquet_filter_pushdown.slt`;
- `parquet_statistics.slt` (no statistics, thus unknown rows): the plan
  is the plan of main again;
- two Parquet integration tests that check the filter in the scan use one
  target partition.

PR: #22384

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/core/tests/parquet/expr_adapter.rs |   6 +-
 .../core/tests/parquet/filter_pushdown.rs     |   4 +
 .../datasource-parquet/src/opener/mod.rs      |  14 +-
 datafusion/datasource-parquet/src/source.rs   | 115 ++++++-
 datafusion/datasource/src/file.rs             |  20 ++
 .../datasource/src/file_scan_config/mod.rs    | 312 +++++++++++++++++-
 .../proto-models/proto/datafusion.proto       |   4 +
 .../proto-models/src/generated/pbjson.rs      |  18 +
 .../proto-models/src/generated/prost.rs       |   4 +
 datafusion/proto/tests/cases/plans/sources.rs |  43 +++
 .../test_files/parquet_filter_pushdown.slt    |  69 ++++
 .../test_files/parquet_statistics.slt         |   5 +-
 12 files changed, 608 insertions(+), 6 deletions(-)

diff --git a/datafusion/core/tests/parquet/expr_adapter.rs b/datafusion/core/tests/parquet/expr_adapter.rs
index dfed2eb5bda74..c2668a0734d8a 100644
--- a/datafusion/core/tests/parquet/expr_adapter.rs
+++ b/datafusion/core/tests/parquet/expr_adapter.rs
@@ -875,10 +875,14 @@ async fn test_all_null_struct_decimal_cast_filter_pushdown() -> Result<()> {
         write_parquet(batch, Arc::clone(&store), "null_decimal/data.parquet").await;
 
         for pushdown_filters in [false, true] {
+            // One target partition: with more, the scan of one file keeps the
+            // filter in a `FilterExec` above it, which runs in more
+            // partitions. This test checks the filter in the scan.
             let mut config = SessionConfig::new()
                 .with_collect_statistics(false)
                 .with_parquet_pruning(false)
-                .with_parquet_page_index_pruning(false);
+                .with_parquet_page_index_pruning(false)
+                .with_target_partitions(1);
             config.options_mut().execution.parquet.pushdown_filters = pushdown_filters;
             let ctx = SessionContext::new_with_config(config);
             register_memory_listing_table(
diff --git a/datafusion/core/tests/parquet/filter_pushdown.rs b/datafusion/core/tests/parquet/filter_pushdown.rs
index d337979e5fd00..a450d62d1cc87 100644
--- a/datafusion/core/tests/parquet/filter_pushdown.rs
+++ b/datafusion/core/tests/parquet/filter_pushdown.rs
@@ -655,6 +655,10 @@ async fn predicate_cache_stats_issue_19561() -> datafusion_common::Result<()> {
     // force to get multiple batches to trigger repeated metric compound bug
     config.options_mut().execution.batch_size =
         datafusion_common::config::ConfigNonZeroUsize::try_new(1)?;
+    // One target partition: with more, the scan of one file keeps the
+    // filter in a `FilterExec` above it (8 rows are 8 batches), and this
+    // test checks the row filter of the scan.
+    config.options_mut().execution.target_partitions = 1;
     let ctx = SessionContext::new_with_config(config);
     // The cache is on by default, and used when filter pushdown is enabled
     PredicateCacheTest {
diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index 9f28d1ef94e4a..5108ff93890f8 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -252,6 +252,9 @@ pub(super) struct ParquetMorselizer {
     pub preserve_order: bool,
     /// Optional predicate to apply during the scan
     pub predicate: Option<Arc<dyn PhysicalExpr>>,
+    /// If true, the scan uses `predicate` only to prune: a `FilterExec`
+    /// above the scan applies it.
+    pub pruning_only_predicate: bool,
     /// Table schema, including partition columns.
     pub table_schema: TableSchema,
     /// Optional hint for how large the initial request to read parquet metadata
@@ -455,6 +458,7 @@ struct PreparedParquetOpen {
     output_schema: SchemaRef,
     projection: ProjectionExprs,
     predicate: Option<Arc<dyn PhysicalExpr>>,
+    pruning_only_predicate: bool,
     /// Per-scan virtual-column state, Arc-cloned from [`ParquetMorselizer`] so
     /// each file shares validated fields, precomputed null replacements, and
     /// the logical-with-virtual schema. `None` when no virtual columns were
@@ -556,8 +560,14 @@ impl DecoderReadPlans {
         //
         // Either way every conjunct is applied; nothing is silently dropped.
         // ---------------------------------------------------------------
+        // A pruning-only predicate is applied by a `FilterExec` above the
+        // scan: the scan uses it only to prune.
+        let applied_predicate = prepared
+            .predicate
+            .as_ref()
+            .filter(|_| !prepared.pruning_only_predicate);
         let (row_filter_context, post_scan_conjuncts) =
-            match (prepared.pushdown_filters, prepared.predicate.as_ref()) {
+            match (prepared.pushdown_filters, applied_predicate) {
                 // Pushdown enabled: precompute the candidate list once per file.
                 // Both the initial `RowFilter` and any per-RG rebuilds (via
                 // `RowFilterContext::build_row_filter`) reuse it, so tree walks
@@ -1005,6 +1015,7 @@ impl ParquetMorselizer {
             output_schema,
             projection,
             predicate,
+            pruning_only_predicate: self.pruning_only_predicate,
             virtual_state: self.virtual_state.as_ref().map(Arc::clone),
             reorder_predicates: self.reorder_filters,
             pushdown_filters: self.pushdown_filters,
@@ -2613,6 +2624,7 @@ mod test {
                 limit: self.limit,
                 preserve_order: self.preserve_order,
                 predicate: self.predicate,
+                pruning_only_predicate: false,
                 table_schema,
                 metadata_size_hint: self.metadata_size_hint,
                 metrics: self.metrics,
diff --git a/datafusion/datasource-parquet/src/source.rs b/datafusion/datasource-parquet/src/source.rs
index c1e7e47b459e3..686ca4f1ad5a4 100644
--- a/datafusion/datasource-parquet/src/source.rs
+++ b/datafusion/datasource-parquet/src/source.rs
@@ -304,6 +304,10 @@ pub struct ParquetSource {
     pub(crate) table_schema: TableSchema,
     /// Optional predicate for row filtering during parquet scan
     pub(crate) predicate: Option<Arc<dyn PhysicalExpr>>,
+    /// If true, the scan uses [`Self::predicate`] only to prune: a
+    /// `FilterExec` above the scan applies it (see
+    /// [`FileSource::try_pushdown_pruning_filters`]).
+    pub(crate) pruning_only_predicate: bool,
     /// Optional user defined parquet file reader factory
     pub(crate) parquet_file_reader_factory: Option<Arc<dyn ParquetFileReaderFactory>>,
     /// Optional policy for deriving each file's Arrow schema after footer loading.
@@ -342,6 +346,7 @@ impl ParquetSource {
             table_parquet_options: TableParquetOptions::default(),
             metrics: ExecutionPlanMetricsSet::new(),
             predicate: None,
+            pruning_only_predicate: false,
             parquet_file_reader_factory: None,
             schema_provider: None,
             batch_size: None,
@@ -644,7 +649,7 @@ impl FileSource for ParquetSource {
             self.table_schema.virtual_columns(),
             self.table_schema.file_schema(),
             self.predicate.as_ref(),
-            self.pushdown_filters(),
+            self.pushdown_filters() && !self.pruning_only_predicate,
         )?;
 
         Ok(Box::new(ParquetMorselizer {
@@ -656,6 +661,7 @@ impl FileSource for ParquetSource {
             limit: base_config.limit,
             preserve_order: base_config.preserve_order,
             predicate: self.predicate.clone(),
+            pruning_only_predicate: self.pruning_only_predicate,
             table_schema: self.table_schema.clone(),
             metadata_size_hint: self.metadata_size_hint,
             metrics: self.metrics().clone(),
@@ -894,6 +900,16 @@ impl FileSource for ParquetSource {
             None => conjunction(allowed_filters),
         };
         source.predicate = Some(predicate);
+        if self.pruning_only_predicate {
+            // The scan applies all conjuncts of its predicate or none of
+            // them. It uses its predicate only to prune (a `FilterExec`
+            // above the scan applies the filters that it got before), thus it
+            // uses the new filters only to prune too.
+            return Ok(FilterPushdownPropagation::with_parent_pushdown_result(
+                vec![PushedDown::No; filters.len()],
+            )
+            .with_updated_node(Arc::new(source) as _));
+        }
         source = source.with_pushdown_filters(pushdown_filters);
         let source = Arc::new(source);
         // The parquet scan always accepts pushable filters: report each
@@ -910,6 +926,36 @@ impl FileSource for ParquetSource {
         .with_updated_node(source))
     }
 
+    /// Takes the pushable `filters` for pruning only, as the scan does with
+    /// all filters when `pushdown_filters` is false on main. The scan applies
+    /// either all conjuncts of its predicate or none of them, thus it refuses
+    /// (`None`) when it already has a predicate that it applies.
+    fn try_pushdown_pruning_filters(
+        &self,
+        filters: &[Arc<dyn PhysicalExpr>],
+        _config: &ConfigOptions,
+    ) -> datafusion_common::Result<Option<Arc<dyn FileSource>>> {
+        if self.predicate.is_some() && !self.pruning_only_predicate {
+            return Ok(None);
+        }
+        let pushable_schema = self.table_schema.schema_without_virtual_columns();
+        let pushable = filters
+            .iter()
+            .filter(|filter| {
+                can_expr_be_pushed_down_with_schemas(filter, pushable_schema)
+            })
+            .cloned()
+            .collect_vec();
+        if pushable.is_empty() {
+            return Ok(None);
+        }
+        let mut source = self.clone();
+        source.predicate =
+            Some(conjunction(self.predicate.iter().cloned().chain(pushable)));
+        source.pruning_only_predicate = true;
+        Ok(Some(Arc::new(source)))
+    }
+
     /// Try to optimize the scan to produce data in the requested sort order.
     ///
     /// Inputs:
@@ -1112,6 +1158,7 @@ impl FileSource for ParquetSource {
             // Carried by `base`.
             table_schema: _,
             predicate,
+            pruning_only_predicate,
             // Rebuilt from the decode context.
             parquet_file_reader_factory: _,
             // Requires a custom codec for serialization.
@@ -1158,6 +1205,7 @@ impl FileSource for ParquetSource {
             sort_order_for_reorder,
             reverse_row_groups: *reverse_row_groups,
             metadata_size_hint,
+            pruning_only_predicate: *pruning_only_predicate,
         };
         Ok(Some(protobuf::PhysicalPlanNode {
             physical_plan_type: Some(PhysicalPlanType::ParquetScan(node)),
@@ -1199,6 +1247,7 @@ impl ParquetSource {
             sort_order_for_reorder,
             reverse_row_groups,
             metadata_size_hint,
+            pruning_only_predicate,
         } = scan;
 
         let base_conf = base_conf.as_ref().ok_or_else(|| {
@@ -1283,6 +1332,7 @@ impl ParquetSource {
         if let Some(predicate) = predicate {
             source = source.with_predicate(predicate);
         }
+        source.pruning_only_predicate = *pruning_only_predicate;
         let base_config =
             FileScanConfig::try_from_proto(base_conf, ctx, Arc::new(source))?;
         Ok(DataSourceExec::from_data_source(base_config))
@@ -2108,6 +2158,69 @@ mod tests {
         }
     }
 
+    /// Filters for pruning only stay above the scan. The scan applies all
+    /// conjuncts of its predicate or none of them.
+    #[test]
+    fn pruning_filters_stay_above_the_scan() {
+        use arrow::datatypes::{DataType, Field, Schema};
+        use datafusion_common::config::ConfigOptions;
+        use datafusion_expr::{col, lit as logical_lit};
+        use datafusion_physical_expr::planner::logical2physical;
+        use datafusion_physical_plan::filter_pushdown::PushedDown;
+
+        let schema = Arc::new(Schema::new(vec![Field::new(
+            "value",
+            DataType::Int64,
+            false,
+        )]));
+        let filter = |v: i64| logical2physical(&col("value").gt(logical_lit(v)), &schema);
+        let config = ConfigOptions::default();
+        let downcast = |source: &Arc<dyn FileSource>| {
+            source.downcast_ref::<ParquetSource>().unwrap().clone()
+        };
+
+        // A source without a predicate takes filters for pruning only.
+        let source = ParquetSource::new(Arc::clone(&schema));
+        let pruning = downcast(
+            &source
+                .try_pushdown_pruning_filters(&[filter(1)], &config)
+                .unwrap()
+                .expect("the source takes filters for pruning only"),
+        );
+        assert!(pruning.pruning_only_predicate);
+        assert_eq!(
+            pruning.predicate.as_ref().unwrap().to_string(),
+            "value@0 > 1"
+        );
+
+        // It uses later filters only to prune too.
+        let prop = pruning
+            .try_pushdown_filters(vec![filter(2)], &config)
+            .unwrap();
+        assert!(matches!(prop.filters[..], [PushedDown::No]));
+        let later = downcast(&prop.updated_node.unwrap());
+        assert!(later.pruning_only_predicate);
+        assert_eq!(
+            later.predicate.as_ref().unwrap().to_string(),
+            "value@0 > 1 AND value@0 > 2"
+        );
+
+        // A source that applies its predicate does not take filters for
+        // pruning only.
+        let prop = source
+            .try_pushdown_filters(vec![filter(1)], &config)
+            .unwrap();
+        assert!(matches!(prop.filters[..], [PushedDown::Yes]));
+        let applied = downcast(&prop.updated_node.unwrap());
+        assert!(!applied.pruning_only_predicate);
+        assert!(
+            applied
+                .try_pushdown_pruning_filters(&[filter(2)], &config)
+                .unwrap()
+                .is_none()
+        );
+    }
+
     #[test]
     fn test_try_pushdown_filters_rejects_virtual_column_refs() {
         // Virtual columns are produced by the reader and cannot be referenced
diff --git a/datafusion/datasource/src/file.rs b/datafusion/datasource/src/file.rs
index f1a94f2e12363..88c98ad058476 100644
--- a/datafusion/datasource/src/file.rs
+++ b/datafusion/datasource/src/file.rs
@@ -208,6 +208,26 @@ pub trait FileSource: Any + Send + Sync {
         ))
     }
 
+    /// Try to push down filters that stay above the scan, for pruning only.
+    ///
+    /// A `FilterExec` above the scan applies `filters`, thus the source does
+    /// not need to apply them to the rows. It can use them to prune, for
+    /// example files, row groups and pages. `filters` are in terms of the
+    /// unprojected table schema, as for [`Self::try_pushdown_filters`].
+    ///
+    /// `FileScanConfig` calls this method instead of
+    /// [`Self::try_pushdown_filters`] when a `FilterExec` above the scan runs
+    /// in more partitions than the scan. Returns the new source, or `None`
+    /// (the default) if the source does not support it: then
+    /// `FileScanConfig` calls [`Self::try_pushdown_filters`].
+    fn try_pushdown_pruning_filters(
+        &self,
+        _filters: &[Arc<dyn PhysicalExpr>],
+        _config: &ConfigOptions,
+    ) -> Result<Option<Arc<dyn FileSource>>> {
+        Ok(None)
+    }
+
     /// Try to create a new FileSource that can produce data in the specified sort order.
     ///
     /// This method attempts to optimize data retrieval to match the requested ordering.
diff --git a/datafusion/datasource/src/file_scan_config/mod.rs b/datafusion/datasource/src/file_scan_config/mod.rs
index 61b41fd1e95a9..740f715592303 100644
--- a/datafusion/datasource/src/file_scan_config/mod.rs
+++ b/datafusion/datasource/src/file_scan_config/mod.rs
@@ -52,7 +52,9 @@ use datafusion_common::stats::{Precision, is_known_empty};
 use datafusion_physical_expr::expressions::{BinaryExpr, Column};
 use datafusion_physical_expr::projection::{ProjectionExprs, ProjectionMapping};
 use datafusion_physical_expr::utils::reassign_expr_columns;
-use datafusion_physical_expr::{EquivalenceProperties, Partitioning, split_conjunction};
+use datafusion_physical_expr::{
+    DynamicFilterTracking, EquivalenceProperties, Partitioning, split_conjunction,
+};
 use datafusion_physical_expr_adapter::PhysicalExprAdapterFactory;
 use datafusion_physical_expr_common::physical_expr::{PhysicalExpr, is_volatile};
 use datafusion_physical_expr_common::sort_expr::{LexOrdering, PhysicalSortExpr};
@@ -62,8 +64,9 @@ use datafusion_physical_plan::execution_plan::SchedulingType;
 use datafusion_physical_plan::{
     DisplayAs, DisplayFormatType,
     display::{ProjectSchemaDisplay, display_orderings},
-    filter_pushdown::FilterPushdownPropagation,
+    filter_pushdown::{FilterPushdownPropagation, PushedDown},
     metrics::ExecutionPlanMetricsSet,
+    repartition::round_robin_beneficial_for_rows,
 };
 use log::{debug, warn};
 use std::any::Any;
@@ -1055,6 +1058,28 @@ impl DataSource for FileScanConfig {
             .map(|filter| reassign_expr_columns(filter, table_schema))
             .collect::<Result<Vec<_>>>()?;
 
+        // A filter that the scan applies runs in the partitions of the scan.
+        // If the optimizer would run a `FilterExec` above this scan in more
+        // partitions than the scan has, the filters stay above the scan
+        // (`PushedDown::No`), if the file source can use them for pruning
+        // only. This is only for the filters of a `FilterExec`: a dynamic
+        // filter (of a join, a TopK or an aggregate) has no `FilterExec`
+        // above the scan.
+        if !remapped_filters.iter().any(|filter| {
+            DynamicFilterTracking::classify(filter).contains_dynamic_filter()
+        }) && self.filters_run_in_more_partitions_above(config)?
+            && let Some(file_source) = self
+                .file_source
+                .try_pushdown_pruning_filters(&remapped_filters, config)?
+        {
+            let mut new_file_scan_config = self.clone();
+            new_file_scan_config.file_source = file_source;
+            return Ok(FilterPushdownPropagation {
+                filters: vec![PushedDown::No; remapped_filters.len()],
+                updated_node: Some(Arc::new(new_file_scan_config) as _),
+            });
+        }
+
         let result = self
             .file_source
             .try_pushdown_filters(remapped_filters, config)?;
@@ -1306,6 +1331,49 @@ impl FileScanConfig {
         )
     }
 
+    /// Returns `true` if a filter above this scan runs in more partitions than
+    /// a filter in this scan.
+    ///
+    /// A filter that the scan applies runs in the partitions of the scan. A
+    /// `FilterExec` above the scan runs in the partitions of its input. If
+    /// the scan cannot give `target_partitions` partitions, the
+    /// `EnforceDistribution` rule puts a round-robin `RepartitionExec`
+    /// between the scan and the `FilterExec`, when the input can have more
+    /// rows than one batch. The `FilterExec` then runs in
+    /// `target_partitions` partitions.
+    ///
+    /// This function uses the same checks as `EnforceDistribution`: the
+    /// partitions that [`DataSource::repartitioned`] gives, and
+    /// [`round_robin_beneficial_for_rows`] on the rows that the scan reads.
+    /// The rows before the filter are the input of that round-robin
+    /// repartition. An exact count is a limit: when it is at most one batch,
+    /// a round-robin repartition cannot split the work.
+    fn filters_run_in_more_partitions_above(
+        &self,
+        config: &ConfigOptions,
+    ) -> Result<bool> {
+        let target_partitions = config.execution.target_partitions;
+        let partitions = self.output_partitioning().partition_count();
+        if !config.optimizer.enable_round_robin_repartition
+            || partitions >= target_partitions
+            || !round_robin_beneficial_for_rows(&self.statistics.num_rows, config)
+        {
+            return Ok(false);
+        }
+        if !config.optimizer.repartition_file_scans {
+            return Ok(true);
+        }
+        let repartitioned = self.repartitioned(
+            target_partitions,
+            config.optimizer.repartition_file_min_size,
+            self.eq_properties().output_ordering(),
+        )?;
+        let partitions = repartitioned
+            .map(|source| source.output_partitioning().partition_count())
+            .unwrap_or(partitions);
+        Ok(partitions < target_partitions)
+    }
+
     /// Get the file schema (schema of the files without partition columns)
     pub fn file_schema(&self) -> &SchemaRef {
         self.file_source.table_schema().file_schema()
@@ -4127,4 +4195,244 @@ mod tests {
 
         assert!(!would_duplicate_costly_exprs(&inner, &outer));
     }
+
+    /// Tests for the filters that stay above a scan that cannot give the
+    /// parallelism of a filter above it.
+    mod filters_above_scan {
+        use super::*;
+        use datafusion_physical_expr::expressions::{binary, lit};
+
+        /// A file source that applies all filters that it gets. If
+        /// `pruning_filters` is true, it also takes filters for pruning only.
+        #[derive(Clone)]
+        struct FilteringSource {
+            metrics: ExecutionPlanMetricsSet,
+            table_schema: TableSchema,
+            pruning_filters: bool,
+            filter: Option<Arc<dyn PhysicalExpr>>,
+            pruning_only: bool,
+        }
+
+        impl FileSource for FilteringSource {
+            fn create_file_opener(
+                &self,
+                _object_store: Arc<dyn ObjectStore>,
+                _base_config: &FileScanConfig,
+                _partition: usize,
+            ) -> Result<Arc<dyn crate::file_stream::FileOpener>> {
+                unimplemented!()
+            }
+
+            fn table_schema(&self) -> &TableSchema {
+                &self.table_schema
+            }
+
+            fn with_batch_size(&self, _batch_size: usize) -> Arc<dyn FileSource> {
+                Arc::new(self.clone())
+            }
+
+            fn metrics(&self) -> &ExecutionPlanMetricsSet {
+                &self.metrics
+            }
+
+            fn file_type(&self) -> &str {
+                "filtering"
+            }
+
+            fn filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
+                self.filter.clone()
+            }
+
+            fn try_pushdown_filters(
+                &self,
+                filters: Vec<Arc<dyn PhysicalExpr>>,
+                _config: &ConfigOptions,
+            ) -> Result<FilterPushdownPropagation<Arc<dyn FileSource>>> {
+                let pushed_down = vec![PushedDown::Yes; filters.len()];
+                let source = Self {
+                    filter: Some(datafusion_physical_expr::conjunction(filters)),
+                    pruning_only: false,
+                    ..self.clone()
+                };
+                Ok(
+                    FilterPushdownPropagation::with_parent_pushdown_result(pushed_down)
+                        .with_updated_node(Arc::new(source) as _),
+                )
+            }
+
+            fn try_pushdown_pruning_filters(
+                &self,
+                filters: &[Arc<dyn PhysicalExpr>],
+                _config: &ConfigOptions,
+            ) -> Result<Option<Arc<dyn FileSource>>> {
+                Ok(self.pruning_filters.then(|| {
+                    Arc::new(Self {
+                        filter: Some(datafusion_physical_expr::conjunction(
+                            filters.iter().cloned(),
+                        )),
+                        pruning_only: true,
+                        ..self.clone()
+                    }) as _
+                }))
+            }
+
+            fn apply_expressions(
+                &self,
+                _f: &mut dyn FnMut(&Arc<dyn PhysicalExpr>) -> Result<TreeNodeRecursion>,
+            ) -> Result<TreeNodeRecursion> {
+                Ok(TreeNodeRecursion::Continue)
+            }
+        }
+
+        fn schema() -> SchemaRef {
+            Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, false)]))
+        }
+
+        /// A scan of `files` files of `file_size` bytes each, one file in each
+        /// partition, with `num_rows` rows in total.
+        fn scan_with(
+            files: usize,
+            file_size: u64,
+            num_rows: usize,
+            pruning_filters: bool,
+        ) -> FileScanConfig {
+            let schema = schema();
+            let source = FilteringSource {
+                metrics: ExecutionPlanMetricsSet::new(),
+                table_schema: TableSchema::from(&schema),
+                pruning_filters,
+                filter: None,
+                pruning_only: false,
+            };
+            let files = (0..files)
+                .map(|i| {
+                    FileGroup::new(vec![PartitionedFile::new(
+                        format!("f{i}.parquet"),
+                        file_size,
+                    )])
+                })
+                .collect::<Vec<_>>();
+            FileScanConfigBuilder::new(
+                ObjectStoreUrl::local_filesystem(),
+                Arc::new(source),
+            )
+            .with_file_groups(files)
+            .with_statistics(
+                Statistics::new_unknown(&schema)
+                    .with_num_rows(Precision::Exact(num_rows)),
+            )
+            .build()
+        }
+
+        fn scan(files: usize, file_size: u64, num_rows: usize) -> FileScanConfig {
+            scan_with(files, file_size, num_rows, true)
+        }
+
+        fn config(target_partitions: usize) -> ConfigOptions {
+            let mut config = ConfigOptions::default();
+            config.execution.target_partitions = target_partitions;
+            config
+        }
+
+        fn a_gt_5() -> Arc<dyn PhysicalExpr> {
+            binary(
+                col("a", &schema()).unwrap(),
+                Operator::Gt,
+                lit(5),
+                &schema(),
+            )
+            .unwrap()
+        }
+
+        /// Pushes `a > 5` into `scan`. Returns if the scan applies it, and
+        /// if the new file source uses it for pruning only.
+        fn push(scan: &FileScanConfig, config: &ConfigOptions) -> (PushedDown, bool) {
+            push_filter(scan, a_gt_5(), config)
+        }
+
+        fn push_filter(
+            scan: &FileScanConfig,
+            filter: Arc<dyn PhysicalExpr>,
+            config: &ConfigOptions,
+        ) -> (PushedDown, bool) {
+            let expected = filter.to_string();
+            let result = scan.try_pushdown_filters(vec![filter], config).unwrap();
+            let node = result.updated_node.expect("the source takes the filter");
+            let node = node.downcast_ref::<FileScanConfig>().unwrap();
+            let source = node
+                .file_source
+                .as_ref()
+                .downcast_ref::<FilteringSource>()
+                .unwrap();
+            assert_eq!(source.filter.as_ref().unwrap().to_string(), expected);
+            (result.filters[0], source.pruning_only)
+        }
+
+        #[test]
+        fn filter_stays_above_a_scan_that_cannot_split() {
+            // One small file: the scan has one partition and cannot split
+            // the file, thus a filter above the scan runs in 4 partitions.
+            let (pushed_down, pruning_only) = push(&scan(1, 1024, 100_000), &config(4));
+            assert!(matches!(pushed_down, PushedDown::No));
+            assert!(pruning_only);
+        }
+
+        #[test]
+        fn scan_applies_the_filter_when_it_has_the_partitions() {
+            // One file for each target partition.
+            let (pushed_down, pruning_only) = push(&scan(4, 1024, 100_000), &config(4));
+            assert!(matches!(pushed_down, PushedDown::Yes));
+            assert!(!pruning_only);
+
+            // One large file that the scan splits into 4 byte ranges.
+            let (pushed_down, _) = push(&scan(1, 1 << 30, 100_000), &config(4));
+            assert!(matches!(pushed_down, PushedDown::Yes));
+
+            // One target partition.
+            let (pushed_down, _) = push(&scan(1, 1024, 100_000), &config(1));
+            assert!(matches!(pushed_down, PushedDown::Yes));
+        }
+
+        #[test]
+        fn scan_applies_the_filter_when_a_round_robin_does_not_help() {
+            // The scan reads at most one batch: a round-robin repartition
+            // cannot split the work.
+            let (pushed_down, _) = push(&scan(1, 1024, 8192), &config(4));
+            assert!(matches!(pushed_down, PushedDown::Yes));
+            let (pushed_down, _) = push(&scan(1, 1024, 0), &config(4));
+            assert!(matches!(pushed_down, PushedDown::Yes));
+
+            // Round-robin repartitions are off.
+            let mut config = config(4);
+            config.optimizer.enable_round_robin_repartition = false;
+            let (pushed_down, _) = push(&scan(1, 1024, 100_000), &config);
+            assert!(matches!(pushed_down, PushedDown::Yes));
+        }
+
+        #[test]
+        fn dynamic_filter_goes_into_the_scan() {
+            // A dynamic filter has no `FilterExec` above the scan, thus the
+            // scan applies it.
+            use datafusion_physical_expr::expressions::DynamicFilterPhysicalExpr;
+            let dynamic: Arc<dyn PhysicalExpr> =
+                Arc::new(DynamicFilterPhysicalExpr::new(
+                    vec![col("a", &schema()).unwrap()],
+                    a_gt_5(),
+                ));
+            let (pushed_down, pruning_only) =
+                push_filter(&scan(1, 1024, 100_000), dynamic, &config(4));
+            assert!(matches!(pushed_down, PushedDown::Yes));
+            assert!(!pruning_only);
+        }
+
+        #[test]
+        fn source_without_pruning_filters_applies_the_filter() {
+            // The source does not take filters for pruning only: the scan
+            // pushes them as before.
+            let (pushed_down, pruning_only) =
+                push(&scan_with(1, 1024, 100_000, false), &config(4));
+            assert!(matches!(pushed_down, PushedDown::Yes));
+            assert!(!pruning_only);
+        }
+    }
 }
diff --git a/datafusion/proto-models/proto/datafusion.proto b/datafusion/proto-models/proto/datafusion.proto
index 10acc53ccb2b3..ccd830ef74eea 100644
--- a/datafusion/proto-models/proto/datafusion.proto
+++ b/datafusion/proto-models/proto/datafusion.proto
@@ -1331,6 +1331,10 @@ message ParquetScanExecNode {
 
   // Source-specific footer prefetch size. Absent means no hint.
   optional uint64 metadata_size_hint = 7;
+
+  // If true, the scan uses `predicate` only to prune: a filter above the
+  // scan applies it.
+  bool pruning_only_predicate = 8;
 }
 
 message CsvScanExecNode {
diff --git a/datafusion/proto-models/src/generated/pbjson.rs b/datafusion/proto-models/src/generated/pbjson.rs
index e6e465cb0aa3c..4779760866fc6 100644
--- a/datafusion/proto-models/src/generated/pbjson.rs
+++ b/datafusion/proto-models/src/generated/pbjson.rs
@@ -16785,6 +16785,9 @@ impl serde::Serialize for ParquetScanExecNode {
         if self.metadata_size_hint.is_some() {
             len += 1;
         }
+        if self.pruning_only_predicate {
+            len += 1;
+        }
         let mut struct_ser = serializer.serialize_struct("datafusion.ParquetScanExecNode", len)?;
         if let Some(v) = self.base_conf.as_ref() {
             struct_ser.serialize_field("baseConf", v)?;
@@ -16806,6 +16809,9 @@ impl serde::Serialize for ParquetScanExecNode {
             #[allow(clippy::needless_borrows_for_generic_args)]
             struct_ser.serialize_field("metadataSizeHint", ToString::to_string(&v).as_str())?;
         }
+        if self.pruning_only_predicate {
+            struct_ser.serialize_field("pruningOnlyPredicate", &self.pruning_only_predicate)?;
+        }
         struct_ser.end()
     }
 }
@@ -16827,6 +16833,8 @@ impl<'de> serde::Deserialize<'de> for ParquetScanExecNode {
             "reverseRowGroups",
             "metadata_size_hint",
             "metadataSizeHint",
+            "pruning_only_predicate",
+            "pruningOnlyPredicate",
         ];
 
         #[allow(clippy::enum_variant_names)]
@@ -16837,6 +16845,7 @@ impl<'de> serde::Deserialize<'de> for ParquetScanExecNode {
             SortOrderForReorder,
             ReverseRowGroups,
             MetadataSizeHint,
+            PruningOnlyPredicate,
         }
         impl<'de> serde::Deserialize<'de> for GeneratedField {
             fn deserialize<D>(deserializer: D) -> std::result::Result<GeneratedField, D::Error>
@@ -16864,6 +16873,7 @@ impl<'de> serde::Deserialize<'de> for ParquetScanExecNode {
                             "sortOrderForReorder" | "sort_order_for_reorder" => Ok(GeneratedField::SortOrderForReorder),
                             "reverseRowGroups" | "reverse_row_groups" => Ok(GeneratedField::ReverseRowGroups),
                             "metadataSizeHint" | "metadata_size_hint" => Ok(GeneratedField::MetadataSizeHint),
+                            "pruningOnlyPredicate" | "pruning_only_predicate" => Ok(GeneratedField::PruningOnlyPredicate),
                             _ => Err(serde::de::Error::unknown_field(value, FIELDS)),
                         }
                     }
@@ -16889,6 +16899,7 @@ impl<'de> serde::Deserialize<'de> for ParquetScanExecNode {
                 let mut sort_order_for_reorder__ = None;
                 let mut reverse_row_groups__ = None;
                 let mut metadata_size_hint__ = None;
+                let mut pruning_only_predicate__ = None;
                 while let Some(k) = map_.next_key()? {
                     match k {
                         GeneratedField::BaseConf => {
@@ -16929,6 +16940,12 @@ impl<'de> serde::Deserialize<'de> for ParquetScanExecNode {
                                 map_.next_value::<::std::option::Option<::pbjson::private::NumberDeserialize<_>>>()?.map(|x| x.0)
                             ;
                         }
+                        GeneratedField::PruningOnlyPredicate => {
+                            if pruning_only_predicate__.is_some() {
+                                return Err(serde::de::Error::duplicate_field("pruningOnlyPredicate"));
+                            }
+                            pruning_only_predicate__ = Some(map_.next_value()?);
+                        }
                     }
                 }
                 Ok(ParquetScanExecNode {
@@ -16938,6 +16955,7 @@ impl<'de> serde::Deserialize<'de> for ParquetScanExecNode {
                     sort_order_for_reorder: sort_order_for_reorder__,
                     reverse_row_groups: reverse_row_groups__.unwrap_or_default(),
                     metadata_size_hint: metadata_size_hint__,
+                    pruning_only_predicate: pruning_only_predicate__.unwrap_or_default(),
                 })
             }
         }
diff --git a/datafusion/proto-models/src/generated/prost.rs b/datafusion/proto-models/src/generated/prost.rs
index 2480d26e46a7f..5c4e4ce4de200 100644
--- a/datafusion/proto-models/src/generated/prost.rs
+++ b/datafusion/proto-models/src/generated/prost.rs
@@ -2035,6 +2035,10 @@ pub struct ParquetScanExecNode {
     /// Source-specific footer prefetch size. Absent means no hint.
     #[prost(uint64, optional, tag = "7")]
     pub metadata_size_hint: ::core::option::Option<u64>,
+    /// If true, the scan uses `predicate` only to prune: a filter above the
+    /// scan applies it.
+    #[prost(bool, tag = "8")]
+    pub pruning_only_predicate: bool,
 }
 #[derive(Clone, PartialEq, ::prost::Message)]
 pub struct CsvScanExecNode {
diff --git a/datafusion/proto/tests/cases/plans/sources.rs b/datafusion/proto/tests/cases/plans/sources.rs
index c4897357ee00b..d96f688c27941 100644
--- a/datafusion/proto/tests/cases/plans/sources.rs
+++ b/datafusion/proto/tests/cases/plans/sources.rs
@@ -166,6 +166,49 @@ fn roundtrip_parquet_exec_with_pruning_predicate() -> Result<()> {
     Ok(())
 }
 
+/// A predicate that the scan uses only to prune stays pruning-only after a
+/// round trip: the decoded scan does not apply it.
+#[test]
+fn roundtrip_parquet_exec_with_pruning_only_predicate() -> Result<()> {
+    use datafusion::datasource::physical_plan::FileSource;
+
+    let file_schema =
+        Arc::new(Schema::new(vec![Field::new("col", DataType::Utf8, false)]));
+    let predicate: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
+        Arc::new(Column::new("col", 0)),
+        Operator::Eq,
+        lit("1"),
+    ));
+    let file_source = ParquetSource::new(Arc::clone(&file_schema))
+        .try_pushdown_pruning_filters(&[predicate], &Default::default())?
+        .expect("the source takes filters for pruning only");
+    let scan_config =
+        FileScanConfigBuilder::new(ObjectStoreUrl::local_filesystem(), file_source)
+            .with_file_groups(vec![FileGroup::new(vec![PartitionedFile::new(
+                "/path/to/file.parquet".to_string(),
+                1024,
+            )])])
+            .build();
+
+    let ctx = SessionContext::new();
+    let codec = DefaultPhysicalExtensionCodec {};
+    let roundtripped = roundtrip_test_and_return(
+        DataSourceExec::from_data_source(scan_config),
+        &ctx,
+        &codec,
+        &DefaultPhysicalProtoConverter {},
+    )?;
+    let node = PhysicalPlanNode::try_from_physical_plan(roundtripped, &codec)?;
+    let Some(protobuf::physical_plan_node::PhysicalPlanType::ParquetScan(scan)) =
+        node.physical_plan_type
+    else {
+        return internal_err!("Expected ParquetScan node");
+    };
+    assert!(scan.predicate.is_some());
+    assert!(scan.pruning_only_predicate);
+    Ok(())
+}
+
 #[tokio::test]
 async fn roundtrip_parquet_exec_with_sort_pushdown() -> Result<()> {
     let ctx = all_types_context().await?;
diff --git a/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt b/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt
index 224640cab0997..ba2163dc06710 100644
--- a/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt
+++ b/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt
@@ -1042,3 +1042,72 @@ set datafusion.execution.parquet.reorder_filters = false;
 
 statement ok
 DROP TABLE dict_filter_bug;
+
+
+###
+# A filter stays above a scan that cannot give the target partitions.
+#
+# The filter in a scan runs in the partitions of the scan. One small file
+# gives one partition. A `FilterExec` above a round-robin repartition runs in
+# `target_partitions` partitions, thus the filter stays above the scan. The
+# scan uses it only for pruning.
+###
+
+statement ok
+set datafusion.execution.parquet.pushdown_filters = true;
+
+statement ok
+set datafusion.execution.target_partitions = 4;
+
+statement ok
+COPY (SELECT value AS a, value % 7 AS b FROM generate_series(1, 20000))
+TO 'test_files/scratch/parquet_filter_pushdown/one_file/data.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE one_file STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_filter_pushdown/one_file/';
+
+query TT
+EXPLAIN SELECT a FROM one_file WHERE b = 3;
+----
+logical_plan
+01)Projection: one_file.a
+02)--Filter: one_file.b = Int64(3)
+03)----TableScan: one_file projection=[a, b], partial_filters=[one_file.b = Int64(3)]
+physical_plan
+01)FilterExec: b@1 = 3, projection=[a@0]
+02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, maintains_sort_order=true
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/one_file/data.parquet]]}, projection=[a, b], output_ordering=[a@0 ASC NULLS LAST], file_type=parquet, predicate=b@1 = 3, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 3 AND 3 <= b_max@1, required_guarantees=[b in (3)]
+
+query I
+SELECT count(*) FROM one_file WHERE b = 3;
+----
+2857
+
+# With one target partition, the filter above cannot run in more partitions:
+# the scan applies it.
+statement ok
+set datafusion.execution.target_partitions = 1;
+
+query TT
+EXPLAIN SELECT a FROM one_file WHERE b = 3;
+----
+logical_plan
+01)Projection: one_file.a
+02)--Filter: one_file.b = Int64(3)
+03)----TableScan: one_file projection=[a, b], partial_filters=[one_file.b = Int64(3)]
+physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/one_file/data.parquet]]}, projection=[a], output_ordering=[a@0 ASC NULLS LAST], file_type=parquet, predicate=b@1 = 3, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 3 AND 3 <= b_max@1, required_guarantees=[b in (3)]
+
+query I
+SELECT count(*) FROM one_file WHERE b = 3;
+----
+2857
+
+statement ok
+DROP TABLE one_file;
+
+statement ok
+set datafusion.execution.target_partitions = 4;
+
+statement ok
+set datafusion.execution.parquet.pushdown_filters = false;
diff --git a/datafusion/sqllogictest/test_files/parquet_statistics.slt b/datafusion/sqllogictest/test_files/parquet_statistics.slt
index 1eea640a17cb0..2d1c0ba6b4932 100644
--- a/datafusion/sqllogictest/test_files/parquet_statistics.slt
+++ b/datafusion/sqllogictest/test_files/parquet_statistics.slt
@@ -102,7 +102,10 @@ LOCATION 'test_files/scratch/parquet_statistics/test_table';
 query TT
 EXPLAIN SELECT * FROM test_table WHERE column1 = 1;
 ----
-physical_plan DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/1.parquet]]}, projection=[column1], file_type=parquet, predicate=column1@0 = 1, pruning_predicate=column1_null_count@2 != row_count@3 AND column1_min@0 <= 1 AND 1 <= column1_max@1, required_guarantees=[column1 in (1)], statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:)]]
+physical_plan
+01)FilterExec: column1@0 = 1, statistics=[Rows=Absent, Bytes=Absent, [(Col[0]: Min=Exact(Int64(1)) Max=Exact(Int64(1)) Null=Exact(0) Distinct=Inexact(1))]]
+02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2, statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:)]]
+03)----DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/0.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_statistics/test_table/1.parquet]]}, projection=[column1], file_type=parquet, predicate=column1@0 = 1, pruning_predicate=column1_null_count@2 != row_count@3 AND column1_min@0 <= 1 AND 1 <= column1_max@1, required_guarantees=[column1 in (1)], statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:)]]
 
 # cleanup
 statement ok

From fe6aa04149bbcc3dbf9bd5d433e09e8a9092e2ae Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Wed, 23 Sep 2026 20:05:21 -0500
Subject: [PATCH 06/42] feat: add OptionalFilterPhysicalExpr to mark filters
 not needed for correctness

Add `OptionalFilterPhysicalExpr`, a transparent wrapper that marks a filter
as optional: a consumer can skip it without changing the query result. A
consumer can skip it only when the wrapper is a direct conjunct of the root
AND chain of its predicate. In all other positions the wrapper is
transparent, because `evaluate()` always evaluates the inner expression.
`snapshot()` returns the inner expression, so pruning sees through it.

Also add:
- `split_optional` and `is_optional_filter` helpers in
  `physical_expr::utils` for consumers
- `PhysicalOptionalFilterNode` proto message (field 29 in
  `PhysicalExprNode`) with self-encoding `try_to_proto`/`try_from_proto`

No producer uses the wrapper yet, so there is no behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../physical-expr/src/expressions/mod.rs      |   2 +
 .../src/expressions/optional_filter.rs        | 357 ++++++++++++++++++
 .../physical-expr/src/simplifier/mod.rs       |  32 +-
 datafusion/physical-expr/src/utils/mod.rs     | 136 ++++++-
 .../proto-models/proto/datafusion.proto       |   7 +
 .../proto-models/src/generated/pbjson.rs      | 105 ++++++
 .../proto-models/src/generated/prost.rs       |  10 +-
 .../proto/src/physical_plan/from_proto.rs     |   7 +-
 .../tests/cases/plans/dynamic_filters.rs      |  65 +++-
 datafusion/pruning/src/pruning_predicate.rs   |  41 ++
 10 files changed, 757 insertions(+), 5 deletions(-)
 create mode 100644 datafusion/physical-expr/src/expressions/optional_filter.rs

diff --git a/datafusion/physical-expr/src/expressions/mod.rs b/datafusion/physical-expr/src/expressions/mod.rs
index f2f9285de560a..d36f734afef39 100644
--- a/datafusion/physical-expr/src/expressions/mod.rs
+++ b/datafusion/physical-expr/src/expressions/mod.rs
@@ -33,6 +33,7 @@ mod literal;
 mod negative;
 mod no_op;
 mod not;
+mod optional_filter;
 mod similar_to_pattern;
 mod try_cast;
 mod unknown_column;
@@ -59,6 +60,7 @@ pub use literal::{Literal, lit};
 pub use negative::{NegativeExpr, negative};
 pub use no_op::NoOp;
 pub use not::{NotExpr, not};
+pub use optional_filter::OptionalFilterPhysicalExpr;
 pub(crate) use similar_to_pattern::translate_scalar;
 pub use similar_to_pattern::{SqlSimilarToPattern, sql_similar_to_regex};
 pub use try_cast::{TryCastExpr, try_cast, try_cast_with_target_field};
diff --git a/datafusion/physical-expr/src/expressions/optional_filter.rs b/datafusion/physical-expr/src/expressions/optional_filter.rs
new file mode 100644
index 0000000000000..5561f90e89345
--- /dev/null
+++ b/datafusion/physical-expr/src/expressions/optional_filter.rs
@@ -0,0 +1,357 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`OptionalFilterPhysicalExpr`]: a marker for filters that are not needed
+//! for correctness. See the type documentation for the contract that
+//! producers, consumers and rewriters must obey.
+
+use std::fmt;
+use std::hash::Hash;
+use std::sync::Arc;
+
+use crate::PhysicalExpr;
+
+use arrow::array::BooleanArray;
+use arrow::datatypes::{DataType, FieldRef, Schema};
+use arrow::record_batch::RecordBatch;
+use datafusion_common::{Result, assert_eq_or_internal_err};
+use datafusion_expr::ColumnarValue;
+use datafusion_expr::interval_arithmetic::Interval;
+use datafusion_expr::sort_properties::ExprProperties;
+
+/// Marks the inner filter as *optional*: it is not needed for correctness.
+///
+/// Some filters are only performance hints. For example, a hash join can push
+/// a dynamic filter into the probe side scan, but the join itself still
+/// removes the rows that do not match. Such a filter is *optional*: a
+/// consumer can skip it (for example, when the filter does not remove enough
+/// rows to be worth its cost) and the query result stays the same.
+///
+/// # Contract
+///
+/// * **Skip only on the root AND chain.** A consumer can skip an optional
+///   filter only when the `Optional` node is a direct conjunct of the root
+///   `AND` chain of its predicate. For example, in `a AND Optional(b)` the
+///   consumer can skip `b`. Use [`split_optional`] to find these conjuncts.
+/// * **Transparent everywhere else.** [`PhysicalExpr::evaluate`] always
+///   evaluates the inner expression. Thus an `Optional` in a different
+///   position (for example under `NOT`, `IS NULL`, `CASE` or `OR`) can make a
+///   query slower, but it cannot make the result incorrect.
+/// * **Rewriters must not move nodes across the wrapper.** A rewrite must not
+///   move an expression into or out of an `Optional`. For example,
+///   `NOT(Optional(x))` must not become `Optional(NOT(x))`, because that
+///   would make a required filter optional.
+/// * **Pruning sees through the wrapper.** [`PhysicalExpr::snapshot`]
+///   returns the inner expression, so [`snapshot_physical_expr`] removes the
+///   wrapper. Thus statistics pruning uses an optional filter the same as a
+///   required filter.
+///
+/// [`split_optional`]: crate::utils::split_optional
+/// [`snapshot_physical_expr`]: datafusion_physical_expr_common::physical_expr::snapshot_physical_expr
+#[derive(Debug, Eq)]
+pub struct OptionalFilterPhysicalExpr {
+    inner: Arc<dyn PhysicalExpr>,
+}
+
+// Manually derive PartialEq and Hash to work around https://github.com/rust-lang/rust/issues/78808
+impl PartialEq for OptionalFilterPhysicalExpr {
+    fn eq(&self, other: &Self) -> bool {
+        self.inner.eq(&other.inner)
+    }
+}
+
+impl Hash for OptionalFilterPhysicalExpr {
+    fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+        self.inner.hash(state);
+    }
+}
+
+impl OptionalFilterPhysicalExpr {
+    /// Create a new optional filter that wraps `inner`.
+    pub fn new(inner: Arc<dyn PhysicalExpr>) -> Self {
+        Self { inner }
+    }
+
+    /// Get the wrapped filter expression.
+    pub fn inner(&self) -> &Arc<dyn PhysicalExpr> {
+        &self.inner
+    }
+}
+
+impl fmt::Display for OptionalFilterPhysicalExpr {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "Optional({})", self.inner)
+    }
+}
+
+impl PhysicalExpr for OptionalFilterPhysicalExpr {
+    fn data_type(&self, input_schema: &Schema) -> Result<DataType> {
+        self.inner.data_type(input_schema)
+    }
+
+    fn nullable(&self, input_schema: &Schema) -> Result<bool> {
+        self.inner.nullable(input_schema)
+    }
+
+    fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+        self.inner.evaluate(batch)
+    }
+
+    fn return_field(&self, input_schema: &Schema) -> Result<FieldRef> {
+        self.inner.return_field(input_schema)
+    }
+
+    fn evaluate_selection(
+        &self,
+        batch: &RecordBatch,
+        selection: &BooleanArray,
+    ) -> Result<ColumnarValue> {
+        self.inner.evaluate_selection(batch, selection)
+    }
+
+    fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+        vec![&self.inner]
+    }
+
+    fn with_new_children(
+        self: Arc<Self>,
+        children: Vec<Arc<dyn PhysicalExpr>>,
+    ) -> Result<Arc<dyn PhysicalExpr>> {
+        assert_eq_or_internal_err!(
+            children.len(),
+            1,
+            "OptionalFilterPhysicalExpr: expected 1 child"
+        );
+        Ok(Arc::new(Self::new(Arc::clone(&children[0]))))
+    }
+
+    // The wrapper is the identity function, so the bounds and properties of
+    // the child are also the bounds and properties of the wrapper.
+    fn evaluate_bounds(&self, children: &[&Interval]) -> Result<Interval> {
+        Ok(children[0].clone())
+    }
+
+    fn propagate_constraints(
+        &self,
+        interval: &Interval,
+        children: &[&Interval],
+    ) -> Result<Option<Vec<Interval>>> {
+        Ok(children[0].intersect(interval)?.map(|result| vec![result]))
+    }
+
+    fn get_properties(&self, children: &[ExprProperties]) -> Result<ExprProperties> {
+        Ok(children[0].clone())
+    }
+
+    fn fmt_sql(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        self.inner.fmt_sql(f)
+    }
+
+    /// Returns the inner expression, so that snapshot consumers (for example
+    /// pruning) see through the wrapper.
+    ///
+    /// [`snapshot_physical_expr`] transforms the tree bottom up, so the inner
+    /// expression is already a snapshot when this method is called. For
+    /// example, `Optional(DynamicFilter)` becomes the current expression of
+    /// the dynamic filter.
+    ///
+    /// [`snapshot_physical_expr`]: datafusion_physical_expr_common::physical_expr::snapshot_physical_expr
+    fn snapshot(&self) -> Result<Option<Arc<dyn PhysicalExpr>>> {
+        Ok(Some(Arc::clone(&self.inner)))
+    }
+
+    fn snapshot_generation(&self) -> u64 {
+        // The wrapper is not dynamic. `snapshot_generation(expr)` walks the
+        // tree and adds the generation of the inner expression.
+        0
+    }
+
+    #[cfg(feature = "proto")]
+    fn try_to_proto(
+        &self,
+        ctx: &datafusion_physical_expr_common::physical_expr::proto_encode::PhysicalExprEncodeCtx<'_>,
+    ) -> Result<Option<datafusion_proto_models::protobuf::PhysicalExprNode>> {
+        use datafusion_proto_models::protobuf;
+
+        Ok(Some(protobuf::PhysicalExprNode {
+            expr_id: None,
+            expr_type: Some(protobuf::physical_expr_node::ExprType::OptionalFilter(
+                Box::new(protobuf::PhysicalOptionalFilterNode {
+                    inner: Some(Box::new(ctx.encode_child(&self.inner)?)),
+                }),
+            )),
+        }))
+    }
+}
+
+#[cfg(feature = "proto")]
+impl OptionalFilterPhysicalExpr {
+    /// Reconstruct an [`OptionalFilterPhysicalExpr`] from its protobuf
+    /// representation.
+    pub fn try_from_proto(
+        node: &datafusion_proto_models::protobuf::PhysicalExprNode,
+        ctx: &datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx<'_>,
+    ) -> Result<Arc<dyn PhysicalExpr>> {
+        use datafusion_physical_expr_common::expect_expr_variant;
+        use datafusion_proto_models::protobuf;
+
+        let optional = expect_expr_variant!(
+            node,
+            protobuf::physical_expr_node::ExprType::OptionalFilter,
+            "OptionalFilter",
+        );
+        let inner = ctx.decode_required_expression(
+            optional.inner.as_deref(),
+            "OptionalFilter",
+            "inner",
+        )?;
+
+        Ok(Arc::new(Self::new(inner)))
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::expressions::{BinaryExpr, DynamicFilterPhysicalExpr, col, lit, not};
+
+    use arrow::array::{ArrayRef, Int32Array};
+    use arrow::datatypes::Field;
+    use datafusion_common::cast::as_boolean_array;
+    use datafusion_expr::Operator;
+    use datafusion_physical_expr_common::physical_expr::{
+        fmt_sql, snapshot_generation, snapshot_physical_expr,
+    };
+
+    fn schema() -> Arc<Schema> {
+        Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, true)]))
+    }
+
+    /// `a > 2`
+    fn a_gt_2(schema: &Schema) -> Arc<dyn PhysicalExpr> {
+        Arc::new(BinaryExpr::new(
+            col("a", schema).unwrap(),
+            Operator::Gt,
+            lit(2i32),
+        ))
+    }
+
+    fn optional(inner: Arc<dyn PhysicalExpr>) -> Arc<dyn PhysicalExpr> {
+        Arc::new(OptionalFilterPhysicalExpr::new(inner))
+    }
+
+    #[test]
+    fn evaluate_equals_inner() -> Result<()> {
+        let schema = schema();
+        let values: ArrayRef = Arc::new(Int32Array::from(vec![Some(1), None, Some(3)]));
+        let batch = RecordBatch::try_new(Arc::clone(&schema), vec![values])?;
+
+        let inner = a_gt_2(&schema);
+        let wrapped = optional(Arc::clone(&inner));
+
+        let expected = inner.evaluate(&batch)?.into_array(batch.num_rows())?;
+        let actual = wrapped.evaluate(&batch)?.into_array(batch.num_rows())?;
+        assert_eq!(as_boolean_array(&actual)?, as_boolean_array(&expected)?);
+
+        let selection = BooleanArray::from(vec![true, false, true]);
+        let expected = inner
+            .evaluate_selection(&batch, &selection)?
+            .into_array(batch.num_rows())?;
+        let actual = wrapped
+            .evaluate_selection(&batch, &selection)?
+            .into_array(batch.num_rows())?;
+        assert_eq!(as_boolean_array(&actual)?, as_boolean_array(&expected)?);
+
+        assert_eq!(wrapped.data_type(&schema)?, DataType::Boolean);
+        assert!(wrapped.nullable(&schema)?);
+        Ok(())
+    }
+
+    #[test]
+    fn display_and_fmt_sql() {
+        let schema = schema();
+        let wrapped = optional(a_gt_2(&schema));
+        assert_eq!(wrapped.to_string(), "Optional(a@0 > 2)");
+        assert_eq!(fmt_sql(wrapped.as_ref()).to_string(), "a > 2");
+
+        let negated = not(Arc::clone(&wrapped)).unwrap();
+        assert_eq!(negated.to_string(), "NOT Optional(a@0 > 2)");
+    }
+
+    #[test]
+    fn children_and_with_new_children() -> Result<()> {
+        let schema = schema();
+        let wrapped = optional(a_gt_2(&schema));
+        assert_eq!(wrapped.children().len(), 1);
+
+        let new_inner = lit(true);
+        let rewrapped = Arc::clone(&wrapped).with_new_children(vec![new_inner])?;
+        let rewrapped = rewrapped
+            .downcast_ref::<OptionalFilterPhysicalExpr>()
+            .expect("wrapper is kept");
+        assert_eq!(rewrapped.inner().to_string(), "true");
+
+        assert!(wrapped.with_new_children(vec![]).is_err());
+        Ok(())
+    }
+
+    #[test]
+    fn eq_and_hash_use_inner() {
+        use std::collections::HashSet;
+
+        let schema = schema();
+        let a = optional(a_gt_2(&schema));
+        let b = optional(a_gt_2(&schema));
+        let c = optional(lit(true));
+        assert_eq!(&a, &b);
+        assert_ne!(&a, &c);
+        // The wrapper is not equal to the inner expression.
+        assert_ne!(&a, &a_gt_2(&schema));
+
+        let set: HashSet<_> = [a, b, c].into_iter().collect();
+        assert_eq!(set.len(), 2);
+    }
+
+    #[test]
+    fn snapshot_sees_through_dynamic_filter() -> Result<()> {
+        let schema = schema();
+        let dynamic = Arc::new(DynamicFilterPhysicalExpr::new(
+            vec![col("a", &schema)?],
+            lit(true),
+        ));
+        let wrapped = optional(Arc::clone(&dynamic) as Arc<dyn PhysicalExpr>);
+
+        assert_eq!(
+            snapshot_physical_expr(Arc::clone(&wrapped))?.to_string(),
+            "true"
+        );
+
+        let generation = snapshot_generation(&wrapped);
+        dynamic.update(a_gt_2(&schema))?;
+        assert_ne!(snapshot_generation(&wrapped), generation);
+        assert_eq!(
+            snapshot_physical_expr(Arc::clone(&wrapped))?.to_string(),
+            "a@0 > 2"
+        );
+
+        // A static inner expression is also unwrapped.
+        let wrapped = optional(a_gt_2(&schema));
+        assert_eq!(wrapped.snapshot_generation(), 0);
+        assert_eq!(snapshot_physical_expr(wrapped)?.to_string(), "a@0 > 2");
+        Ok(())
+    }
+}
diff --git a/datafusion/physical-expr/src/simplifier/mod.rs b/datafusion/physical-expr/src/simplifier/mod.rs
index af87ce7f61d35..18fd29709510e 100644
--- a/datafusion/physical-expr/src/simplifier/mod.rs
+++ b/datafusion/physical-expr/src/simplifier/mod.rs
@@ -96,7 +96,8 @@ mod tests {
     use super::*;
     use crate::ScalarFunctionExpr;
     use crate::expressions::{
-        BinaryExpr, CastExpr, Literal, NotExpr, TryCastExpr, col, in_list, lit,
+        BinaryExpr, CastExpr, Literal, NotExpr, OptionalFilterPhysicalExpr, TryCastExpr,
+        col, in_list, lit,
     };
     use arrow::datatypes::{DataType, Field};
     use datafusion_common::ScalarValue;
@@ -238,6 +239,35 @@ mod tests {
         Ok(())
     }
 
+    #[test]
+    fn test_not_optional_filter_unchanged() -> Result<()> {
+        let schema = not_test_schema();
+        let simplifier = PhysicalExprSimplifier::new(&schema);
+        let optional = |e: Arc<dyn PhysicalExpr>| -> Arc<dyn PhysicalExpr> {
+            Arc::new(OptionalFilterPhysicalExpr::new(e))
+        };
+
+        // NOT(Optional(c > 5)) is unchanged: the NOT must not move into the
+        // Optional, because NOT(c > 5) is a required filter.
+        let c_gt_5: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
+            col("c", &schema)?,
+            Operator::Gt,
+            lit(ScalarValue::Int32(Some(5))),
+        ));
+        let expr: Arc<dyn PhysicalExpr> =
+            Arc::new(NotExpr::new(optional(Arc::clone(&c_gt_5))));
+        assert_not_simplify(&simplifier, Arc::clone(&expr), expr);
+
+        // NOT(Optional(NOT(a))) is unchanged: no double negation elimination
+        // across the Optional.
+        let expr: Arc<dyn PhysicalExpr> = Arc::new(NotExpr::new(optional(Arc::new(
+            NotExpr::new(col("a", &schema)?),
+        ))));
+        assert_not_simplify(&simplifier, Arc::clone(&expr), expr);
+
+        Ok(())
+    }
+
     #[test]
     fn test_not_literal() -> Result<()> {
         let schema = not_test_schema();
diff --git a/datafusion/physical-expr/src/utils/mod.rs b/datafusion/physical-expr/src/utils/mod.rs
index 1be57c9192626..88d483a92d8c5 100644
--- a/datafusion/physical-expr/src/utils/mod.rs
+++ b/datafusion/physical-expr/src/utils/mod.rs
@@ -21,7 +21,7 @@ pub use guarantee::{Guarantee, LiteralGuarantee};
 use std::borrow::Borrow;
 use std::sync::Arc;
 
-use crate::expressions::{BinaryExpr, Column, Literal};
+use crate::expressions::{BinaryExpr, Column, Literal, OptionalFilterPhysicalExpr};
 use crate::tree_node::ExprContext;
 use crate::{
     AcrossPartitions, ConstExpr, EquivalenceProperties, PhysicalExpr, PhysicalSortExpr,
@@ -46,6 +46,42 @@ pub fn split_conjunction(
     split_impl(Operator::And, predicate, vec![])
 }
 
+/// Split the root `AND` chain of `predicate` into required and optional
+/// filters.
+///
+/// Returns `(required, optional)`. `required` holds the conjuncts that must be
+/// applied. `optional` holds the *inner* expressions of the conjuncts that are
+/// an [`OptionalFilterPhysicalExpr`], which a consumer can skip without
+/// affecting correctness.
+///
+/// Only the root `AND` chain is examined (the same conjuncts as
+/// [`split_conjunction`]). An `Optional` in any other position, for example
+/// `NOT(Optional(x))` or `Optional(x) OR y`, stays inside a required conjunct.
+///
+/// For example, `a AND Optional(b) AND (c AND Optional(d))` gives
+/// `([a, c], [b, d])`.
+#[expect(clippy::type_complexity)]
+pub fn split_optional(
+    predicate: &Arc<dyn PhysicalExpr>,
+) -> (Vec<Arc<dyn PhysicalExpr>>, Vec<Arc<dyn PhysicalExpr>>) {
+    let mut required = vec![];
+    let mut optional = vec![];
+    for conjunct in split_conjunction(predicate) {
+        match conjunct.downcast_ref::<OptionalFilterPhysicalExpr>() {
+            Some(opt) => optional.push(Arc::clone(opt.inner())),
+            None => required.push(Arc::clone(conjunct)),
+        }
+    }
+    (required, optional)
+}
+
+/// Returns `true` if `expr` itself is an [`OptionalFilterPhysicalExpr`].
+///
+/// This does not examine the children of `expr`.
+pub fn is_optional_filter(expr: &Arc<dyn PhysicalExpr>) -> bool {
+    expr.is::<OptionalFilterPhysicalExpr>()
+}
+
 impl ConstExpr {
     /// Collects predicate-derived constants from equality conjunctions.
     ///
@@ -644,4 +680,102 @@ pub(crate) mod tests {
 
         Ok(())
     }
+
+    fn optional_test_schema() -> Schema {
+        Schema::new(vec![
+            Field::new("a", DataType::Boolean, true),
+            Field::new("b", DataType::Boolean, true),
+            Field::new("c", DataType::Boolean, true),
+            Field::new("d", DataType::Boolean, true),
+        ])
+    }
+
+    fn optional(inner: Arc<dyn PhysicalExpr>) -> Arc<dyn PhysicalExpr> {
+        Arc::new(OptionalFilterPhysicalExpr::new(inner))
+    }
+
+    fn and(
+        left: Arc<dyn PhysicalExpr>,
+        right: Arc<dyn PhysicalExpr>,
+    ) -> Arc<dyn PhysicalExpr> {
+        Arc::new(BinaryExpr::new(left, Operator::And, right))
+    }
+
+    fn or(
+        left: Arc<dyn PhysicalExpr>,
+        right: Arc<dyn PhysicalExpr>,
+    ) -> Arc<dyn PhysicalExpr> {
+        Arc::new(BinaryExpr::new(left, Operator::Or, right))
+    }
+
+    fn to_strings(exprs: &[Arc<dyn PhysicalExpr>]) -> Vec<String> {
+        exprs.iter().map(|e| e.to_string()).collect()
+    }
+
+    #[test]
+    fn test_split_optional_root_chain() -> Result<()> {
+        let schema = optional_test_schema();
+        let [a, b, c, d] = ["a", "b", "c", "d"].map(|n| col(n, &schema).unwrap());
+
+        // a AND Optional(b) AND (c AND Optional(d))
+        let predicate = and(
+            and(Arc::clone(&a), optional(Arc::clone(&b))),
+            and(Arc::clone(&c), optional(Arc::clone(&d))),
+        );
+        let (required, optional_filters) = split_optional(&predicate);
+        assert_eq!(to_strings(&required), vec!["a@0", "c@2"]);
+        assert_eq!(to_strings(&optional_filters), vec!["b@1", "d@3"]);
+
+        // A single Optional at the root is optional.
+        let predicate = optional(Arc::clone(&a));
+        let (required, optional_filters) = split_optional(&predicate);
+        assert!(required.is_empty());
+        assert_eq!(to_strings(&optional_filters), vec!["a@0"]);
+
+        // An Optional that wraps an AND is one optional filter.
+        let predicate = optional(and(Arc::clone(&a), Arc::clone(&b)));
+        let (required, optional_filters) = split_optional(&predicate);
+        assert!(required.is_empty());
+        assert_eq!(to_strings(&optional_filters), vec!["a@0 AND b@1"]);
+
+        // No Optional: everything is required.
+        let predicate = and(Arc::clone(&a), Arc::clone(&b));
+        let (required, optional_filters) = split_optional(&predicate);
+        assert_eq!(to_strings(&required), vec!["a@0", "b@1"]);
+        assert!(optional_filters.is_empty());
+        Ok(())
+    }
+
+    #[test]
+    fn test_split_optional_not_on_root_chain() -> Result<()> {
+        let schema = optional_test_schema();
+        let [x, y] = ["a", "b"].map(|n| col(n, &schema).unwrap());
+
+        // NOT(Optional(x)) is required
+        let predicate = crate::expressions::not(optional(Arc::clone(&x)))?;
+        let (required, optional_filters) = split_optional(&predicate);
+        assert_eq!(required, vec![Arc::clone(&predicate)]);
+        assert!(optional_filters.is_empty());
+
+        // Optional(x) OR y is required
+        let predicate = or(optional(Arc::clone(&x)), Arc::clone(&y));
+        let (required, optional_filters) = split_optional(&predicate);
+        assert_eq!(required, vec![Arc::clone(&predicate)]);
+        assert!(optional_filters.is_empty());
+        Ok(())
+    }
+
+    #[test]
+    fn test_is_optional_filter() -> Result<()> {
+        let schema = optional_test_schema();
+        let a = col("a", &schema)?;
+
+        assert!(is_optional_filter(&optional(Arc::clone(&a))));
+        assert!(!is_optional_filter(&a));
+        // Only the node itself is examined.
+        assert!(!is_optional_filter(&crate::expressions::not(optional(
+            Arc::clone(&a)
+        ))?));
+        Ok(())
+    }
 }
diff --git a/datafusion/proto-models/proto/datafusion.proto b/datafusion/proto-models/proto/datafusion.proto
index ccd830ef74eea..7fd9529a2ee25 100644
--- a/datafusion/proto-models/proto/datafusion.proto
+++ b/datafusion/proto-models/proto/datafusion.proto
@@ -1093,6 +1093,8 @@ message PhysicalExprNode {
     PhysicalLambdaVariableExprNode lambda_variable = 26;
     PhysicalRangeExprNode range_expr = 27;
     PhysicalSqlSimilarToPatternNode sql_similar_to_pattern = 28;
+
+    PhysicalOptionalFilterNode optional_filter = 29;
   }
 }
 
@@ -1104,6 +1106,11 @@ message PhysicalDynamicFilterNode {
   bool is_complete = 5;
 }
 
+// Marks the wrapped filter as optional: it is not needed for correctness.
+message PhysicalOptionalFilterNode {
+  PhysicalExprNode inner = 1;
+}
+
 message PhysicalSqlSimilarToPatternNode {
   PhysicalExprNode expr = 1;
 }
diff --git a/datafusion/proto-models/src/generated/pbjson.rs b/datafusion/proto-models/src/generated/pbjson.rs
index 4779760866fc6..ef8b74442c71e 100644
--- a/datafusion/proto-models/src/generated/pbjson.rs
+++ b/datafusion/proto-models/src/generated/pbjson.rs
@@ -19567,6 +19567,9 @@ impl serde::Serialize for PhysicalExprNode {
                 physical_expr_node::ExprType::SqlSimilarToPattern(v) => {
                     struct_ser.serialize_field("sqlSimilarToPattern", v)?;
                 }
+                physical_expr_node::ExprType::OptionalFilter(v) => {
+                    struct_ser.serialize_field("optionalFilter", v)?;
+                }
             }
         }
         struct_ser.end()
@@ -19626,6 +19629,8 @@ impl<'de> serde::Deserialize<'de> for PhysicalExprNode {
             "rangeExpr",
             "sql_similar_to_pattern",
             "sqlSimilarToPattern",
+            "optional_filter",
+            "optionalFilter",
         ];
 
         #[allow(clippy::enum_variant_names)]
@@ -19657,6 +19662,7 @@ impl<'de> serde::Deserialize<'de> for PhysicalExprNode {
             LambdaVariable,
             RangeExpr,
             SqlSimilarToPattern,
+            OptionalFilter,
         }
         impl<'de> serde::Deserialize<'de> for GeneratedField {
             fn deserialize<D>(deserializer: D) -> std::result::Result<GeneratedField, D::Error>
@@ -19705,6 +19711,7 @@ impl<'de> serde::Deserialize<'de> for PhysicalExprNode {
                             "lambdaVariable" | "lambda_variable" => Ok(GeneratedField::LambdaVariable),
                             "rangeExpr" | "range_expr" => Ok(GeneratedField::RangeExpr),
                             "sqlSimilarToPattern" | "sql_similar_to_pattern" => Ok(GeneratedField::SqlSimilarToPattern),
+                            "optionalFilter" | "optional_filter" => Ok(GeneratedField::OptionalFilter),
                             _ => Err(serde::de::Error::unknown_field(value, FIELDS)),
                         }
                     }
@@ -19916,6 +19923,13 @@ impl<'de> serde::Deserialize<'de> for PhysicalExprNode {
                                 return Err(serde::de::Error::duplicate_field("sqlSimilarToPattern"));
                             }
                             expr_type__ = map_.next_value::<::std::option::Option<_>>()?.map(physical_expr_node::ExprType::SqlSimilarToPattern)
+;
+                        }
+                        GeneratedField::OptionalFilter => {
+                            if expr_type__.is_some() {
+                                return Err(serde::de::Error::duplicate_field("optionalFilter"));
+                            }
+                            expr_type__ = map_.next_value::<::std::option::Option<_>>()?.map(physical_expr_node::ExprType::OptionalFilter)
 ;
                         }
                     }
@@ -21377,6 +21391,97 @@ impl<'de> serde::Deserialize<'de> for PhysicalNot {
         deserializer.deserialize_struct("datafusion.PhysicalNot", FIELDS, GeneratedVisitor)
     }
 }
+impl serde::Serialize for PhysicalOptionalFilterNode {
+    #[allow(deprecated)]
+    fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+    where
+        S: serde::Serializer,
+    {
+        use serde::ser::SerializeStruct;
+        let mut len = 0;
+        if self.inner.is_some() {
+            len += 1;
+        }
+        let mut struct_ser = serializer.serialize_struct("datafusion.PhysicalOptionalFilterNode", len)?;
+        if let Some(v) = self.inner.as_ref() {
+            struct_ser.serialize_field("inner", v)?;
+        }
+        struct_ser.end()
+    }
+}
+impl<'de> serde::Deserialize<'de> for PhysicalOptionalFilterNode {
+    #[allow(deprecated)]
+    fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        const FIELDS: &[&str] = &[
+            "inner",
+        ];
+
+        #[allow(clippy::enum_variant_names)]
+        enum GeneratedField {
+            Inner,
+        }
+        impl<'de> serde::Deserialize<'de> for GeneratedField {
+            fn deserialize<D>(deserializer: D) -> std::result::Result<GeneratedField, D::Error>
+            where
+                D: serde::Deserializer<'de>,
+            {
+                struct GeneratedVisitor;
+
+                impl serde::de::Visitor<'_> for GeneratedVisitor {
+                    type Value = GeneratedField;
+
+                    fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+                        write!(formatter, "expected one of: {:?}", &FIELDS)
+                    }
+
+                    #[allow(unused_variables)]
+                    fn visit_str<E>(self, value: &str) -> std::result::Result<GeneratedField, E>
+                    where
+                        E: serde::de::Error,
+                    {
+                        match value {
+                            "inner" => Ok(GeneratedField::Inner),
+                            _ => Err(serde::de::Error::unknown_field(value, FIELDS)),
+                        }
+                    }
+                }
+                deserializer.deserialize_identifier(GeneratedVisitor)
+            }
+        }
+        struct GeneratedVisitor;
+        impl<'de> serde::de::Visitor<'de> for GeneratedVisitor {
+            type Value = PhysicalOptionalFilterNode;
+
+            fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+                formatter.write_str("struct datafusion.PhysicalOptionalFilterNode")
+            }
+
+            fn visit_map<V>(self, mut map_: V) -> std::result::Result<PhysicalOptionalFilterNode, V::Error>
+                where
+                    V: serde::de::MapAccess<'de>,
+            {
+                let mut inner__ = None;
+                while let Some(k) = map_.next_key()? {
+                    match k {
+                        GeneratedField::Inner => {
+                            if inner__.is_some() {
+                                return Err(serde::de::Error::duplicate_field("inner"));
+                            }
+                            inner__ = map_.next_value()?;
+                        }
+                    }
+                }
+                Ok(PhysicalOptionalFilterNode {
+                    inner: inner__,
+                })
+            }
+        }
+        deserializer.deserialize_struct("datafusion.PhysicalOptionalFilterNode", FIELDS, GeneratedVisitor)
+    }
+}
 impl serde::Serialize for PhysicalPlanNode {
     #[allow(deprecated)]
     fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
diff --git a/datafusion/proto-models/src/generated/prost.rs b/datafusion/proto-models/src/generated/prost.rs
index 5c4e4ce4de200..f2e0d8a56d5f5 100644
--- a/datafusion/proto-models/src/generated/prost.rs
+++ b/datafusion/proto-models/src/generated/prost.rs
@@ -1613,7 +1613,7 @@ pub struct PhysicalExprNode {
     pub expr_id: ::core::option::Option<u64>,
     #[prost(
         oneof = "physical_expr_node::ExprType",
-        tags = "1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28"
+        tags = "1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29"
     )]
     pub expr_type: ::core::option::Option<physical_expr_node::ExprType>,
 }
@@ -1682,6 +1682,8 @@ pub mod physical_expr_node {
         SqlSimilarToPattern(
             ::prost::alloc::boxed::Box<super::PhysicalSqlSimilarToPatternNode>,
         ),
+        #[prost(message, tag = "29")]
+        OptionalFilter(::prost::alloc::boxed::Box<super::PhysicalOptionalFilterNode>),
     }
 }
 #[derive(Clone, PartialEq, ::prost::Message)]
@@ -1697,6 +1699,12 @@ pub struct PhysicalDynamicFilterNode {
     #[prost(bool, tag = "5")]
     pub is_complete: bool,
 }
+/// Marks the wrapped filter as optional: it is not needed for correctness.
+#[derive(Clone, PartialEq, ::prost::Message)]
+pub struct PhysicalOptionalFilterNode {
+    #[prost(message, optional, boxed, tag = "1")]
+    pub inner: ::core::option::Option<::prost::alloc::boxed::Box<PhysicalExprNode>>,
+}
 #[derive(Clone, PartialEq, ::prost::Message)]
 pub struct PhysicalSqlSimilarToPatternNode {
     #[prost(message, optional, boxed, tag = "1")]
diff --git a/datafusion/proto/src/physical_plan/from_proto.rs b/datafusion/proto/src/physical_plan/from_proto.rs
index bc443149df413..91ec5180c22ed 100644
--- a/datafusion/proto/src/physical_plan/from_proto.rs
+++ b/datafusion/proto/src/physical_plan/from_proto.rs
@@ -52,7 +52,9 @@ use super::{
 };
 use crate::protobuf::physical_expr_node::ExprType;
 use crate::{convert_required, protobuf};
-use datafusion_physical_expr::expressions::DynamicFilterPhysicalExpr;
+use datafusion_physical_expr::expressions::{
+    DynamicFilterPhysicalExpr, OptionalFilterPhysicalExpr,
+};
 
 /// Parses a physical sort expression from a protobuf.
 ///
@@ -362,6 +364,9 @@ pub fn parse_physical_expr_with_converter(
         ExprType::DynamicFilter(_) => {
             DynamicFilterPhysicalExpr::try_from_proto(proto, &decode_ctx)?
         }
+        ExprType::OptionalFilter(_) => {
+            OptionalFilterPhysicalExpr::try_from_proto(proto, &decode_ctx)?
+        }
         ExprType::SqlSimilarToPattern(_) => {
             SqlSimilarToPattern::try_from_proto(proto, &decode_ctx)?
         }
diff --git a/datafusion/proto/tests/cases/plans/dynamic_filters.rs b/datafusion/proto/tests/cases/plans/dynamic_filters.rs
index 7892ffd9a1ab0..7d07e59233411 100644
--- a/datafusion/proto/tests/cases/plans/dynamic_filters.rs
+++ b/datafusion/proto/tests/cases/plans/dynamic_filters.rs
@@ -40,7 +40,8 @@ use datafusion::physical_plan::aggregates::{
 };
 use datafusion::physical_plan::empty::EmptyExec;
 use datafusion::physical_plan::expressions::{
-    BinaryExpr, Column, DynamicFilterPhysicalExpr, PhysicalSortExpr, lit,
+    BinaryExpr, Column, DynamicFilterPhysicalExpr, OptionalFilterPhysicalExpr,
+    PhysicalSortExpr, lit,
 };
 use datafusion::physical_plan::filter::FilterExec;
 use datafusion::physical_plan::joins::{HashJoinExec, PartitionMode};
@@ -326,6 +327,68 @@ fn test_dynamic_filter_roundtrip_dedupe() -> Result<()> {
     Ok(())
 }
 
+// An `Optional` wrapper survives the roundtrip, and a dynamic filter inside it
+// is deduped with an unwrapped clone of the same dynamic filter.
+#[test]
+fn test_optional_dynamic_filter_roundtrip_dedupe() -> Result<()> {
+    let schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int64, false)]));
+    let dynamic_filter = make_dynamic_filter();
+    let optional_filter =
+        Arc::new(OptionalFilterPhysicalExpr::new(Arc::clone(&dynamic_filter)))
+            as Arc<dyn PhysicalExpr>;
+
+    let (optional_after_roundtrip, dynamic_after_roundtrip) =
+        roundtrip_dynamic_filter_expr_pair(
+            Arc::clone(&optional_filter),
+            Arc::clone(&dynamic_filter),
+            schema,
+        )?;
+
+    assert_eq!(
+        optional_filter.to_string(),
+        optional_after_roundtrip.to_string()
+    );
+    let inner_after_roundtrip = optional_after_roundtrip
+        .downcast_ref::<OptionalFilterPhysicalExpr>()
+        .expect("Expected OptionalFilterPhysicalExpr")
+        .inner();
+    assert_dynamic_filters_equal(&dynamic_filter, inner_after_roundtrip);
+    assert_dynamic_filters_equal(&dynamic_filter, &dynamic_after_roundtrip);
+
+    // Assert referential integrity through the wrapper.
+    assert_dynamic_filter_update_is_visible(
+        inner_after_roundtrip,
+        &dynamic_after_roundtrip,
+    )?;
+
+    Ok(())
+}
+
+// An `Optional` wrapper around a static expression survives the roundtrip.
+#[test]
+fn test_optional_filter_roundtrip() -> Result<()> {
+    let schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int64, false)]));
+    let expr = Arc::new(OptionalFilterPhysicalExpr::new(Arc::new(BinaryExpr::new(
+        Arc::new(Column::new("a", 0)),
+        Operator::Gt,
+        lit(5_i64),
+    )))) as Arc<dyn PhysicalExpr>;
+
+    let codec = DefaultPhysicalExtensionCodec {};
+    let converter = DefaultPhysicalProtoConverter {};
+    let proto = converter.physical_expr_to_proto(&expr, &codec)?;
+    let ctx = SessionContext::new();
+    let task_ctx = ctx.task_ctx();
+    let decode_ctx = PhysicalPlanDecodeContext::new(task_ctx.as_ref(), &codec);
+    let roundtrip = converter.proto_to_physical_expr(&proto, &schema, &decode_ctx)?;
+
+    assert!(roundtrip.is::<OptionalFilterPhysicalExpr>());
+    assert_eq!(&expr, &roundtrip);
+    assert_eq!(roundtrip.to_string(), "Optional(a@0 > 5)");
+
+    Ok(())
+}
+
 /// Roundtrip test for an execution plan where there are multiple instances of a dynamic filter
 /// with different children.
 #[test]
diff --git a/datafusion/pruning/src/pruning_predicate.rs b/datafusion/pruning/src/pruning_predicate.rs
index b49b72058e0cd..0332ff903b005 100644
--- a/datafusion/pruning/src/pruning_predicate.rs
+++ b/datafusion/pruning/src/pruning_predicate.rs
@@ -3448,6 +3448,47 @@ mod tests {
         assert_eq!(result, expected);
     }
 
+    /// Pruning sees through `OptionalFilterPhysicalExpr`: an optional filter
+    /// prunes the same as the inner filter.
+    #[test]
+    fn prune_optional_filter_same_as_inner() {
+        let (schema, statistics) = int32_setup();
+        let expected = &[true, true, false, true, true];
+
+        let required = logical2physical(&col("i").gt(lit(0)), &schema);
+        let optional = Arc::new(phys_expr::OptionalFilterPhysicalExpr::new(Arc::clone(
+            &required,
+        ))) as Arc<dyn PhysicalExpr>;
+        let dynamic = Arc::new(DynamicFilterPhysicalExpr::new(
+            collect_columns(&required)
+                .into_iter()
+                .map(|c| Arc::new(c) as Arc<dyn PhysicalExpr>)
+                .collect(),
+            Arc::clone(&required),
+        )) as Arc<dyn PhysicalExpr>;
+        let optional_dynamic =
+            Arc::new(phys_expr::OptionalFilterPhysicalExpr::new(dynamic))
+                as Arc<dyn PhysicalExpr>;
+
+        let build = |expr: Arc<dyn PhysicalExpr>| {
+            PruningPredicateBuilder::new()
+                .with_file_schema(Arc::clone(&schema))
+                .try_build(expr)
+                .unwrap()
+        };
+        let baseline = build(required);
+        assert_eq!(baseline.prune(&statistics).unwrap(), expected);
+
+        for expr in [optional, optional_dynamic] {
+            let p = build(expr);
+            assert_eq!(
+                p.predicate_expr().to_string(),
+                baseline.predicate_expr().to_string()
+            );
+            assert_eq!(p.prune(&statistics).unwrap(), expected);
+        }
+    }
+
     #[test]
     fn row_group_predicate_lt_bool() -> Result<()> {
         let schema = Schema::new(vec![Field::new("c1", DataType::Boolean, false)]);

From 2246afc4cb3c6d92fa7d816fecd4482c1c8b7967 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 18:19:41 -0500
Subject: [PATCH 07/42] feat: keep optional filters pruning-only in the Parquet
 post-scan path

With `pushdown_filters = false`, the scan evaluates all accepted filters for each row after the decode. This includes the optional filters (hash join, TopK and aggregate dynamic filters, which the producers wrap in `Optional(...)`). The join dynamic filter evaluated for each row in the scan caused a 1.16x TPC-H regression.

Optional filters are not needed for correctness. Thus the post-scan filter now never gets an optional conjunct (a root `AND` conjunct found with `split_optional`):

- `pushdown_filters = false`: only the required conjuncts run post-scan. Optional conjuncts are used only for statistics, page index, bloom filter and file pruning.
- `pushdown_filters = true`: an optional conjunct that the row filter rejects for a file is not used for that file. A required conjunct that is rejected still runs post-scan.
- A whole-file row filter build error sends only the required conjuncts to the post-scan filter.

Accepted optional conjuncts with `pushdown_filters = true` stay row filter predicates, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../datasource-parquet/src/opener/mod.rs      | 146 +++++++++++++-----
 .../datasource-parquet/src/push_decoder.rs    |  14 +-
 .../datasource-parquet/src/row_filter.rs      |  34 +++-
 3 files changed, 143 insertions(+), 51 deletions(-)

diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index 5108ff93890f8..a2d5c7d495345 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -61,7 +61,7 @@ use datafusion_common::{
 use datafusion_datasource::{PartitionedFile, TableSchema};
 use datafusion_physical_expr::expressions::{Column, DynamicFilterTracking, Literal};
 use datafusion_physical_expr::simplifier::PhysicalExprSimplifier;
-use datafusion_physical_expr::utils::collect_columns;
+use datafusion_physical_expr::utils::{collect_columns, split_optional};
 use datafusion_physical_expr_adapter::PhysicalExprAdapterFactory;
 use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
 use datafusion_physical_expr_common::sort_expr::LexOrdering;
@@ -558,7 +558,10 @@ impl DecoderReadPlans {
         //   `RowFilter` machinery cannot evaluate on this file — the rejected
         //   conjuncts returned by `RowFilterContext::try_new`).
         //
-        // Either way every conjunct is applied; nothing is silently dropped.
+        // Either way every required conjunct is applied; nothing is silently
+        // dropped. Optional conjuncts (see `split_optional`) are not needed
+        // for correctness. They never go to the post-scan filter: they are
+        // row filter predicates or they are used only for statistics pruning.
         // ---------------------------------------------------------------
         // A pruning-only predicate is applied by a `FilterExec` above the
         // scan: the scan uses it only to prune.
@@ -582,15 +585,13 @@ impl DecoderReadPlans {
                     prepared.file_metrics.clone(),
                     prepared.max_predicate_cache_size,
                 ),
-                // Pushdown disabled: the whole predicate runs post-scan (in-scan
-                // equivalent of a `FilterExec`).
-                (false, Some(predicate)) => (
-                    None,
-                    datafusion_physical_expr::split_conjunction(predicate)
-                        .into_iter()
-                        .cloned()
-                        .collect(),
-                ),
+                // Pushdown disabled: the required conjuncts run post-scan
+                // (in-scan equivalent of a `FilterExec`). Optional conjuncts
+                // (for example hash join dynamic filters) are not needed for
+                // correctness and are expensive to evaluate for each row,
+                // thus they are used only for statistics pruning, as before
+                // the scan accepted the filters.
+                (false, Some(predicate)) => (None, split_optional(predicate).0),
                 (_, None) => (None, Vec::new()),
             };
 
@@ -5396,13 +5397,101 @@ mod test {
     /// post-scan filter, so only the rows with a non-null struct survive.
     #[tokio::test]
     async fn rejected_struct_conjunct_runs_post_scan_not_dropped() {
+        let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+        let (schema, file) = write_struct_file(&store).await;
+
+        // `s IS NOT NULL` references a whole struct, which `PushdownChecker`
+        // flags as non-primitive — `FilterCandidateBuilder::build` returns
+        // `Ok(None)` and the conjunct lands in `rejected`.
+        let predicate = logical2physical(&col("s").is_not_null(), &schema);
+
+        let morselizer = ParquetMorselizerBuilder::new()
+            .with_store(Arc::clone(&store))
+            .with_schema(Arc::clone(&schema))
+            .with_predicate(predicate)
+            // The RowFilter path: emulates the post-`try_pushdown_filters`
+            // state where the parent `FilterExec` has already been removed
+            // and the scan owns the conjunct.
+            .with_pushdown_filters(true)
+            .build();
+
+        let stream = open_file(&morselizer, file).await.unwrap();
+        let (_, rows) = count_batches_and_rows(stream).await;
+
+        // 2 rows have a non-null struct. Before the fix this returned 3
+        // (the conjunct was silently dropped).
+        assert_eq!(
+            rows, 2,
+            "expected 2 rows with non-null struct; the rejected conjunct must \
+             be applied post-scan, not silently dropped"
+        );
+    }
+
+    /// An optional conjunct is not needed for correctness. The scan never
+    /// evaluates it after the decode: not when `pushdown_filters` is false,
+    /// and not when the `RowFilter` rejects it for the file. A required
+    /// conjunct in the same situations is evaluated after the decode.
+    #[tokio::test]
+    async fn optional_conjunct_is_never_evaluated_post_scan() {
+        let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+        let (schema, file) = write_struct_file(&store).await;
+
+        // `s IS NOT NULL` is rejected by the `RowFilter` (whole struct).
+        // `id > 1` can be a `RowFilter` predicate.
+        let rejected = logical2physical(&col("s").is_not_null(), &schema);
+        let pushable = logical2physical(&col("id").gt(lit(1)), &schema);
+        let optional = |expr: &Arc<dyn PhysicalExpr>| -> Arc<dyn PhysicalExpr> {
+            Arc::new(
+                datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr::new(
+                    Arc::clone(expr),
+                ),
+            )
+        };
+
+        // (predicate, pushdown_filters, expected rows, expected post-scan rows)
+        let cases: Vec<(Arc<dyn PhysicalExpr>, bool, usize, usize)> = vec![
+            // Required conjuncts: applied post-scan (#22384).
+            (Arc::clone(&rejected), false, 2, 3),
+            (Arc::clone(&rejected), true, 2, 3),
+            (Arc::clone(&pushable), false, 2, 3),
+            // Optional conjuncts: never evaluated post-scan.
+            (optional(&rejected), false, 3, 0),
+            (optional(&rejected), true, 3, 0),
+            (optional(&pushable), false, 3, 0),
+            // An optional conjunct that the `RowFilter` accepts is a row
+            // filter predicate.
+            (optional(&pushable), true, 2, 0),
+        ];
+        for (predicate, pushdown, expected_rows, expected_post_scan_rows) in cases {
+            let metrics = ExecutionPlanMetricsSet::new();
+            let morselizer = ParquetMorselizerBuilder::new()
+                .with_store(Arc::clone(&store))
+                .with_schema(Arc::clone(&schema))
+                .with_predicate(Arc::clone(&predicate))
+                .with_pushdown_filters(pushdown)
+                .with_metrics(metrics.clone())
+                .build();
+            let stream = open_file(&morselizer, file.clone()).await.unwrap();
+            let (_, rows) = count_batches_and_rows(stream).await;
+            let post_scan_rows = counter_metric_value(&metrics, "post_scan_rows_pruned")
+                + counter_metric_value(&metrics, "post_scan_rows_matched");
+            assert_eq!(
+                (rows, post_scan_rows),
+                (expected_rows, expected_post_scan_rows),
+                "predicate {predicate}, pushdown_filters {pushdown}"
+            );
+        }
+    }
+
+    /// Writes a file with the columns `id` (Int32: 1, 2, 3) and `s`
+    /// (Struct{value: Int32, label: Utf8}; row 1 is null).
+    async fn write_struct_file(
+        store: &Arc<dyn ObjectStore>,
+    ) -> (SchemaRef, PartitionedFile) {
         use arrow::array::{Int32Array, StringArray, StructArray};
         use arrow::buffer::NullBuffer;
         use arrow::datatypes::Fields;
 
-        let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
-
-        // Schema: id (Int32), s (Struct{value: Int32, label: Utf8}).
         let struct_fields: Fields = vec![
             Arc::new(Field::new("value", DataType::Int32, true)),
             Arc::new(Field::new("label", DataType::Utf8, true)),
@@ -5432,7 +5521,7 @@ mod test {
         .unwrap();
 
         let data_size = write_parquet_batches(
-            Arc::clone(&store),
+            Arc::clone(store),
             "rejected.parquet",
             vec![batch],
             None,
@@ -5440,32 +5529,7 @@ mod test {
         .await;
 
         let file = PartitionedFile::new("rejected.parquet".to_string(), data_size as u64);
-
-        // `s IS NOT NULL` references a whole struct, which `PushdownChecker`
-        // flags as non-primitive — `FilterCandidateBuilder::build` returns
-        // `Ok(None)` and the conjunct lands in `rejected`.
-        let predicate = logical2physical(&col("s").is_not_null(), &schema);
-
-        let morselizer = ParquetMorselizerBuilder::new()
-            .with_store(Arc::clone(&store))
-            .with_schema(Arc::clone(&schema))
-            .with_predicate(predicate)
-            // The RowFilter path: emulates the post-`try_pushdown_filters`
-            // state where the parent `FilterExec` has already been removed
-            // and the scan owns the conjunct.
-            .with_pushdown_filters(true)
-            .build();
-
-        let stream = open_file(&morselizer, file).await.unwrap();
-        let (_, rows) = count_batches_and_rows(stream).await;
-
-        // 2 rows have a non-null struct. Before the fix this returned 3
-        // (the conjunct was silently dropped).
-        assert_eq!(
-            rows, 2,
-            "expected 2 rows with non-null struct; the rejected conjunct must \
-             be applied post-scan, not silently dropped"
-        );
+        (schema, file)
     }
 
     /// Helpers for tests that exercise parquet virtual columns
diff --git a/datafusion/datasource-parquet/src/push_decoder.rs b/datafusion/datasource-parquet/src/push_decoder.rs
index e38cec26da65a..61464405a06e9 100644
--- a/datafusion/datasource-parquet/src/push_decoder.rs
+++ b/datafusion/datasource-parquet/src/push_decoder.rs
@@ -58,7 +58,7 @@ use parquet::file::metadata::ParquetMetaData;
 
 use datafusion_common::{DataFusionError, Result, internal_err};
 use datafusion_physical_expr::expressions::DynamicFilterTracking;
-use datafusion_physical_expr::split_conjunction;
+use datafusion_physical_expr::utils::split_optional;
 use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
 use datafusion_physical_plan::metrics::{BaselineMetrics, Count, Gauge};
 use datafusion_pruning::{PruningPredicate, PruningPredicateBuilder};
@@ -412,15 +412,15 @@ impl RowFilterContext {
                 (context, rejected)
             }
             Err(e) => {
-                // Whole-file build failure: route every conjunct post-scan
-                // rather than silently dropping the predicate.
+                // Whole-file build failure: route every required conjunct
+                // post-scan rather than silently dropping the predicate.
+                // Optional conjuncts are not needed for correctness, thus
+                // they are not evaluated after the scan.
                 debug!(
                     "Ignoring error prebuilding row filter candidates: {e}; \
-                     all conjuncts will be evaluated post-scan"
+                     all required conjuncts will be evaluated post-scan"
                 );
-                let rejected =
-                    split_conjunction(predicate).into_iter().cloned().collect();
-                (None, rejected)
+                (None, split_optional(predicate).0)
             }
         }
     }
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 96c304a38f661..8cf0a2468c628 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -78,7 +78,7 @@ use parquet::file::metadata::ParquetMetaData;
 use datafusion_common::Result;
 use datafusion_common::cast::as_boolean_array;
 use datafusion_common::tree_node::TreeNode;
-use datafusion_physical_expr::utils::reassign_expr_columns;
+use datafusion_physical_expr::utils::{is_optional_filter, reassign_expr_columns};
 use datafusion_physical_expr::{PhysicalExpr, split_conjunction};
 
 use datafusion_physical_plan::metrics;
@@ -416,11 +416,14 @@ fn size_of_columns(columns: &[usize], metadata: &ParquetMetaData) -> Result<usiz
 /// `Ok((row_filter, rejected))` where:
 /// * `row_filter` is `Some` if at least one conjunct can be evaluated as an
 ///   `ArrowPredicate`, `None` otherwise.
-/// * `rejected` holds the conjuncts that *cannot* be evaluated as an
+/// * `rejected` holds the required conjuncts that *cannot* be evaluated as an
 ///   `ArrowPredicate` (for example whole-struct references or columns missing
 ///   from this file's physical schema). The caller MUST apply these elsewhere
 ///   — e.g. as a post-scan filter — otherwise the predicate is relaxed and the
-///   query returns wrong results.
+///   query returns wrong results. Optional conjuncts (see
+///   [`split_optional`](datafusion_physical_expr::utils::split_optional))
+///   that cannot be evaluated are not used and are not in `rejected`: they
+///   are not needed for correctness.
 ///
 /// `Err(e)` if an error occurs while building the filter.
 ///
@@ -493,6 +496,11 @@ impl PrebuiltRowFilterCandidate {
 /// entirely. The second element holds the conjuncts that cannot be evaluated
 /// as an `ArrowPredicate` on this file; see [`build_row_filter`] for why the
 /// caller must apply them elsewhere.
+///
+/// An optional conjunct (an `OptionalFilterPhysicalExpr` on the root `AND`
+/// chain) that cannot be evaluated as an `ArrowPredicate` for this file is
+/// not used. It is not in the rejected conjuncts either, thus the caller does
+/// not evaluate it after the scan. This is always safe.
 #[expect(clippy::type_complexity)]
 pub(crate) fn prebuild_row_filter_candidates(
     expr: &Arc<dyn PhysicalExpr>,
@@ -512,6 +520,26 @@ pub(crate) fn prebuild_row_filter_candidates(
     let mut candidates: Vec<FilterCandidate> = Vec::with_capacity(predicates.len());
     let mut rejected: Vec<Arc<dyn PhysicalExpr>> = Vec::new();
     for predicate in predicates {
+        // An optional conjunct that cannot be pushed down for this file is not
+        // needed for correctness: do not use it, and do not send it to the
+        // post-scan filter.
+        if is_optional_filter(predicate) {
+            match FilterCandidateBuilder::new(
+                Arc::clone(predicate),
+                Arc::clone(file_schema),
+            )
+            .build(metadata)
+            {
+                Ok(Some(candidate)) => candidates.push(candidate),
+                Ok(None) => {}
+                Err(e) => {
+                    log::debug!(
+                        "Ignoring optional filter that cannot be pushed down: {e}"
+                    );
+                }
+            }
+            continue;
+        }
         match FilterCandidateBuilder::new(Arc::clone(predicate), Arc::clone(file_schema))
             .build(metadata)?
         {

From ab2b6e27efcca86549c7676174b1f60cc2f3c0ba Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 14:52:54 -0500
Subject: [PATCH 08/42] feat: add filter_stats (FilterCost, Clock) for filter
 measurements

Add the `datafusion_physical_expr::filter_stats` module with the shared
primitives that adaptive filter code uses to measure filters at runtime:

- `Clock`: a monotonic clock in nanoseconds that tests can replace.
  `SystemClock` is the real clock. `ManualClock` moves only when a test
  moves it, thus decisions that use time are deterministic in tests.
- `FilterCost`: the rows in, the rows out and the evaluation time of one
  filter, and the derived cost for each row and rows removed for each
  nanosecond.
- `duration_nanos`: a `Duration` in nanoseconds, saturated to `u64::MAX`.

No code uses the module yet, thus behavior does not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/physical-expr/src/filter_stats.rs | 199 +++++++++++++++++++
 datafusion/physical-expr/src/lib.rs          |   1 +
 2 files changed, 200 insertions(+)
 create mode 100644 datafusion/physical-expr/src/filter_stats.rs

diff --git a/datafusion/physical-expr/src/filter_stats.rs b/datafusion/physical-expr/src/filter_stats.rs
new file mode 100644
index 0000000000000..f5f5c4f03f8b1
--- /dev/null
+++ b/datafusion/physical-expr/src/filter_stats.rs
@@ -0,0 +1,199 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Measurements of filter evaluations at runtime.
+//!
+//! Operators that adapt to the data measure the filters that they evaluate:
+//! the rows in, the rows that pass and the evaluation time. This module has
+//! the shared parts:
+//!
+//! * [`Clock`]: a monotonic clock that tests can replace, so that decisions
+//!   that use time are deterministic in tests. [`SystemClock`] is the real
+//!   clock and [`ManualClock`] is a clock that only moves when a test moves
+//!   it.
+//! * [`FilterCost`]: the counts and the time of one filter, and the values
+//!   derived from them (cost for each row, rows removed for each
+//!   nanosecond).
+//!
+//! For example, an operator can use them to pause a filter that costs more
+//! than it saves, or to change the order of the conjuncts of a predicate.
+
+use std::fmt::Debug;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::Duration;
+
+use datafusion_common::instant::Instant;
+
+/// A monotonic clock in nanoseconds.
+///
+/// Production code uses [`SystemClock`]. Tests use [`ManualClock`] (or their
+/// own implementation), thus decisions that use time are deterministic in
+/// tests.
+pub trait Clock: Debug + Send + Sync {
+    /// Nanoseconds since an arbitrary fixed point. The value never
+    /// decreases.
+    fn now_nanos(&self) -> u64;
+}
+
+/// The real monotonic [`Clock`].
+#[derive(Debug, Clone, Copy)]
+pub struct SystemClock {
+    start: Instant,
+}
+
+impl SystemClock {
+    /// Creates a clock whose zero is now.
+    pub fn new() -> Self {
+        Self {
+            start: Instant::now(),
+        }
+    }
+
+    /// A shared [`SystemClock`], as a trait object.
+    pub fn shared() -> Arc<dyn Clock> {
+        Arc::new(Self::new())
+    }
+}
+
+impl Default for SystemClock {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl Clock for SystemClock {
+    fn now_nanos(&self) -> u64 {
+        u64::try_from(self.start.elapsed().as_nanos()).unwrap_or(u64::MAX)
+    }
+}
+
+/// A [`Clock`] that moves only when [`Self::advance`] is called. For tests.
+#[derive(Debug, Default)]
+pub struct ManualClock {
+    nanos: AtomicU64,
+}
+
+impl ManualClock {
+    /// Creates a clock at zero.
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    /// Moves the clock forward by `nanos` nanoseconds.
+    pub fn advance(&self, nanos: u64) {
+        self.nanos.fetch_add(nanos, Ordering::Relaxed);
+    }
+}
+
+impl Clock for ManualClock {
+    fn now_nanos(&self) -> u64 {
+        self.nanos.load(Ordering::Relaxed)
+    }
+}
+
+/// Returns the nanoseconds in `elapsed`, saturated to `u64::MAX`.
+pub fn duration_nanos(elapsed: Duration) -> u64 {
+    u64::try_from(elapsed.as_nanos()).unwrap_or(u64::MAX)
+}
+
+/// The measurements of one filter: the rows that it was evaluated on, the
+/// rows that passed it, and the evaluation time.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
+pub struct FilterCost {
+    /// Rows that the filter was evaluated on.
+    pub rows_in: u64,
+    /// Rows that passed the filter (`true`; `null` does not pass).
+    pub rows_out: u64,
+    /// Evaluation time in nanoseconds.
+    pub nanos: u64,
+}
+
+impl FilterCost {
+    /// Adds the result of one evaluation. `rows_out` larger than `rows_in`
+    /// is used as `rows_in`.
+    pub fn add(&mut self, rows_in: u64, rows_out: u64, nanos: u64) {
+        self.rows_in = self.rows_in.saturating_add(rows_in);
+        self.rows_out = self.rows_out.saturating_add(rows_out.min(rows_in));
+        self.nanos = self.nanos.saturating_add(nanos);
+    }
+
+    /// Rows that the filter removed.
+    pub fn rows_removed(&self) -> u64 {
+        self.rows_in.saturating_sub(self.rows_out)
+    }
+
+    /// Nanoseconds for each evaluated row, or `None` if the filter was not
+    /// evaluated on any row.
+    pub fn nanos_per_row(&self) -> Option<f64> {
+        (self.rows_in > 0).then(|| self.nanos as f64 / self.rows_in as f64)
+    }
+
+    /// Rows removed for each nanosecond, `(1 + rows_in - rows_out) / nanos`,
+    /// or `None` if the filter was not evaluated on any row. A larger value
+    /// is a better filter to evaluate first. This is the ranking key of
+    /// Velox (Pedreira et al., VLDB 2022). The `1 +` ranks a filter that
+    /// removes no rows by its cost, and a zero time is used as 1 ns.
+    pub fn rows_removed_per_nano(&self) -> Option<f64> {
+        (self.rows_in > 0)
+            .then(|| (1 + self.rows_removed()) as f64 / self.nanos.max(1) as f64)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn filter_cost_derived_values() {
+        let empty = FilterCost::default();
+        assert_eq!(empty.nanos_per_row(), None);
+        assert_eq!(empty.rows_removed_per_nano(), None);
+
+        let mut cost = FilterCost::default();
+        cost.add(100, 25, 1_000);
+        cost.add(100, 200, 1_000);
+        assert_eq!(cost.rows_in, 200);
+        // `rows_out` is at most `rows_in` for each evaluation.
+        assert_eq!(cost.rows_out, 125);
+        assert_eq!(cost.rows_removed(), 75);
+        assert_eq!(cost.nanos_per_row(), Some(10.0));
+        assert_eq!(cost.rows_removed_per_nano(), Some(76.0 / 2_000.0));
+
+        // A zero time is used as 1 ns.
+        let mut free = FilterCost::default();
+        free.add(10, 0, 0);
+        assert_eq!(free.rows_removed_per_nano(), Some(11.0));
+    }
+
+    #[test]
+    fn manual_clock_moves_only_when_advanced() {
+        let clock = ManualClock::new();
+        assert_eq!(clock.now_nanos(), 0);
+        clock.advance(5);
+        clock.advance(7);
+        assert_eq!(clock.now_nanos(), 12);
+    }
+
+    #[test]
+    fn system_clock_is_monotonic() {
+        let clock = SystemClock::new();
+        let first = clock.now_nanos();
+        assert!(clock.now_nanos() >= first);
+        assert_eq!(duration_nanos(Duration::from_micros(3)), 3_000);
+    }
+}
diff --git a/datafusion/physical-expr/src/lib.rs b/datafusion/physical-expr/src/lib.rs
index 80e9f88b510ed..75d9d7ea70c2b 100644
--- a/datafusion/physical-expr/src/lib.rs
+++ b/datafusion/physical-expr/src/lib.rs
@@ -34,6 +34,7 @@ pub mod binary_map {
 pub mod async_scalar_function;
 pub mod equivalence;
 pub mod expressions;
+pub mod filter_stats;
 pub mod higher_order_function;
 pub mod intervals;
 mod partitioning;

From ab74ffa5dbea9eb65af61bb6be8768c8b86dcc5d Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 15:02:40 -0500
Subject: [PATCH 09/42] feat: add OptionalFilterGate to pause optional filters
 that cost more than they save

Add a runtime gate that pauses optional filters (filters that are not
needed for correctness, such as hash join and TopK dynamic filters) when
they cost more than they save. The gate is a per-stream state machine
(Evaluate / Paused with exponential backoff) that restarts evaluation
when the filter changes. The gate finds the dynamic filters one time
with `DynamicFilterTracking::classify` and then polls their
subscriptions, so a check does not walk the filter tree. Gates do not
share state.

At the end of each window of evaluated batches the gate pauses the
filter if the window removed no rows, or if its evaluation time is
larger than the work that the removed rows save:
`(rows_in - rows_out) * saving_ns_per_row`. The saving for each row is
the configured minimum plus an optional value that the consumer measures
and updates (`MeasuredRowSaving`). The cost rule has a margin (pause
above 1.1x the saving, resume below 0.9x) so that a filter does not
switch on and off when cost and saving are almost equal.

Add the `datafusion.execution.optional_filter_min_saving_ns_per_row`
option (default 20). No operator uses the gate yet, so behavior does not
change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/common/src/config.rs               |   11 +
 .../expressions/dynamic_filters/tracker.rs    |    2 +-
 datafusion/physical-expr/src/lib.rs           |    1 +
 .../physical-expr/src/optional_filter_gate.rs | 1005 +++++++++++++++++
 .../test_files/information_schema.slt         |    2 +
 docs/source/user-guide/configs.md             |    1 +
 6 files changed, 1021 insertions(+), 1 deletion(-)
 create mode 100644 datafusion/physical-expr/src/optional_filter_gate.rs

diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index 913845a4219e4..24d52eb080fad 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -1184,6 +1184,17 @@ config_namespace! {
         ///
         /// Disabled by default, set to a number greater than 0 for enabling it.
         pub hash_join_buffering_capacity: usize, default = 0
+
+        /// The assumed work, in nanoseconds, that each row removed by an
+        /// optional filter saves downstream. Optional filters are filters that
+        /// are not needed for correctness, such as the dynamic filters that
+        /// hash joins and TopK push down into scans. When an operator evaluates
+        /// optional filters adaptively, it pauses an optional filter whose
+        /// evaluation costs more than the work that it saves. Consumers that can
+        /// measure the saving (the Parquet scan) add their measured decode cost.
+        /// The default is about the cost of a hash table probe for one row. The
+        /// best value depends on the hardware.
+        pub optional_filter_min_saving_ns_per_row: f64, default = 20.0
     }
 }
 
diff --git a/datafusion/physical-expr/src/expressions/dynamic_filters/tracker.rs b/datafusion/physical-expr/src/expressions/dynamic_filters/tracker.rs
index fd4c18b07e2cd..6f0a830d07a85 100644
--- a/datafusion/physical-expr/src/expressions/dynamic_filters/tracker.rs
+++ b/datafusion/physical-expr/src/expressions/dynamic_filters/tracker.rs
@@ -157,7 +157,7 @@ impl DynamicFilterTracker {
     }
 
     /// `true` once every watched filter has completed and been dropped.
-    fn is_exhausted(&self) -> bool {
+    pub(crate) fn is_exhausted(&self) -> bool {
         self.subscriptions.is_empty()
     }
 }
diff --git a/datafusion/physical-expr/src/lib.rs b/datafusion/physical-expr/src/lib.rs
index 75d9d7ea70c2b..fba82a96b0c46 100644
--- a/datafusion/physical-expr/src/lib.rs
+++ b/datafusion/physical-expr/src/lib.rs
@@ -37,6 +37,7 @@ pub mod expressions;
 pub mod filter_stats;
 pub mod higher_order_function;
 pub mod intervals;
+pub mod optional_filter_gate;
 mod partitioning;
 mod physical_expr;
 pub mod planner;
diff --git a/datafusion/physical-expr/src/optional_filter_gate.rs b/datafusion/physical-expr/src/optional_filter_gate.rs
new file mode 100644
index 0000000000000..8275430baca00
--- /dev/null
+++ b/datafusion/physical-expr/src/optional_filter_gate.rs
@@ -0,0 +1,1005 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A runtime gate that pauses optional filters that cost more than they
+//! save.
+//!
+//! An *optional filter* is a filter that is not needed for correctness, for
+//! example a dynamic filter that a hash join or a TopK pushes down into a
+//! scan. An operator can skip such a filter and still produce correct
+//! results. When the filter removes few rows, or when it is expensive (for
+//! example a hash table lookup with many columns), the cost to evaluate it
+//! can be larger than the benefit.
+//!
+//! [`OptionalFilterGate`] decides, batch by batch, if a stream evaluates the
+//! filter or skips it. Each stream has its own gate, and gates do not share
+//! state.
+//!
+//! # State machine
+//!
+//! ```text
+//!                 keep (see "Decision")
+//!              (reset backoff, new window)
+//!                     +-------+
+//!                     |       |
+//!                     v       |
+//!               +-------------+-+           pause                 +---------------------+
+//!  start ------>|   Evaluate    |-------------------------------->|       Paused        |
+//!               | (window of    |   (pause for `backoff` batches, | (skip `remaining`   |
+//!               | sample_batches|    then double `backoff`)       |  batches)           |
+//!               | batches)      |<--------------------------------|                     |
+//!               +---------------+   pause ends: probe with a      +---------------------+
+//!                                   fresh window
+//! ```
+//!
+//! * In `Evaluate`, the gate collects the rows in, the rows out and the
+//!   evaluation time of `sample_batches` batches (a *window*). Then it
+//!   decides (see below). To pause, it goes to `Paused` for `backoff` batches
+//!   and doubles `backoff` (up to `max_pause_batches`). To keep the filter,
+//!   it stays in `Evaluate`, sets `backoff` to `initial_pause_batches` and
+//!   starts a new window.
+//! * In `Paused`, the gate skips batches. It does not change counters or
+//!   `backoff` for skipped batches. When the pause ends, the gate evaluates
+//!   a new window (a *probe*).
+//! * Before each batch the gate checks if the filter changed (for example a
+//!   dynamic filter got new bounds). If so, the gate goes to `Evaluate` with
+//!   an empty window and sets `backoff` to `initial_pause_batches`.
+//!
+//! # Decision
+//!
+//! At the end of each window, the gate pauses the filter if one of these
+//! rules is true:
+//!
+//! 1. The filter removed no rows in the window.
+//! 2. The evaluation time of the window (`cost_ns`) is larger than the work
+//!    that the removed rows save (`saving_ns`):
+//!
+//!    ```text
+//!    saving_ns = (rows_in - rows_out) * saving_ns_per_row
+//!    saving_ns_per_row = min_saving_ns_per_row + measured saving
+//!    ```
+//!
+//!    `min_saving_ns_per_row` comes from the configuration. It is the work
+//!    that a removed row saves after the filter, for example a hash table
+//!    probe in a join. The *measured saving* is optional: a consumer that
+//!    can measure more work that a removed row saves (the Parquet scan
+//!    measures the decode time of the columns that the filter does not read)
+//!    gives it in a shared [`MeasuredRowSaving`] and updates it at any time.
+//!
+//!    To prevent a filter from switching on and off when the cost and the
+//!    saving are almost equal, this rule has a margin: a running filter is
+//!    paused only if `cost_ns > saving_ns * 1.1`, and a probe after a pause
+//!    turns the filter on again only if `cost_ns < saving_ns * 0.9`.
+//!
+//! Thus a filter that removes most rows but is expensive is paused, and a
+//! cheap filter stays on also when it removes only some of the rows.
+//!
+//! The gate measures time with a [`Clock`]. Tests use a [`ManualClock`], so
+//! that the decisions are deterministic.
+//!
+//! [`ManualClock`]: crate::filter_stats::ManualClock
+//!
+//! # Change detection
+//!
+//! The gate walks the filter one time, when it is created, with
+//! [`DynamicFilterTracking::classify`]. The walk subscribes to each
+//! [`DynamicFilterPhysicalExpr`] in the filter that is not complete. Before
+//! each batch, the gate polls these subscriptions with
+//! [`DynamicFilterTracker::changed`]. When nothing changed, this is one
+//! atomic load for each subscription. The tracker drops a subscription when
+//! its filter is complete. A filter without dynamic filters, or with only
+//! complete dynamic filters, is never polled and never resets the gate.
+//!
+//! [`DynamicFilterPhysicalExpr`]: crate::expressions::DynamicFilterPhysicalExpr
+//! [`DynamicFilterTracker::changed`]: crate::expressions::DynamicFilterTracker::changed
+
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::Duration;
+
+use datafusion_common::config::ExecutionOptions;
+use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
+
+use crate::expressions::DynamicFilterTracking;
+use crate::filter_stats::{Clock, FilterCost, SystemClock, duration_nanos};
+
+/// A running filter is paused by the cost rule only if its cost is larger
+/// than this multiple of its saving. See the [module documentation](self).
+const PAUSE_COST_MARGIN: f64 = 1.1;
+
+/// A probe turns a paused filter on again (by the cost rule) only if its
+/// cost is smaller than this multiple of its saving.
+const RESUME_COST_MARGIN: f64 = 0.9;
+
+/// Configuration of an [`OptionalFilterGate`].
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub struct OptionalFilterGateConfig {
+    /// Number of evaluated batches in one window. The gate makes a decision
+    /// at the end of each window. Values smaller than 1 are used as 1.
+    pub sample_batches: usize,
+    /// Number of batches to skip at the first pause, and after the filter
+    /// was selective again. Values smaller than 1 are used as 1.
+    pub initial_pause_batches: usize,
+    /// Maximum number of batches to skip in one pause. Values smaller than
+    /// `initial_pause_batches` are used as `initial_pause_batches`.
+    pub max_pause_batches: usize,
+    /// Work, in nanoseconds, that each row removed by the filter saves after
+    /// the filter, at the least. The gate adds the saving that the consumer
+    /// measures (see [`MeasuredRowSaving`]). The gate pauses a filter whose
+    /// evaluation time is larger than the saving of the rows that it
+    /// removes. Negative values are used as 0.
+    pub min_saving_ns_per_row: f64,
+}
+
+impl Default for OptionalFilterGateConfig {
+    fn default() -> Self {
+        Self {
+            sample_batches: 2,
+            initial_pause_batches: 4,
+            max_pause_batches: 32,
+            min_saving_ns_per_row: 20.0,
+        }
+    }
+}
+
+impl From<&ExecutionOptions> for OptionalFilterGateConfig {
+    /// Uses `optional_filter_min_saving_ns_per_row` from `options`, and the
+    /// default values for the other fields.
+    fn from(options: &ExecutionOptions) -> Self {
+        Self {
+            min_saving_ns_per_row: options.optional_filter_min_saving_ns_per_row,
+            ..Default::default()
+        }
+    }
+}
+
+impl OptionalFilterGateConfig {
+    /// Returns a copy with the documented minimum values applied.
+    fn normalized(self) -> Self {
+        let sample_batches = self.sample_batches.max(1);
+        let initial_pause_batches = self.initial_pause_batches.max(1);
+        let max_pause_batches = self.max_pause_batches.max(initial_pause_batches);
+        Self {
+            sample_batches,
+            initial_pause_batches,
+            max_pause_batches,
+            min_saving_ns_per_row: self.min_saving_ns_per_row.max(0.0),
+        }
+    }
+}
+
+/// The decision of an [`OptionalFilterGate`] for one batch.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum GateDecision {
+    /// Evaluate the filter on this batch, then call
+    /// [`OptionalFilterGate::record`] with the row counts and the time.
+    Evaluate,
+    /// Do not evaluate the filter on this batch. Let all rows pass.
+    Skip,
+}
+
+/// Work, in nanoseconds, that each row removed by an optional filter saves,
+/// as measured by the consumer of the filter. The gate adds it to
+/// [`OptionalFilterGateConfig::min_saving_ns_per_row`].
+///
+/// The consumer creates one value, gives a clone of the [`Arc`] to the gate
+/// with [`OptionalFilterGate::with_measured_saving`], and updates it at any
+/// time with [`Self::set_ns_per_row`]. The gate reads it at each decision.
+/// For example, the Parquet scan sets it to the time to decode the columns
+/// that the filter does not read, for each row.
+///
+/// The value is an `f64` in an [`AtomicU64`], thus reads and updates are
+/// cheap and lock-free.
+#[derive(Debug, Default)]
+pub struct MeasuredRowSaving {
+    /// The bits of the `f64` value.
+    ns_per_row_bits: AtomicU64,
+}
+
+impl MeasuredRowSaving {
+    /// Creates a value of 0 ns.
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    /// Sets the measured saving for each removed row, in nanoseconds.
+    /// Values that are negative or not finite are used as 0.
+    pub fn set_ns_per_row(&self, ns_per_row: f64) {
+        let ns_per_row = if ns_per_row.is_finite() {
+            ns_per_row.max(0.0)
+        } else {
+            0.0
+        };
+        self.ns_per_row_bits
+            .store(ns_per_row.to_bits(), Ordering::Relaxed);
+    }
+
+    /// The measured saving for each removed row, in nanoseconds.
+    pub fn ns_per_row(&self) -> f64 {
+        f64::from_bits(self.ns_per_row_bits.load(Ordering::Relaxed))
+    }
+}
+
+/// The state of an [`OptionalFilterGate`].
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum GateState {
+    /// Evaluate the filter and collect counts and time for the current
+    /// window.
+    Evaluate {
+        window: FilterCost,
+        batches_in_window: usize,
+    },
+    /// Skip the filter for `remaining_batches` more batches.
+    Paused { remaining_batches: usize },
+}
+
+impl GateState {
+    const fn new_window() -> Self {
+        Self::Evaluate {
+            window: FilterCost {
+                rows_in: 0,
+                rows_out: 0,
+                nanos: 0,
+            },
+            batches_in_window: 0,
+        }
+    }
+}
+
+/// Decides, batch by batch, if one stream evaluates an optional filter.
+///
+/// See the [module documentation](self) for the state machine. A gate is
+/// for one stream only. Do not share it between streams.
+///
+/// Call [`Self::begin_batch`] before each batch. If it returns
+/// [`GateDecision::Evaluate`], evaluate [`Self::filter`] on the batch and
+/// then call [`Self::record`] with the row counts and the evaluation time.
+/// Measure the time with [`Self::clock`], so that tests can replace it.
+#[derive(Debug)]
+pub struct OptionalFilterGate {
+    filter: Arc<dyn PhysicalExpr>,
+    /// The dynamic filters in `filter` that can still change.
+    tracking: DynamicFilterTracking,
+    config: OptionalFilterGateConfig,
+    /// The clock that consumers use to measure the evaluation time.
+    clock: Arc<dyn Clock>,
+    /// The saving that the consumer measures, added to
+    /// `config.min_saving_ns_per_row`.
+    measured_saving: Option<Arc<MeasuredRowSaving>>,
+    state: GateState,
+    /// True while the current window is a probe after a pause. The cost
+    /// rule then uses [`RESUME_COST_MARGIN`].
+    probing: bool,
+    /// Length of the next pause, in batches.
+    backoff: usize,
+    /// True after `begin_batch` returned `Evaluate` and before `record`.
+    awaiting_record: bool,
+    pauses: usize,
+}
+
+impl OptionalFilterGate {
+    /// Creates a gate for `filter`, a boolean expression. The gate starts to
+    /// evaluate the filter.
+    ///
+    /// The gate uses a [`SystemClock`] and no measured saving. See
+    /// [`Self::with_clock`] and [`Self::with_measured_saving`].
+    ///
+    /// This walks `filter` one time to find its dynamic filters.
+    pub fn new(filter: Arc<dyn PhysicalExpr>, config: OptionalFilterGateConfig) -> Self {
+        let config = config.normalized();
+        let tracking = DynamicFilterTracking::classify(&filter);
+        Self {
+            filter,
+            tracking,
+            config,
+            clock: SystemClock::shared(),
+            measured_saving: None,
+            state: GateState::new_window(),
+            probing: false,
+            backoff: config.initial_pause_batches,
+            awaiting_record: false,
+            pauses: 0,
+        }
+    }
+
+    /// Uses `clock` as the clock of this gate, see [`Self::clock`].
+    pub fn with_clock(mut self, clock: Arc<dyn Clock>) -> Self {
+        self.clock = clock;
+        self
+    }
+
+    /// Adds `saving` to [`OptionalFilterGateConfig::min_saving_ns_per_row`]
+    /// at each decision. See [`MeasuredRowSaving`].
+    pub fn with_measured_saving(mut self, saving: Arc<MeasuredRowSaving>) -> Self {
+        self.measured_saving = Some(saving);
+        self
+    }
+
+    /// The filter of this gate.
+    pub fn filter(&self) -> &Arc<dyn PhysicalExpr> {
+        &self.filter
+    }
+
+    /// The clock to measure the evaluation time that is given to
+    /// [`Self::record`].
+    pub fn clock(&self) -> &Arc<dyn Clock> {
+        &self.clock
+    }
+
+    /// The work, in nanoseconds, that the gate assumes each removed row
+    /// saves now: the configured minimum plus the measured saving.
+    pub fn saving_ns_per_row(&self) -> f64 {
+        let measured = self
+            .measured_saving
+            .as_ref()
+            .map_or(0.0, |saving| saving.ns_per_row());
+        self.config.min_saving_ns_per_row + measured
+    }
+
+    /// Call before each batch. Returns if the caller must evaluate the
+    /// filter on the batch or skip it.
+    ///
+    /// If the result is [`GateDecision::Evaluate`], call [`Self::record`]
+    /// after the evaluation.
+    pub fn begin_batch(&mut self) -> GateDecision {
+        // Only a filter with dynamic filters that are not complete can
+        // change. When nothing changed, this is one atomic load for each
+        // such dynamic filter.
+        if let Some(tracker) = self.tracking.watcher()
+            && tracker.changed()
+        {
+            self.state = GateState::new_window();
+            self.probing = false;
+            self.backoff = self.config.initial_pause_batches;
+        }
+
+        match &mut self.state {
+            GateState::Paused { remaining_batches } => {
+                // Only count down. Counters and backoff change only at
+                // decision points.
+                *remaining_batches = remaining_batches.saturating_sub(1);
+                if *remaining_batches == 0 {
+                    // The next batch is a probe.
+                    self.state = GateState::new_window();
+                    self.probing = true;
+                }
+                self.awaiting_record = false;
+                GateDecision::Skip
+            }
+            GateState::Evaluate { .. } => {
+                self.awaiting_record = true;
+                GateDecision::Evaluate
+            }
+        }
+    }
+
+    /// Records the result of an evaluation that [`Self::begin_batch`]
+    /// requested. `rows_in` is the number of rows evaluated, `rows_out`
+    /// the number of rows that passed (a null result does not pass) and
+    /// `elapsed` the evaluation time.
+    ///
+    /// Calls without a matching `begin_batch` that returned
+    /// [`GateDecision::Evaluate`] are ignored.
+    pub fn record(&mut self, rows_in: usize, rows_out: usize, elapsed: Duration) {
+        if !std::mem::take(&mut self.awaiting_record) {
+            return;
+        }
+        let GateState::Evaluate {
+            window,
+            batches_in_window,
+        } = &mut self.state
+        else {
+            return;
+        };
+        window.add(rows_in as u64, rows_out as u64, duration_nanos(elapsed));
+        *batches_in_window += 1;
+        if *batches_in_window >= self.config.sample_batches {
+            let window = *window;
+            self.decide(window);
+        }
+    }
+
+    /// Number of times the gate paused the filter.
+    pub fn pauses(&self) -> usize {
+        self.pauses
+    }
+
+    /// True if the gate skips the next batch, unless the filter changes
+    /// before it.
+    pub fn is_paused(&self) -> bool {
+        matches!(self.state, GateState::Paused { .. })
+    }
+
+    /// Makes a decision at the end of a window.
+    fn decide(&mut self, window: FilterCost) {
+        if window.rows_in == 0 {
+            // No rows, thus no information. Start a new window.
+            self.state = GateState::new_window();
+            return;
+        }
+        if self.should_pause(&window) {
+            self.start_pause(self.backoff);
+        } else {
+            self.state = GateState::new_window();
+            self.probing = false;
+            self.backoff = self.config.initial_pause_batches;
+        }
+    }
+
+    /// The decision rules, see the [module documentation](self).
+    fn should_pause(&self, window: &FilterCost) -> bool {
+        let rows_removed = window.rows_removed();
+        if rows_removed == 0 {
+            return true;
+        }
+        // The filter costs more than it saves.
+        let cost_ns = window.nanos as f64;
+        let saving_ns = rows_removed as f64 * self.saving_ns_per_row();
+        if self.probing {
+            // Turn the filter on again only if it is clearly worth its cost.
+            cost_ns >= saving_ns * RESUME_COST_MARGIN
+        } else {
+            cost_ns > saving_ns * PAUSE_COST_MARGIN
+        }
+    }
+
+    /// Goes to `Paused` for `pause_batches` batches and doubles the backoff.
+    fn start_pause(&mut self, pause_batches: usize) {
+        let pause_batches = pause_batches.max(1);
+        self.probing = false;
+        self.state = GateState::Paused {
+            remaining_batches: pause_batches,
+        };
+        self.backoff = pause_batches
+            .saturating_mul(2)
+            .min(self.config.max_pause_batches);
+        self.pauses += 1;
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::expressions::{BinaryExpr, Column, DynamicFilterPhysicalExpr, col, lit};
+    use crate::filter_stats::ManualClock;
+    use arrow::array::{Array, BooleanArray, Int32Array, RecordBatch};
+    use arrow::datatypes::{DataType, Field, Schema};
+    use datafusion_common::cast::as_boolean_array;
+    use datafusion_expr::Operator;
+
+    const ROWS: usize = 1000;
+
+    fn static_filter() -> Arc<dyn PhysicalExpr> {
+        lit(true)
+    }
+
+    /// A gate with the default configuration and a clock that does not
+    /// move: the evaluation time is 0, thus only a window that removes no
+    /// rows pauses the filter.
+    fn gate_with(filter: Arc<dyn PhysicalExpr>) -> OptionalFilterGate {
+        OptionalFilterGate::new(filter, OptionalFilterGateConfig::default())
+            .with_clock(Arc::new(ManualClock::new()))
+    }
+
+    fn new_gate() -> OptionalFilterGate {
+        gate_with(static_filter())
+    }
+
+    /// Feeds one batch with the given pass ratio and an evaluation time of
+    /// 0. Returns the decision.
+    fn feed(gate: &mut OptionalFilterGate, pass_ratio: f64) -> GateDecision {
+        feed_timed(gate, pass_ratio, 0.0)
+    }
+
+    /// Feeds one batch with the given pass ratio and an evaluation time of
+    /// `ns_per_row` for each row. Returns the decision.
+    fn feed_timed(
+        gate: &mut OptionalFilterGate,
+        pass_ratio: f64,
+        ns_per_row: f64,
+    ) -> GateDecision {
+        let decision = gate.begin_batch();
+        if decision == GateDecision::Evaluate {
+            let elapsed = Duration::from_nanos((ROWS as f64 * ns_per_row) as u64);
+            gate.record(ROWS, (ROWS as f64 * pass_ratio) as usize, elapsed);
+        }
+        decision
+    }
+
+    /// Feeds `n` batches and returns how many the gate evaluated.
+    fn feed_n(gate: &mut OptionalFilterGate, n: usize, pass_ratio: f64) -> usize {
+        (0..n)
+            .filter(|_| feed(gate, pass_ratio) == GateDecision::Evaluate)
+            .count()
+    }
+
+    /// Feeds batches until the gate evaluates one. Returns the number of
+    /// skipped batches.
+    fn skip_until_probe(gate: &mut OptionalFilterGate, pass_ratio: f64) -> usize {
+        let mut skipped = 0;
+        while feed(gate, pass_ratio) == GateDecision::Skip {
+            skipped += 1;
+            assert!(skipped < 10_000, "gate never probes");
+        }
+        skipped
+    }
+
+    /// Evaluates the filter of `gate` on `batch` like a consumer does, or
+    /// skips it. Returns `None` if the gate skipped the filter.
+    fn evaluate(
+        gate: &mut OptionalFilterGate,
+        batch: &RecordBatch,
+    ) -> Option<BooleanArray> {
+        if gate.begin_batch() == GateDecision::Skip {
+            return None;
+        }
+        let num_rows = batch.num_rows();
+        let start = gate.clock().now_nanos();
+        let result = gate
+            .filter()
+            .evaluate(batch)
+            .unwrap()
+            .into_array(num_rows)
+            .unwrap();
+        let result = as_boolean_array(&result).unwrap().clone();
+        let elapsed = gate.clock().now_nanos().saturating_sub(start);
+        // `true_count` does not count nulls.
+        gate.record(num_rows, result.true_count(), Duration::from_nanos(elapsed));
+        Some(result)
+    }
+
+    #[test]
+    fn pauses_filter_that_removes_no_rows() {
+        let mut gate = new_gate();
+        assert_eq!(feed(&mut gate, 1.0), GateDecision::Evaluate);
+        assert!(!gate.is_paused());
+        assert_eq!(feed(&mut gate, 1.0), GateDecision::Evaluate);
+        assert!(gate.is_paused());
+        assert_eq!(gate.pauses(), 1);
+
+        // Pauses for `initial_pause_batches` batches.
+        for _ in 0..4 {
+            assert_eq!(feed(&mut gate, 1.0), GateDecision::Skip);
+        }
+        // Then it probes.
+        assert_eq!(feed(&mut gate, 1.0), GateDecision::Evaluate);
+    }
+
+    #[test]
+    fn keeps_free_filter_that_removes_rows() {
+        let mut gate = new_gate();
+        assert_eq!(feed_n(&mut gate, 100, 0.5), 100);
+        assert_eq!(gate.pauses(), 0);
+        // A free filter that removes only 1% of the rows also stays on.
+        assert_eq!(feed_n(&mut gate, 10, 0.99), 10);
+        assert_eq!(gate.pauses(), 0);
+    }
+
+    #[test]
+    fn backoff_doubles_up_to_cap_and_resets() {
+        let mut gate = new_gate();
+        let mut observed = vec![];
+        assert_eq!(feed_n(&mut gate, 2, 1.0), 2);
+        for _ in 0..6 {
+            observed.push(skip_until_probe(&mut gate, 1.0));
+            // `skip_until_probe` evaluated the first batch of the probe
+            // window. One more batch closes the window.
+            assert_eq!(feed(&mut gate, 1.0), GateDecision::Evaluate);
+        }
+        assert_eq!(observed, vec![4, 8, 16, 32, 32, 32]);
+        assert_eq!(gate.pauses(), 7);
+
+        // A selective probe resets the backoff.
+        assert!(gate.is_paused());
+        let skipped = skip_until_probe(&mut gate, 0.1);
+        assert_eq!(skipped, 32);
+        assert_eq!(feed(&mut gate, 0.1), GateDecision::Evaluate);
+        assert!(!gate.is_paused());
+        assert_eq!(gate.backoff, 4);
+
+        // The next pause is short again.
+        assert_eq!(feed_n(&mut gate, 2, 1.0), 2);
+        assert_eq!(skip_until_probe(&mut gate, 1.0), 4);
+    }
+
+    fn dynamic_filter() -> (Arc<DynamicFilterPhysicalExpr>, Arc<dyn PhysicalExpr>) {
+        let column: Arc<dyn PhysicalExpr> = Arc::new(Column::new("a", 0));
+        let dynamic = Arc::new(DynamicFilterPhysicalExpr::new(vec![column], lit(true)));
+        let filter = Arc::clone(&dynamic) as Arc<dyn PhysicalExpr>;
+        (dynamic, filter)
+    }
+
+    fn a_gt(value: i32) -> Arc<dyn PhysicalExpr> {
+        Arc::new(BinaryExpr::new(
+            Arc::new(Column::new("a", 0)),
+            Operator::Gt,
+            lit(value),
+        ))
+    }
+
+    #[test]
+    fn generation_change_restarts_evaluation() {
+        let (dynamic, filter) = dynamic_filter();
+        let mut gate = gate_with(filter);
+
+        // Pause two times: the backoff grows to 16.
+        assert_eq!(feed_n(&mut gate, 2, 1.0), 2);
+        skip_until_probe(&mut gate, 1.0);
+        feed(&mut gate, 1.0);
+        assert!(gate.is_paused());
+        assert_eq!(gate.backoff, 16);
+
+        // A new generation of the filter ends the pause at once.
+        dynamic.update(a_gt(10)).unwrap();
+        assert_eq!(feed(&mut gate, 1.0), GateDecision::Evaluate);
+        assert!(!gate.is_paused());
+        assert_eq!(gate.backoff, 4);
+
+        // The new window has an empty history: one more batch decides.
+        assert_eq!(feed(&mut gate, 1.0), GateDecision::Evaluate);
+        assert!(gate.is_paused());
+        assert_eq!(skip_until_probe(&mut gate, 1.0), 4);
+    }
+
+    #[test]
+    fn static_filter_is_never_watched() {
+        // A filter without dynamic filters, and a filter whose dynamic
+        // filters are all complete, can not change: the gate never polls
+        // them and never resets.
+        let (dynamic, complete) = dynamic_filter();
+        dynamic.mark_complete();
+        for (filter, expected_complete) in [(static_filter(), false), (complete, true)] {
+            let mut gate = gate_with(filter);
+            match (&gate.tracking, expected_complete) {
+                (DynamicFilterTracking::Static, false)
+                | (DynamicFilterTracking::AllComplete, true) => {}
+                (other, _) => panic!("unexpected tracking {other:?}"),
+            }
+            assert!(gate.tracking.watcher().is_none());
+
+            assert_eq!(feed_n(&mut gate, 2, 1.0), 2);
+            assert!(gate.is_paused());
+            assert_eq!(skip_until_probe(&mut gate, 1.0), 4);
+            assert_eq!(feed(&mut gate, 1.0), GateDecision::Evaluate);
+            assert_eq!(skip_until_probe(&mut gate, 1.0), 8);
+            assert_eq!(gate.pauses(), 2);
+        }
+    }
+
+    #[test]
+    fn completed_filter_stops_being_watched() {
+        let (dynamic, filter) = dynamic_filter();
+        let mut gate = gate_with(filter);
+        assert!(gate.tracking.watcher().is_some());
+        assert_eq!(feed_n(&mut gate, 2, 1.0), 2);
+        assert!(gate.is_paused());
+
+        // The final update restarts the evaluation one time.
+        dynamic.update(a_gt(10)).unwrap();
+        dynamic.mark_complete();
+        assert_eq!(feed(&mut gate, 1.0), GateDecision::Evaluate);
+        assert!(!gate.is_paused());
+        // The tracker dropped the subscription of the complete filter.
+        assert!(gate.tracking.watcher().unwrap().is_exhausted());
+
+        // No more resets: the gate pauses and backs off as usual.
+        assert_eq!(feed(&mut gate, 1.0), GateDecision::Evaluate);
+        assert!(gate.is_paused());
+        assert_eq!(skip_until_probe(&mut gate, 1.0), 4);
+        assert_eq!(feed(&mut gate, 1.0), GateDecision::Evaluate);
+        assert_eq!(skip_until_probe(&mut gate, 1.0), 8);
+    }
+
+    #[test]
+    fn topk_like_filter_stays_on() {
+        // The filter gets tighter over time and changes often, like a TopK
+        // dynamic filter. At the start it removes almost no rows.
+        let (dynamic, filter) = dynamic_filter();
+        let mut gate = gate_with(filter);
+        for i in 0..100 {
+            if i % 2 == 0 {
+                dynamic.update(a_gt(i)).unwrap();
+            }
+            let pass_ratio = 1.0 - (i as f64 / 100.0);
+            assert_eq!(
+                feed(&mut gate, pass_ratio),
+                GateDecision::Evaluate,
+                "batch {i}"
+            );
+        }
+        assert_eq!(gate.pauses(), 0);
+    }
+
+    #[test]
+    fn skewed_input_probe_re_enables_filter() {
+        let mut gate = new_gate();
+        // Data that the filter does not remove first: the gate pauses with
+        // growing backoff.
+        assert_eq!(feed_n(&mut gate, 2, 1.0), 2);
+        for _ in 0..3 {
+            skip_until_probe(&mut gate, 1.0);
+            feed(&mut gate, 1.0);
+        }
+        assert!(gate.is_paused());
+
+        // Then the data becomes selective. A probe finds it.
+        let skipped = skip_until_probe(&mut gate, 0.05);
+        assert!(skipped <= 32);
+        assert_eq!(feed(&mut gate, 0.05), GateDecision::Evaluate);
+        assert!(!gate.is_paused());
+        // The filter stays on for the rest of the input.
+        assert_eq!(feed_n(&mut gate, 50, 0.05), 50);
+    }
+
+    /// Regression test: a paused gate must not change its backoff or its
+    /// counters for each skipped batch. Only decisions change them.
+    #[test]
+    fn paused_gate_does_not_grow_backoff_while_skipping() {
+        let mut gate = new_gate();
+        assert_eq!(feed_n(&mut gate, 2, 1.0), 2);
+        assert!(gate.is_paused());
+        let backoff = gate.backoff;
+        let pauses = gate.pauses();
+
+        for _ in 0..3 {
+            assert_eq!(gate.begin_batch(), GateDecision::Skip);
+            // A stray `record` during a pause is ignored.
+            gate.record(ROWS, 0, Duration::from_secs(1));
+            assert_eq!(gate.backoff, backoff);
+            assert_eq!(gate.pauses(), pauses);
+        }
+        assert_eq!(gate.begin_batch(), GateDecision::Skip);
+        assert_eq!(gate.backoff, backoff);
+        // The pause is over: the next batch is evaluated.
+        assert_eq!(gate.begin_batch(), GateDecision::Evaluate);
+    }
+
+    #[test]
+    fn empty_window_makes_no_decision() {
+        let mut gate = new_gate();
+        for _ in 0..10 {
+            assert_eq!(gate.begin_batch(), GateDecision::Evaluate);
+            gate.record(0, 0, Duration::from_micros(1));
+        }
+        assert_eq!(gate.pauses(), 0);
+    }
+
+    fn batch(values: Vec<Option<i32>>) -> RecordBatch {
+        let schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, true)]));
+        RecordBatch::try_new(schema, vec![Arc::new(Int32Array::from(values))]).unwrap()
+    }
+
+    fn col_gt(value: i32) -> Arc<dyn PhysicalExpr> {
+        let schema = Schema::new(vec![Field::new("a", DataType::Int32, true)]);
+        Arc::new(BinaryExpr::new(
+            col("a", &schema).unwrap(),
+            Operator::Gt,
+            lit(value),
+        ))
+    }
+
+    #[test]
+    fn evaluate_selective_filter() {
+        let mut gate = gate_with(col_gt(8));
+        let input = batch((0..10).map(Some).collect());
+        for _ in 0..10 {
+            let result = evaluate(&mut gate, &input).expect("evaluated");
+            assert_eq!(result.true_count(), 1);
+        }
+        assert_eq!(gate.pauses(), 0);
+    }
+
+    #[test]
+    fn evaluate_filter_that_removes_no_rows_skips() {
+        let mut gate = gate_with(col_gt(0));
+        let input = batch((1..=10).map(Some).collect());
+        assert!(evaluate(&mut gate, &input).is_some());
+        assert!(evaluate(&mut gate, &input).is_some());
+        for _ in 0..4 {
+            assert!(evaluate(&mut gate, &input).is_none());
+        }
+        assert!(evaluate(&mut gate, &input).is_some());
+    }
+
+    #[test]
+    fn evaluate_counts_null_as_not_passing() {
+        let mut gate = gate_with(col_gt(0));
+        // 9 of 10 rows are null: the filter removes them.
+        let mut values = vec![None; 9];
+        values.push(Some(5));
+        let input = batch(values);
+        for _ in 0..10 {
+            let result = evaluate(&mut gate, &input).expect("evaluated");
+            assert_eq!(result.null_count(), 9);
+        }
+        assert_eq!(gate.pauses(), 0);
+    }
+
+    #[test]
+    fn config_from_execution_options() {
+        let mut options = ExecutionOptions::default();
+        assert_eq!(
+            OptionalFilterGateConfig::from(&options),
+            OptionalFilterGateConfig::default()
+        );
+        options.optional_filter_min_saving_ns_per_row = 7.5;
+        let config = OptionalFilterGateConfig::from(&options);
+        assert_eq!(config.min_saving_ns_per_row, 7.5);
+        assert_eq!(config.sample_batches, 2);
+    }
+
+    // The tests below use the default `min_saving_ns_per_row` of 20 ns. For
+    // a window of 1000-row batches with pass ratio `p` and cost `c` ns for
+    // each row, the gate compares `c` with `(1 - p) * 20` ns for each row.
+
+    /// Like the dynamic filter of a hash join with a multi-column key: it
+    /// removes 93% of the rows, but costs 70 ns for each row. Each removed
+    /// row saves only 20 ns, thus 18.6 ns for each evaluated row.
+    #[test]
+    fn selective_but_expensive_filter_pauses() {
+        let mut gate = new_gate();
+        assert_eq!(feed_timed(&mut gate, 0.07, 70.0), GateDecision::Evaluate);
+        assert_eq!(feed_timed(&mut gate, 0.07, 70.0), GateDecision::Evaluate);
+        assert!(gate.is_paused());
+        assert_eq!(gate.pauses(), 1);
+
+        // The probes find the same cost: the pauses get longer.
+        assert_eq!(skip_until_probe(&mut gate, 0.07), 4);
+        assert_eq!(feed_timed(&mut gate, 0.07, 70.0), GateDecision::Evaluate);
+        // `skip_until_probe` evaluated the first batch with no time. The
+        // window costs 70 µs and saves 1860 * 20 ns = 37.2 µs: still too
+        // much.
+        assert!(gate.is_paused());
+        assert_eq!(skip_until_probe(&mut gate, 0.07), 8);
+    }
+
+    /// A bound check that costs 2 ns for each row and removes 30% of the
+    /// rows: it saves 6 ns for each row, thus it stays on.
+    #[test]
+    fn cheap_weakly_selective_filter_stays_on() {
+        let mut gate = new_gate();
+        for _ in 0..100 {
+            assert_eq!(feed_timed(&mut gate, 0.7, 2.0), GateDecision::Evaluate);
+        }
+        assert_eq!(gate.pauses(), 0);
+    }
+
+    /// Cheap bounds that cost 1 ns for each row and remove 12% of the rows:
+    /// they save 2.4 ns for each row, thus they stay on. There is no
+    /// separate rule for the fraction of rows that pass.
+    #[test]
+    fn cheap_bounds_that_remove_few_rows_stay_on() {
+        let mut gate = new_gate();
+        for _ in 0..100 {
+            assert_eq!(feed_timed(&mut gate, 0.88, 1.0), GateDecision::Evaluate);
+        }
+        assert_eq!(gate.pauses(), 0);
+    }
+
+    /// Like the dynamic filter of a TopK: a cheap comparison that removes
+    /// almost all rows. It stays on.
+    #[test]
+    fn cheap_very_selective_filter_stays_on() {
+        let mut gate = new_gate();
+        for _ in 0..100 {
+            assert_eq!(feed_timed(&mut gate, 0.001, 3.0), GateDecision::Evaluate);
+        }
+        assert_eq!(gate.pauses(), 0);
+    }
+
+    /// The consumer measures a larger saving (for example the Parquet scan
+    /// measures the decode time of the columns that the filter does not
+    /// read). The next probe turns the expensive filter on again.
+    #[test]
+    fn measured_saving_re_enables_filter_at_next_probe() {
+        let saving = Arc::new(MeasuredRowSaving::new());
+        let mut gate = new_gate().with_measured_saving(Arc::clone(&saving));
+        assert_eq!(gate.saving_ns_per_row(), 20.0);
+        assert_eq!(feed_timed(&mut gate, 0.07, 70.0), GateDecision::Evaluate);
+        assert_eq!(feed_timed(&mut gate, 0.07, 70.0), GateDecision::Evaluate);
+        assert!(gate.is_paused());
+
+        // Each removed row now saves 20 + 80 = 100 ns: 93 ns for each
+        // evaluated row, more than the cost of 70 ns with the margin.
+        saving.set_ns_per_row(80.0);
+        assert_eq!(gate.saving_ns_per_row(), 100.0);
+        // The pause does not end early.
+        for _ in 0..4 {
+            assert_eq!(feed_timed(&mut gate, 0.07, 70.0), GateDecision::Skip);
+        }
+        assert_eq!(feed_timed(&mut gate, 0.07, 70.0), GateDecision::Evaluate);
+        assert_eq!(feed_timed(&mut gate, 0.07, 70.0), GateDecision::Evaluate);
+        assert!(!gate.is_paused());
+        assert_eq!(gate.backoff, 4);
+        for _ in 0..20 {
+            assert_eq!(feed_timed(&mut gate, 0.07, 70.0), GateDecision::Evaluate);
+        }
+
+        // Invalid measured values are used as 0.
+        saving.set_ns_per_row(f64::NAN);
+        assert_eq!(gate.saving_ns_per_row(), 20.0);
+        saving.set_ns_per_row(-5.0);
+        assert_eq!(gate.saving_ns_per_row(), 20.0);
+    }
+
+    /// When the cost is near the saving, the gate keeps its state: a running
+    /// filter stays on and a paused filter stays paused.
+    #[test]
+    fn cost_check_has_hysteresis() {
+        // The filter removes 50% of the rows: the saving is 10 ns for each
+        // evaluated row. A cost of 10.5 ns is between 0.9 and 1.1 times the
+        // saving.
+        let mut gate = new_gate();
+        for _ in 0..20 {
+            assert_eq!(feed_timed(&mut gate, 0.5, 10.5), GateDecision::Evaluate);
+        }
+        assert!(!gate.is_paused());
+
+        // Pause it with a larger cost.
+        assert_eq!(feed_timed(&mut gate, 0.5, 30.0), GateDecision::Evaluate);
+        assert_eq!(feed_timed(&mut gate, 0.5, 30.0), GateDecision::Evaluate);
+        assert!(gate.is_paused());
+        // A probe with the cost near the saving does not turn it on.
+        for _ in 0..4 {
+            assert_eq!(feed_timed(&mut gate, 0.5, 10.5), GateDecision::Skip);
+        }
+        assert_eq!(feed_timed(&mut gate, 0.5, 10.5), GateDecision::Evaluate);
+        assert_eq!(feed_timed(&mut gate, 0.5, 10.5), GateDecision::Evaluate);
+        assert!(gate.is_paused());
+        assert_eq!(gate.pauses(), 2);
+        // A probe with a clearly lower cost turns it on.
+        for _ in 0..8 {
+            assert_eq!(feed_timed(&mut gate, 0.5, 8.5), GateDecision::Skip);
+        }
+        assert_eq!(feed_timed(&mut gate, 0.5, 8.5), GateDecision::Evaluate);
+        assert_eq!(feed_timed(&mut gate, 0.5, 8.5), GateDecision::Evaluate);
+        assert!(!gate.is_paused());
+    }
+
+    /// A clock that moves by a fixed step each time it is read.
+    #[derive(Debug)]
+    struct SteppingClock {
+        now: AtomicU64,
+        step: u64,
+    }
+
+    impl Clock for SteppingClock {
+        fn now_nanos(&self) -> u64 {
+            self.now.fetch_add(self.step, Ordering::Relaxed)
+        }
+    }
+
+    /// A consumer measures the time with the clock of the gate.
+    #[test]
+    fn consumer_measures_time_with_gate_clock() {
+        // Each evaluation takes 10 µs for 10 rows: 1000 ns for each row.
+        let clock = Arc::new(SteppingClock {
+            now: AtomicU64::new(0),
+            step: 10_000,
+        });
+        let mut gate =
+            OptionalFilterGate::new(col_gt(8), OptionalFilterGateConfig::default())
+                .with_clock(clock);
+        let input = batch((0..10).map(Some).collect());
+        // The filter removes 90% of the rows, but it costs 1000 ns for each
+        // row, and each removed row saves 20 ns.
+        assert!(evaluate(&mut gate, &input).is_some());
+        assert!(evaluate(&mut gate, &input).is_some());
+        assert!(gate.is_paused());
+        assert!(evaluate(&mut gate, &input).is_none());
+    }
+}
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt b/datafusion/sqllogictest/test_files/information_schema.slt
index 113f63c71257c..1ecc7a97d310e 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -231,6 +231,7 @@ datafusion.execution.max_spill_file_size_bytes 134217728
 datafusion.execution.meta_fetch_concurrency 32
 datafusion.execution.minimum_parallel_output_files 4
 datafusion.execution.objectstore_writer_buffer_size 10485760
+datafusion.execution.optional_filter_min_saving_ns_per_row 20
 datafusion.execution.parquet.allow_single_file_parallelism true
 datafusion.execution.parquet.binary_as_string false
 datafusion.execution.parquet.bloom_filter_fpp NULL
@@ -392,6 +393,7 @@ datafusion.execution.max_spill_file_size_bytes 134217728 Maximum size in bytes f
 datafusion.execution.meta_fetch_concurrency 32 Number of files to read in parallel when inferring schema and statistics
 datafusion.execution.minimum_parallel_output_files 4 Guarantees a minimum level of output files running in parallel. RecordBatches will be distributed in round robin fashion to each parallel writer. Each writer is closed and a new file opened once soft_max_rows_per_output_file is reached.
 datafusion.execution.objectstore_writer_buffer_size 10485760 Size (bytes) of data buffer DataFusion uses when writing output files. This affects the size of the data chunks that are uploaded to remote object stores (e.g. AWS S3). If very large (>= 100 GiB) output files are being written, it may be necessary to increase this size to avoid errors from the remote end point.
+datafusion.execution.optional_filter_min_saving_ns_per_row 20 The assumed work, in nanoseconds, that each row removed by an optional filter saves downstream. Optional filters are filters that are not needed for correctness, such as the dynamic filters that hash joins and TopK push down into scans. When an operator evaluates optional filters adaptively, it pauses an optional filter whose evaluation costs more than the work that it saves. Consumers that can measure the saving (the Parquet scan) add their measured decode cost. The default is about the cost of a hash table probe for one row. The best value depends on the hardware.
 datafusion.execution.parquet.allow_single_file_parallelism true (writing) Controls whether DataFusion will attempt to speed up writing parquet files by serializing them in parallel. Each column in each row group in each output file are serialized in parallel leveraging a maximum possible core count of n_files\*n_row_groups\*n_columns.
 datafusion.execution.parquet.binary_as_string false (reading) If true, parquet reader will read columns of `Binary/LargeBinary` with `Utf8`, and `BinaryView` with `Utf8View`. Parquet files generated by some legacy writers do not correctly set the UTF8 flag for strings, causing string columns to be loaded as BLOB instead. The parquet reader has special optimizations for `Utf8` validation, so reading such columns as strings is significantly faster than reading them as binary and then casting to string.
 datafusion.execution.parquet.bloom_filter_fpp NULL (writing) Sets bloom filter false positive probability. If NULL, uses default parquet writer setting
diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md
index ef4ea00129f29..178e3d109466c 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -145,6 +145,7 @@ The following configuration settings are available:
 | datafusion.execution.objectstore_writer_buffer_size                     | 10485760                  | Size (bytes) of data buffer DataFusion uses when writing output files. This affects the size of the data chunks that are uploaded to remote object stores (e.g. AWS S3). If very large (>= 100 GiB) output files are being written, it may be necessary to increase this size to avoid errors from the remote end point.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | datafusion.execution.enable_ansi_mode                                   | false                     | Whether to enable ANSI SQL mode. The flag is experimental and relevant only for DataFusion Spark built-in functions When `enable_ansi_mode` is set to `true`, the query engine follows ANSI SQL semantics for expressions, casting, and error handling. This means: - **Strict type coercion rules:** implicit casts between incompatible types are disallowed. - **Standard SQL arithmetic behavior:** operations such as division by zero, numeric overflow, or invalid casts raise runtime errors rather than returning `NULL` or adjusted values. - **Consistent ANSI behavior** for string concatenation, comparisons, and `NULL` handling. When `enable_ansi_mode` is `false` (the default), the engine uses a more permissive, non-ANSI mode designed for user convenience and backward compatibility. In this mode: - Implicit casts between types are allowed (e.g., string to integer when possible). - Arithmetic operations are more lenient — for example, `abs()` on the minimum representable integer value returns the input value instead of raising overflow. - Division by zero or invalid casts may return `NULL` instead of failing. # Default `false` — ANSI SQL mode is disabled by default.                          |
 | datafusion.execution.hash_join_buffering_capacity                       | 0                         | How many bytes to buffer in the probe side of hash joins while the build side is concurrently being built. Without this, hash joins will wait until the full materialization of the build side before polling the probe side. This is useful in scenarios where the query is not completely CPU bounded, allowing to do some early work concurrently and reducing the latency of the query. Note that when hash join buffering is enabled, the probe side will start eagerly polling data, not giving time for the producer side of dynamic filters to produce any meaningful predicate. Queries with dynamic filters might see performance degradation. Disabled by default, set to a number greater than 0 for enabling it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| datafusion.execution.optional_filter_min_saving_ns_per_row              | 20                        | The assumed work, in nanoseconds, that each row removed by an optional filter saves downstream. Optional filters are filters that are not needed for correctness, such as the dynamic filters that hash joins and TopK push down into scans. When an operator evaluates optional filters adaptively, it pauses an optional filter whose evaluation costs more than the work that it saves. Consumers that can measure the saving (the Parquet scan) add their measured decode cost. The default is about the cost of a hash table probe for one row. The best value depends on the hardware.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
 | datafusion.optimizer.enable_distinct_aggregation_soft_limit             | true                      | When set to true, the optimizer will push a limit operation into grouped aggregations which have no aggregate expressions, as a soft limit, emitting groups once the limit is reached, before all rows in the group are read.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | datafusion.optimizer.enable_round_robin_repartition                     | true                      | When set to true, the physical plan optimizer will try to add round robin repartitioning to increase parallelism to leverage more CPU cores                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | datafusion.optimizer.enable_topk_aggregation                            | true                      | When set to true, the optimizer will attempt to perform limit operations during aggregations, if possible                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

From 6ae18bcd6c93919a889475a98361945757b812bc Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 11:41:04 -0500
Subject: [PATCH 10/42] feat(gate): add a measured overhead for each evaluated
 row to the cost

A consumer can now give the gate a fixed cost for each evaluated row in
addition to the evaluation time, with
`MeasuredRowSaving::set_overhead_ns_per_row`. The gate adds it to the cost
of each window. The Parquet scan uses it for the fixed cost of a row filter
stage, which is larger than the evaluation time of a cheap predicate.

PR: #25674

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../physical-expr/src/optional_filter_gate.rs | 115 ++++++++++++++----
 1 file changed, 92 insertions(+), 23 deletions(-)

diff --git a/datafusion/physical-expr/src/optional_filter_gate.rs b/datafusion/physical-expr/src/optional_filter_gate.rs
index 8275430baca00..7bcc8e4f7edd3 100644
--- a/datafusion/physical-expr/src/optional_filter_gate.rs
+++ b/datafusion/physical-expr/src/optional_filter_gate.rs
@@ -65,20 +65,24 @@
 //! rules is true:
 //!
 //! 1. The filter removed no rows in the window.
-//! 2. The evaluation time of the window (`cost_ns`) is larger than the work
-//!    that the removed rows save (`saving_ns`):
+//! 2. The cost of the window (`cost_ns`) is larger than the work that the
+//!    removed rows save (`saving_ns`):
 //!
 //!    ```text
+//!    cost_ns   = evaluation time + rows_in * measured overhead
 //!    saving_ns = (rows_in - rows_out) * saving_ns_per_row
 //!    saving_ns_per_row = min_saving_ns_per_row + measured saving
 //!    ```
 //!
 //!    `min_saving_ns_per_row` comes from the configuration. It is the work
 //!    that a removed row saves after the filter, for example a hash table
-//!    probe in a join. The *measured saving* is optional: a consumer that
-//!    can measure more work that a removed row saves (the Parquet scan
-//!    measures the decode time of the columns that the filter does not read)
-//!    gives it in a shared [`MeasuredRowSaving`] and updates it at any time.
+//!    probe in a join. The *measured saving* and the *measured overhead* are
+//!    optional: a consumer that can measure more work that a removed row
+//!    saves (the Parquet scan measures the decode time of the columns that
+//!    the filter does not read), or a fixed cost for each evaluated row in
+//!    addition to the evaluation time (the Parquet scan has a cost for each
+//!    row filter stage), gives them in a shared [`MeasuredRowSaving`] and
+//!    updates them at any time.
 //!
 //!    To prevent a filter from switching on and off when the cost and the
 //!    saving are almost equal, this rule has a margin: a running filter is
@@ -192,22 +196,42 @@ pub enum GateDecision {
     Skip,
 }
 
-/// Work, in nanoseconds, that each row removed by an optional filter saves,
-/// as measured by the consumer of the filter. The gate adds it to
-/// [`OptionalFilterGateConfig::min_saving_ns_per_row`].
+/// The terms of the cost rule that the consumer of an optional filter
+/// measures:
+///
+/// * The *saving*: work, in nanoseconds, that each row removed by the filter
+///   saves. The gate adds it to
+///   [`OptionalFilterGateConfig::min_saving_ns_per_row`]. For example, the
+///   Parquet scan sets it to the time to decode the columns that the filter
+///   does not read, for each removed row that the decoder can skip.
+/// * The *overhead*: work, in nanoseconds, that the consumer does for each
+///   evaluated row because it evaluates the filter, in addition to the
+///   evaluation time. The gate adds it to the cost of each window. For
+///   example, the Parquet scan sets it to the fixed cost of a row filter
+///   stage when the filter is a row filter predicate.
 ///
 /// The consumer creates one value, gives a clone of the [`Arc`] to the gate
 /// with [`OptionalFilterGate::with_measured_saving`], and updates it at any
-/// time with [`Self::set_ns_per_row`]. The gate reads it at each decision.
-/// For example, the Parquet scan sets it to the time to decode the columns
-/// that the filter does not read, for each row.
+/// time with [`Self::set_ns_per_row`] and [`Self::set_overhead_ns_per_row`].
+/// The gate reads it at each decision.
 ///
-/// The value is an `f64` in an [`AtomicU64`], thus reads and updates are
+/// Each value is an `f64` in an [`AtomicU64`], thus reads and updates are
 /// cheap and lock-free.
 #[derive(Debug, Default)]
 pub struct MeasuredRowSaving {
-    /// The bits of the `f64` value.
+    /// The bits of the `f64` saving for each removed row.
     ns_per_row_bits: AtomicU64,
+    /// The bits of the `f64` overhead for each evaluated row.
+    overhead_ns_per_row_bits: AtomicU64,
+}
+
+/// `value` if it is finite and not negative, else 0.
+fn non_negative(value: f64) -> f64 {
+    if value.is_finite() {
+        value.max(0.0)
+    } else {
+        0.0
+    }
 }
 
 impl MeasuredRowSaving {
@@ -219,19 +243,26 @@ impl MeasuredRowSaving {
     /// Sets the measured saving for each removed row, in nanoseconds.
     /// Values that are negative or not finite are used as 0.
     pub fn set_ns_per_row(&self, ns_per_row: f64) {
-        let ns_per_row = if ns_per_row.is_finite() {
-            ns_per_row.max(0.0)
-        } else {
-            0.0
-        };
         self.ns_per_row_bits
-            .store(ns_per_row.to_bits(), Ordering::Relaxed);
+            .store(non_negative(ns_per_row).to_bits(), Ordering::Relaxed);
     }
 
     /// The measured saving for each removed row, in nanoseconds.
     pub fn ns_per_row(&self) -> f64 {
         f64::from_bits(self.ns_per_row_bits.load(Ordering::Relaxed))
     }
+
+    /// Sets the measured overhead for each evaluated row, in nanoseconds.
+    /// Values that are negative or not finite are used as 0.
+    pub fn set_overhead_ns_per_row(&self, ns_per_row: f64) {
+        self.overhead_ns_per_row_bits
+            .store(non_negative(ns_per_row).to_bits(), Ordering::Relaxed);
+    }
+
+    /// The measured overhead for each evaluated row, in nanoseconds.
+    pub fn overhead_ns_per_row(&self) -> f64 {
+        f64::from_bits(self.overhead_ns_per_row_bits.load(Ordering::Relaxed))
+    }
 }
 
 /// The state of an [`OptionalFilterGate`].
@@ -322,8 +353,9 @@ impl OptionalFilterGate {
         self
     }
 
-    /// Adds `saving` to [`OptionalFilterGateConfig::min_saving_ns_per_row`]
-    /// at each decision. See [`MeasuredRowSaving`].
+    /// Adds the saving of `saving` to
+    /// [`OptionalFilterGateConfig::min_saving_ns_per_row`] and its overhead
+    /// to the cost at each decision. See [`MeasuredRowSaving`].
     pub fn with_measured_saving(mut self, saving: Arc<MeasuredRowSaving>) -> Self {
         self.measured_saving = Some(saving);
         self
@@ -350,6 +382,14 @@ impl OptionalFilterGate {
         self.config.min_saving_ns_per_row + measured
     }
 
+    /// The work, in nanoseconds for each evaluated row, that the gate adds
+    /// to the evaluation time now: the measured overhead.
+    pub fn overhead_ns_per_row(&self) -> f64 {
+        self.measured_saving
+            .as_ref()
+            .map_or(0.0, |saving| saving.overhead_ns_per_row())
+    }
+
     /// Call before each batch. Returns if the caller must evaluate the
     /// filter on the batch or skip it.
     ///
@@ -447,7 +487,8 @@ impl OptionalFilterGate {
             return true;
         }
         // The filter costs more than it saves.
-        let cost_ns = window.nanos as f64;
+        let cost_ns =
+            window.nanos as f64 + window.rows_in as f64 * self.overhead_ns_per_row();
         let saving_ns = rows_removed as f64 * self.saving_ns_per_row();
         if self.probing {
             // Turn the filter on again only if it is clearly worth its cost.
@@ -936,6 +977,34 @@ mod tests {
         assert_eq!(gate.saving_ns_per_row(), 20.0);
     }
 
+    /// The consumer measures an overhead for each evaluated row (for example
+    /// the fixed cost of a Parquet row filter stage). A filter that is worth
+    /// its evaluation time alone is paused when the overhead is added.
+    #[test]
+    fn measured_overhead_adds_to_cost() {
+        let saving = Arc::new(MeasuredRowSaving::new());
+        let mut gate = new_gate().with_measured_saving(Arc::clone(&saving));
+        // Removes 50% of the rows: saves 10 ns for each evaluated row. The
+        // evaluation costs 5 ns for each row.
+        for _ in 0..10 {
+            assert_eq!(feed_timed(&mut gate, 0.5, 5.0), GateDecision::Evaluate);
+        }
+        assert!(!gate.is_paused());
+
+        // With 8 ns of overhead, the cost is 13 ns for each row.
+        saving.set_overhead_ns_per_row(8.0);
+        assert_eq!(gate.overhead_ns_per_row(), 8.0);
+        assert_eq!(feed_timed(&mut gate, 0.5, 5.0), GateDecision::Evaluate);
+        assert_eq!(feed_timed(&mut gate, 0.5, 5.0), GateDecision::Evaluate);
+        assert!(gate.is_paused());
+
+        // Invalid measured values are used as 0.
+        saving.set_overhead_ns_per_row(f64::INFINITY);
+        assert_eq!(gate.overhead_ns_per_row(), 0.0);
+        saving.set_overhead_ns_per_row(-1.0);
+        assert_eq!(gate.overhead_ns_per_row(), 0.0);
+    }
+
     /// When the cost is near the saving, the gate keeps its state: a running
     /// filter stays on and a paused filter stays paused.
     #[test]

From ac838c45be4f1d720fef7dd5c3b2308f995b722f Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 12:10:21 -0500
Subject: [PATCH 11/42] feat(gate): share pauses between the gates of one plan
 site

Each gate paid for its own first window and its own probes. A scan opens
its files at the same time, thus a filter that removes nothing cost one
window in each file (TPC-H Q9: five join filters that remove no rows and a
CASE routing filter that costs 83 ns for each row, in each of 12 files).

`SharedGateVerdict` holds the last pause (or end of a pause) of the gates
of one plan site in one atomic word. A gate without evidence of its own
(before its first decision, after a filter change, after a pause) uses a
pause that another gate published after the last verdict that it saw: a
new gate starts paused, and a gate in its first window or in a probe
window stops and pauses. Thus usually only one gate probes after a pause.
A gate that keeps the filter does not use the pauses of other gates
(skewed data). A filter change clears the shared pause, because it was
measured on the old filter.

PR: #25674

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../physical-expr/src/optional_filter_gate.rs | 309 +++++++++++++++++-
 1 file changed, 306 insertions(+), 3 deletions(-)

diff --git a/datafusion/physical-expr/src/optional_filter_gate.rs b/datafusion/physical-expr/src/optional_filter_gate.rs
index 7bcc8e4f7edd3..0d91bf9b7888f 100644
--- a/datafusion/physical-expr/src/optional_filter_gate.rs
+++ b/datafusion/physical-expr/src/optional_filter_gate.rs
@@ -26,8 +26,9 @@
 //! can be larger than the benefit.
 //!
 //! [`OptionalFilterGate`] decides, batch by batch, if a stream evaluates the
-//! filter or skips it. Each stream has its own gate, and gates do not share
-//! state.
+//! filter or skips it. Each stream has its own gate. The gates of one plan
+//! site (for example all files and partitions of one scan) can share their
+//! pauses, see "Shared verdict".
 //!
 //! # State machine
 //!
@@ -110,6 +111,32 @@
 //!
 //! [`DynamicFilterPhysicalExpr`]: crate::expressions::DynamicFilterPhysicalExpr
 //! [`DynamicFilterTracker::changed`]: crate::expressions::DynamicFilterTracker::changed
+//!
+//! # Shared verdict
+//!
+//! Each gate pays for at least one window before its first decision. A scan
+//! opens many files at the same time, and a filter that removes no rows
+//! (for example a hash join filter on a column that has only matching
+//! values) then costs one window in each file. Also each gate probes on its
+//! own after each pause.
+//!
+//! Thus the gates of one plan site can share a [`SharedGateVerdict`] (see
+//! [`OptionalFilterGate::with_shared_verdict`]). Each gate publishes its
+//! pauses and the end of its pauses there. A gate that has no evidence of
+//! its own that the filter is worth its cost (before its first decision,
+//! after a change of the filter, and after a decision to pause) uses a
+//! pause that another gate published after the last shared verdict that it
+//! saw:
+//!
+//! * A new gate starts paused if the shared verdict is a pause.
+//! * A gate in its first window, or in a probe window after a pause, stops
+//!   the window and pauses. Thus after a pause of all gates, usually only
+//!   the first gate at the end of its pause probes the filter.
+//!
+//! A gate that keeps the filter does not use the pauses of other gates:
+//! with skewed data the filter can be worth its cost for some files only.
+//! A gate that sees a change of the filter clears a shared pause, because
+//! it was measured on the old filter, and starts again without evidence.
 
 use std::sync::Arc;
 use std::sync::atomic::{AtomicU64, Ordering};
@@ -265,6 +292,89 @@ impl MeasuredRowSaving {
     }
 }
 
+/// The last pause (or end of a pause) that the gates of one plan site
+/// published, see "Shared verdict" in the [module documentation](self).
+///
+/// Create one value for each plan site (for example each optional filter of
+/// one scan), and give a clone of the [`Arc`] to each gate of the site with
+/// [`OptionalFilterGate::with_shared_verdict`].
+///
+/// The value is one [`AtomicU64`], thus it is lock-free. Gates write it only
+/// at decisions that pause the filter or end a pause, and read it before a
+/// batch only while they have no evidence of their own.
+#[derive(Debug, Default)]
+pub struct SharedGateVerdict {
+    /// A [`Verdict`] and its sequence number, see [`Verdict::pack`].
+    word: AtomicU64,
+}
+
+/// One published verdict of a [`SharedGateVerdict`].
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum Verdict {
+    /// The filter is not paused (or nothing was published yet).
+    Keep,
+    /// A gate paused the filter for this number of batches.
+    Pause(usize),
+}
+
+impl Verdict {
+    /// Largest pause length that fits in the packed word.
+    const MAX_BATCHES: usize = u32::MAX as usize;
+
+    /// Packs the verdict with the sequence number `seq`: the sequence
+    /// number in the low 32 bits, and the pause length (0 for
+    /// [`Verdict::Keep`]) in the high 32 bits.
+    fn pack(self, seq: u32) -> u64 {
+        let batches = match self {
+            Self::Keep => 0,
+            Self::Pause(batches) => batches.clamp(1, Self::MAX_BATCHES) as u64,
+        };
+        (batches << 32) | u64::from(seq)
+    }
+
+    /// The verdict and the sequence number in `word`.
+    fn unpack(word: u64) -> (Self, u32) {
+        let verdict = match (word >> 32) as usize {
+            0 => Self::Keep,
+            batches => Self::Pause(batches),
+        };
+        (verdict, word as u32)
+    }
+}
+
+impl SharedGateVerdict {
+    /// Creates a shared verdict without any published pause.
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    /// True if the last published verdict is a pause.
+    pub fn is_paused(&self) -> bool {
+        matches!(self.load().0, Verdict::Pause(_))
+    }
+
+    /// The current verdict and its sequence number.
+    fn load(&self) -> (Verdict, u32) {
+        Verdict::unpack(self.word.load(Ordering::Acquire))
+    }
+
+    /// Publishes `verdict` if `replace` returns true for the current
+    /// verdict. Returns the sequence number of the published verdict.
+    fn publish_if(
+        &self,
+        verdict: Verdict,
+        replace: impl Fn(Verdict) -> bool,
+    ) -> Option<u32> {
+        self.word
+            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |word| {
+                let (current, seq) = Verdict::unpack(word);
+                replace(current).then(|| verdict.pack(seq.wrapping_add(1)))
+            })
+            .ok()
+            .map(|previous| (previous as u32).wrapping_add(1))
+    }
+}
+
 /// The state of an [`OptionalFilterGate`].
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 enum GateState {
@@ -320,6 +430,17 @@ pub struct OptionalFilterGate {
     /// True after `begin_batch` returned `Evaluate` and before `record`.
     awaiting_record: bool,
     pauses: usize,
+    /// The verdict that this gate shares with the other gates of its plan
+    /// site, if any.
+    shared: Option<Arc<SharedGateVerdict>>,
+    /// Sequence number of the last shared verdict that this gate published
+    /// or saw.
+    shared_seq: u32,
+    /// True while this gate has no evidence of its own that the filter is
+    /// worth its cost: before its first decision, after a change of the
+    /// filter, and after a decision that paused the filter. Then it uses
+    /// the shared pauses of the other gates.
+    uses_shared_pauses: bool,
 }
 
 impl OptionalFilterGate {
@@ -344,9 +465,25 @@ impl OptionalFilterGate {
             backoff: config.initial_pause_batches,
             awaiting_record: false,
             pauses: 0,
+            shared: None,
+            shared_seq: 0,
+            uses_shared_pauses: true,
         }
     }
 
+    /// Shares the pauses of this gate with the other gates of the same plan
+    /// site, see "Shared verdict" in the [module documentation](self). If
+    /// the shared verdict is a pause, the gate starts paused.
+    pub fn with_shared_verdict(mut self, shared: Arc<SharedGateVerdict>) -> Self {
+        let (verdict, seq) = shared.load();
+        self.shared_seq = seq;
+        if let Verdict::Pause(batches) = verdict {
+            self.start_pause(batches);
+        }
+        self.shared = Some(shared);
+        self
+    }
+
     /// Uses `clock` as the clock of this gate, see [`Self::clock`].
     pub fn with_clock(mut self, clock: Arc<dyn Clock>) -> Self {
         self.clock = clock;
@@ -405,6 +542,17 @@ impl OptionalFilterGate {
             self.state = GateState::new_window();
             self.probing = false;
             self.backoff = self.config.initial_pause_batches;
+            self.uses_shared_pauses = true;
+            if let Some(shared) = &self.shared {
+                // A shared pause was measured on the old filter.
+                let paused = |verdict| matches!(verdict, Verdict::Pause(_));
+                if let Some(seq) = shared.publish_if(Verdict::Keep, paused) {
+                    self.shared_seq = seq;
+                }
+            }
+        }
+        if !self.is_paused() {
+            self.use_shared_pause();
         }
 
         match &mut self.state {
@@ -472,11 +620,48 @@ impl OptionalFilterGate {
             return;
         }
         if self.should_pause(&window) {
-            self.start_pause(self.backoff);
+            let batches = self.backoff;
+            self.start_pause(batches);
+            self.uses_shared_pauses = true;
+            self.publish(Verdict::Pause(batches));
         } else {
             self.state = GateState::new_window();
             self.probing = false;
             self.backoff = self.config.initial_pause_batches;
+            if std::mem::take(&mut self.uses_shared_pauses) {
+                // The first decision, or the end of a pause.
+                self.publish(Verdict::Keep);
+            }
+        }
+    }
+
+    /// Before a batch that the gate would evaluate: if this gate uses the
+    /// shared pauses and another gate published a pause after the last
+    /// shared verdict that this gate saw, pauses the filter for the same
+    /// length. The current window is dropped.
+    fn use_shared_pause(&mut self) {
+        if !self.uses_shared_pauses {
+            return;
+        }
+        let Some(shared) = &self.shared else {
+            return;
+        };
+        let (verdict, seq) = shared.load();
+        if seq == self.shared_seq {
+            return;
+        }
+        self.shared_seq = seq;
+        if let Verdict::Pause(batches) = verdict {
+            self.start_pause(batches);
+        }
+    }
+
+    /// Publishes `verdict` to the shared verdict, if any.
+    fn publish(&mut self, verdict: Verdict) {
+        if let Some(shared) = &self.shared {
+            self.shared_seq = shared
+                .publish_if(verdict, |_| true)
+                .expect("an unconditional update always succeeds");
         }
     }
 
@@ -1039,6 +1224,124 @@ mod tests {
         assert!(!gate.is_paused());
     }
 
+    fn shared_gate(
+        filter: Arc<dyn PhysicalExpr>,
+        shared: &Arc<SharedGateVerdict>,
+    ) -> OptionalFilterGate {
+        gate_with(filter).with_shared_verdict(Arc::clone(shared))
+    }
+
+    #[test]
+    fn verdict_pack_round_trip() {
+        for verdict in [Verdict::Keep, Verdict::Pause(1), Verdict::Pause(32)] {
+            assert_eq!(Verdict::unpack(verdict.pack(7)), (verdict, 7));
+            assert_eq!(Verdict::unpack(verdict.pack(u32::MAX)), (verdict, u32::MAX));
+        }
+        // Pause lengths are at least 1 and at most `MAX_BATCHES`.
+        assert_eq!(
+            Verdict::unpack(Verdict::Pause(0).pack(1)).0,
+            Verdict::Pause(1)
+        );
+        assert_eq!(
+            Verdict::unpack(Verdict::Pause(usize::MAX).pack(1)).0,
+            Verdict::Pause(Verdict::MAX_BATCHES)
+        );
+    }
+
+    /// A new gate starts from the shared pause of the other gates.
+    #[test]
+    fn new_gate_starts_with_shared_pause() {
+        let shared = Arc::new(SharedGateVerdict::new());
+        let mut first = shared_gate(static_filter(), &shared);
+        assert!(!first.is_paused());
+        assert_eq!(feed_n(&mut first, 2, 1.0), 2);
+        assert!(first.is_paused());
+        assert!(shared.is_paused());
+
+        // A new gate starts paused for the same length, then probes.
+        let mut second = shared_gate(static_filter(), &shared);
+        assert!(second.is_paused());
+        assert_eq!(second.pauses(), 1);
+        assert_eq!(skip_until_probe(&mut second, 0.5), 4);
+        // The probe keeps the filter: new gates evaluate again.
+        assert_eq!(feed(&mut second, 0.5), GateDecision::Evaluate);
+        assert!(!second.is_paused());
+        assert!(!shared.is_paused());
+        assert!(!shared_gate(static_filter(), &shared).is_paused());
+    }
+
+    /// Gates that start at the same time: a gate in its first window uses
+    /// the pause that another gate published, instead of the rest of its
+    /// own window, and only one gate probes after the pause.
+    #[test]
+    fn first_window_and_probe_use_shared_pause() {
+        let shared = Arc::new(SharedGateVerdict::new());
+        let mut gates: Vec<_> = (0..4)
+            .map(|_| shared_gate(static_filter(), &shared))
+            .collect();
+        // All gates evaluate their first batch.
+        for gate in &mut gates {
+            assert_eq!(feed(gate, 1.0), GateDecision::Evaluate);
+        }
+        // The first gate completes its window and pauses.
+        assert_eq!(feed(&mut gates[0], 1.0), GateDecision::Evaluate);
+        assert!(gates[0].is_paused());
+        // The others do not evaluate the second batch of their window.
+        for gate in &mut gates[1..] {
+            assert_eq!(feed(gate, 1.0), GateDecision::Skip);
+            assert!(gate.is_paused());
+        }
+
+        // Only the first gate probes at the end of its pause. The others
+        // use its verdict: a pause of 8 batches.
+        assert_eq!(skip_until_probe(&mut gates[0], 1.0), 4);
+        assert_eq!(feed(&mut gates[0], 1.0), GateDecision::Evaluate);
+        assert!(gates[0].is_paused());
+        for gate in &mut gates[1..] {
+            let skipped = (0..20)
+                .take_while(|_| feed(gate, 1.0) == GateDecision::Skip)
+                .count();
+            // 3 more batches of the first pause, then 8.
+            assert_eq!(skipped, 3 + 8);
+            assert_eq!(gate.pauses(), 2);
+        }
+    }
+
+    /// A gate that keeps the filter does not use the pauses of other gates
+    /// (skewed data).
+    #[test]
+    fn running_gate_ignores_shared_pause() {
+        let shared = Arc::new(SharedGateVerdict::new());
+        let mut selective = shared_gate(static_filter(), &shared);
+        assert_eq!(feed_n(&mut selective, 2, 0.1), 2);
+        assert!(!selective.is_paused());
+
+        let mut other = shared_gate(static_filter(), &shared);
+        assert_eq!(feed_n(&mut other, 2, 1.0), 2);
+        assert!(other.is_paused());
+        assert!(shared.is_paused());
+
+        assert_eq!(feed_n(&mut selective, 20, 0.1), 20);
+        assert_eq!(selective.pauses(), 0);
+    }
+
+    /// A change of the filter clears the shared pause: it was measured on
+    /// the old filter.
+    #[test]
+    fn change_clears_shared_pause() {
+        let (dynamic, filter) = dynamic_filter();
+        let shared = Arc::new(SharedGateVerdict::new());
+        let mut gate = shared_gate(Arc::clone(&filter), &shared);
+        assert_eq!(feed_n(&mut gate, 2, 1.0), 2);
+        assert!(shared.is_paused());
+
+        dynamic.update(a_gt(1)).unwrap();
+        // The gate sees the change at its next batch.
+        assert_eq!(feed(&mut gate, 0.5), GateDecision::Evaluate);
+        assert!(!shared.is_paused());
+        assert!(!shared_gate(filter, &shared).is_paused());
+    }
+
     /// A clock that moves by a fixed step each time it is read.
     #[derive(Debug)]
     struct SteppingClock {

From 36c73a0d6a1d92809204ac04cc332b718cf79f46 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 12:44:36 -0500
Subject: [PATCH 12/42] feat(gate): decide only on windows of at least
 MIN_OBSERVED_ROWS rows

A gate decided after `sample_batches` batches, whatever their size. After
a selective row filter a batch can have 2 to 7 rows, and the fixed cost
of each call then looks like 600 to 8000 ns for each row: ClickBench Q23
paused the TopK filter on `EventTime` (0.4 ns for each row on full
batches) on such windows, and the shared verdict spread these pauses to
the other files.

All decisions (pause, keep, probe) now need a window of at least
`sample_batches` batches and `MIN_OBSERVED_ROWS` rows. The constant moves
to `filter_stats`, so that the gate and the Parquet filter placement use
the same sample size. All published shared pauses come from such windows.

PR: #25674

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/physical-expr/src/filter_stats.rs  |  9 +++
 .../physical-expr/src/optional_filter_gate.rs | 64 ++++++++++++++-----
 2 files changed, 57 insertions(+), 16 deletions(-)

diff --git a/datafusion/physical-expr/src/filter_stats.rs b/datafusion/physical-expr/src/filter_stats.rs
index f5f5c4f03f8b1..0ea219f4166e0 100644
--- a/datafusion/physical-expr/src/filter_stats.rs
+++ b/datafusion/physical-expr/src/filter_stats.rs
@@ -106,6 +106,15 @@ impl Clock for ManualClock {
     }
 }
 
+/// Minimum number of evaluated rows before an adaptive decision uses the
+/// measurements of a filter. It is one batch of the default
+/// `datafusion.execution.batch_size`. With fewer rows, the evaluation time
+/// is dominated by the fixed cost of each call (for example 2 to 7 rows of
+/// a batch after a selective row filter took 600 to 8000 ns for each row in
+/// ClickBench Q23, against 0.4 ns for each row on full batches), and the
+/// fraction of removed rows is not reliable.
+pub const MIN_OBSERVED_ROWS: u64 = 8192;
+
 /// Returns the nanoseconds in `elapsed`, saturated to `u64::MAX`.
 pub fn duration_nanos(elapsed: Duration) -> u64 {
     u64::try_from(elapsed.as_nanos()).unwrap_or(u64::MAX)
diff --git a/datafusion/physical-expr/src/optional_filter_gate.rs b/datafusion/physical-expr/src/optional_filter_gate.rs
index 0d91bf9b7888f..5da48bcdb80af 100644
--- a/datafusion/physical-expr/src/optional_filter_gate.rs
+++ b/datafusion/physical-expr/src/optional_filter_gate.rs
@@ -48,7 +48,8 @@
 //! ```
 //!
 //! * In `Evaluate`, the gate collects the rows in, the rows out and the
-//!   evaluation time of `sample_batches` batches (a *window*). Then it
+//!   evaluation time of at least `sample_batches` batches and at least
+//!   [`MIN_OBSERVED_ROWS`] rows (a *window*). Then it
 //!   decides (see below). To pause, it goes to `Paused` for `backoff` batches
 //!   and doubles `backoff` (up to `max_pause_batches`). To keep the filter,
 //!   it stays in `Evaluate`, sets `backoff` to `initial_pause_batches` and
@@ -146,7 +147,9 @@ use datafusion_common::config::ExecutionOptions;
 use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
 
 use crate::expressions::DynamicFilterTracking;
-use crate::filter_stats::{Clock, FilterCost, SystemClock, duration_nanos};
+use crate::filter_stats::{
+    Clock, FilterCost, MIN_OBSERVED_ROWS, SystemClock, duration_nanos,
+};
 
 /// A running filter is paused by the cost rule only if its cost is larger
 /// than this multiple of its saving. See the [module documentation](self).
@@ -595,7 +598,12 @@ impl OptionalFilterGate {
         };
         window.add(rows_in as u64, rows_out as u64, duration_nanos(elapsed));
         *batches_in_window += 1;
-        if *batches_in_window >= self.config.sample_batches {
+        // A window has at least `sample_batches` batches and
+        // `MIN_OBSERVED_ROWS` rows: fewer rows are not enough evidence for a
+        // decision.
+        if *batches_in_window >= self.config.sample_batches
+            && window.rows_in >= MIN_OBSERVED_ROWS
+        {
             let window = *window;
             self.decide(window);
         }
@@ -707,7 +715,14 @@ mod tests {
     use datafusion_common::cast::as_boolean_array;
     use datafusion_expr::Operator;
 
-    const ROWS: usize = 1000;
+    /// Rows of each test batch: two batches make a window of
+    /// `MIN_OBSERVED_ROWS` rows.
+    const ROWS: usize = MIN_OBSERVED_ROWS as usize / 2;
+
+    /// `ROWS` values of `value(i)`.
+    fn values(value: impl Fn(i32) -> Option<i32>) -> Vec<Option<i32>> {
+        (0..ROWS as i32).map(value).collect()
+    }
 
     fn static_filter() -> Arc<dyn PhysicalExpr> {
         lit(true)
@@ -994,6 +1009,25 @@ mod tests {
         assert_eq!(gate.begin_batch(), GateDecision::Evaluate);
     }
 
+    /// Small batches (for example after a selective row filter) have a
+    /// large fixed cost for each row. The window grows until it has
+    /// `MIN_OBSERVED_ROWS` rows: small batches do not decide early.
+    #[test]
+    fn window_needs_min_observed_rows() {
+        let mut gate = new_gate();
+        let small = 3;
+        let batches = MIN_OBSERVED_ROWS as usize / small;
+        for _ in 0..batches {
+            assert_eq!(gate.begin_batch(), GateDecision::Evaluate);
+            // 8000 ns for each row, and no row removed.
+            gate.record(small, small, Duration::from_nanos(8000 * small as u64));
+        }
+        assert!(!gate.is_paused());
+        assert_eq!(gate.begin_batch(), GateDecision::Evaluate);
+        gate.record(small, small, Duration::from_nanos(8000 * small as u64));
+        assert!(gate.is_paused());
+    }
+
     #[test]
     fn empty_window_makes_no_decision() {
         let mut gate = new_gate();
@@ -1021,10 +1055,10 @@ mod tests {
     #[test]
     fn evaluate_selective_filter() {
         let mut gate = gate_with(col_gt(8));
-        let input = batch((0..10).map(Some).collect());
+        let input = batch(values(|i| Some(i % 10)));
         for _ in 0..10 {
             let result = evaluate(&mut gate, &input).expect("evaluated");
-            assert_eq!(result.true_count(), 1);
+            assert_eq!(result.true_count(), ROWS / 10);
         }
         assert_eq!(gate.pauses(), 0);
     }
@@ -1032,7 +1066,7 @@ mod tests {
     #[test]
     fn evaluate_filter_that_removes_no_rows_skips() {
         let mut gate = gate_with(col_gt(0));
-        let input = batch((1..=10).map(Some).collect());
+        let input = batch(values(|i| Some(i % 10 + 1)));
         assert!(evaluate(&mut gate, &input).is_some());
         assert!(evaluate(&mut gate, &input).is_some());
         for _ in 0..4 {
@@ -1045,12 +1079,10 @@ mod tests {
     fn evaluate_counts_null_as_not_passing() {
         let mut gate = gate_with(col_gt(0));
         // 9 of 10 rows are null: the filter removes them.
-        let mut values = vec![None; 9];
-        values.push(Some(5));
-        let input = batch(values);
+        let input = batch(values(|i| (i % 10 == 0).then_some(5)));
         for _ in 0..10 {
             let result = evaluate(&mut gate, &input).expect("evaluated");
-            assert_eq!(result.null_count(), 9);
+            assert_eq!(result.null_count(), ROWS - ROWS.div_ceil(10));
         }
         assert_eq!(gate.pauses(), 0);
     }
@@ -1087,8 +1119,8 @@ mod tests {
         assert_eq!(skip_until_probe(&mut gate, 0.07), 4);
         assert_eq!(feed_timed(&mut gate, 0.07, 70.0), GateDecision::Evaluate);
         // `skip_until_probe` evaluated the first batch with no time. The
-        // window costs 70 µs and saves 1860 * 20 ns = 37.2 µs: still too
-        // much.
+        // window costs 70 ns for each row of one batch and saves 0.93 * 20
+        // ns for each row of two batches: still too much.
         assert!(gate.is_paused());
         assert_eq!(skip_until_probe(&mut gate, 0.07), 8);
     }
@@ -1358,15 +1390,15 @@ mod tests {
     /// A consumer measures the time with the clock of the gate.
     #[test]
     fn consumer_measures_time_with_gate_clock() {
-        // Each evaluation takes 10 µs for 10 rows: 1000 ns for each row.
+        // Each evaluation takes 1000 ns for each row.
         let clock = Arc::new(SteppingClock {
             now: AtomicU64::new(0),
-            step: 10_000,
+            step: 1000 * ROWS as u64,
         });
         let mut gate =
             OptionalFilterGate::new(col_gt(8), OptionalFilterGateConfig::default())
                 .with_clock(clock);
-        let input = batch((0..10).map(Some).collect());
+        let input = batch(values(|i| Some(i % 10)));
         // The filter removes 90% of the rows, but it costs 1000 ns for each
         // row, and each removed row saves 20 ns.
         assert!(evaluate(&mut gate, &input).is_some());

From a352cb705a3df7603373bd08b32fe76e9737633e Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 13:51:27 -0500
Subject: [PATCH 13/42] feat(gate): use the measured work of the filter
 producer as the saving

The gate assumed that each removed row saves `min_saving_ns_per_row`
(20 ns) after the filter. For hash join dynamic filters this is the probe
work of the join, and it is much smaller in star joins with small
dimension tables: 3.5 to 8 ns for each probe row on the TPC-DS SF1
`date_dim` joins (Q65, Q67), 2 ns on Q90, 17 ns on TPC-H Q9. Filters that
cost 3 to 7 ns for each row and remove 80% of the rows thus stayed on,
and cost more than the join work they saved (8-18% slower than
`pruning_only` on the bot).

`RemovedRowWork` (in `filter_stats`) is the work that the producer of a
filter does for each row that the filter removes, as the producer
measures it. Each `DynamicFilterPhysicalExpr` has one, shared by all its
derived filters. The gate uses the smallest measured work of the dynamic
filters in its filter as the saving of a removed row, and
`min_saving_ns_per_row` only until the producer has measured
`MIN_OBSERVED_ROWS` rows (a prior).

PR: #25674

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/expressions/dynamic_filters/mod.rs    | 20 +++++
 datafusion/physical-expr/src/filter_stats.rs  | 53 ++++++++++++
 .../physical-expr/src/optional_filter_gate.rs | 84 ++++++++++++++++---
 3 files changed, 146 insertions(+), 11 deletions(-)

diff --git a/datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs b/datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs
index 4d384e952e0c0..2cc08779153b6 100644
--- a/datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs
+++ b/datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs
@@ -21,6 +21,7 @@ use std::{fmt::Display, hash::Hash, sync::Arc};
 use tokio::sync::watch;
 
 use crate::PhysicalExpr;
+use crate::filter_stats::RemovedRowWork;
 use arrow::datatypes::{DataType, Schema};
 #[cfg(feature = "proto")]
 use datafusion_common::internal_datafusion_err;
@@ -94,6 +95,10 @@ pub struct DynamicFilterPhysicalExpr {
     /// But this can have overhead in production, so it's only included in our tests.
     data_type: Arc<RwLock<Option<DataType>>>,
     nullable: Arc<RwLock<Option<bool>>>,
+    /// The work that the producer of the filter does for each row that the
+    /// filter removes, see [`Self::removed_row_work`]. Shared by all
+    /// derived filters.
+    removed_row_work: Arc<RemovedRowWork>,
 }
 
 impl std::fmt::Debug for DynamicFilterPhysicalExpr {
@@ -223,9 +228,20 @@ impl DynamicFilterPhysicalExpr {
             state_watch,
             data_type: Arc::new(RwLock::new(None)),
             nullable: Arc::new(RwLock::new(None)),
+            removed_row_work: Arc::default(),
         }
     }
 
+    /// The work that the producer of this filter does for each row that
+    /// the filter removes before it (for example the probe of a hash join),
+    /// as the producer measures it. A consumer that decides if the filter
+    /// is worth its cost uses it as the saving of a removed row (see
+    /// [`OptionalFilterGate`](crate::optional_filter_gate::OptionalFilterGate)).
+    /// The producer records its work in it.
+    pub fn removed_row_work(&self) -> &Arc<RemovedRowWork> {
+        &self.removed_row_work
+    }
+
     fn remap_children(
         children: &[Arc<dyn PhysicalExpr>],
         remapped_children: Option<&Vec<Arc<dyn PhysicalExpr>>>,
@@ -490,6 +506,7 @@ impl DynamicFilterPhysicalExpr {
             state_watch,
             data_type: Arc::new(RwLock::new(None)),
             nullable: Arc::new(RwLock::new(None)),
+            removed_row_work: Arc::default(),
         }
     }
 }
@@ -518,6 +535,7 @@ impl PhysicalExpr for DynamicFilterPhysicalExpr {
             state_watch: self.state_watch.clone(),
             data_type: Arc::clone(&self.data_type),
             nullable: Arc::clone(&self.nullable),
+            removed_row_work: Arc::clone(&self.removed_row_work),
         }))
     }
 
@@ -614,6 +632,7 @@ impl PhysicalExpr for DynamicFilterPhysicalExpr {
             state_watch: _, // Runtime channel, recreated from inner state by from_parts().
             data_type: _,   // Cached test invariant, recomputed from the expression.
             nullable: _,    // Cached test invariant, recomputed from the expression.
+            removed_row_work: _, // Runtime measurement of the producer.
         } = self;
 
         let children = children
@@ -837,6 +856,7 @@ impl DynamicFilterPhysicalExpr {
             state_watch: self.state_watch.clone(),
             data_type: Arc::clone(&self.data_type),
             nullable: Arc::clone(&self.nullable),
+            removed_row_work: Arc::clone(&self.removed_row_work),
         }
     }
 }
diff --git a/datafusion/physical-expr/src/filter_stats.rs b/datafusion/physical-expr/src/filter_stats.rs
index 0ea219f4166e0..f209a62acd17e 100644
--- a/datafusion/physical-expr/src/filter_stats.rs
+++ b/datafusion/physical-expr/src/filter_stats.rs
@@ -163,10 +163,63 @@ impl FilterCost {
     }
 }
 
+/// The work, in nanoseconds for each row, that an operator does on the rows
+/// that a filter removes before them, as measured by that operator.
+///
+/// For example, a hash join computes the hashes of the join keys of each
+/// probe row and looks them up in its hash table. A row that the dynamic
+/// filter of the join removes in the scan does not get this work, thus this
+/// is the saving of a removed row. The work that depends on a match (the
+/// output of a matched row) is not in it: the filter does not remove
+/// matched rows. The producer of a dynamic filter measures it and the
+/// consumers of the filter read it (see
+/// [`DynamicFilterPhysicalExpr::removed_row_work`]).
+///
+/// Lock-free.
+///
+/// [`DynamicFilterPhysicalExpr::removed_row_work`]: crate::expressions::DynamicFilterPhysicalExpr::removed_row_work
+#[derive(Debug, Default)]
+pub struct RemovedRowWork {
+    rows: AtomicU64,
+    nanos: AtomicU64,
+}
+
+impl RemovedRowWork {
+    /// Creates an empty measurement.
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    /// Adds `rows` rows and `nanos` nanoseconds of work. The two can be
+    /// recorded separately (for example the rows when a batch arrives and
+    /// the time of each step).
+    pub fn record(&self, rows: u64, nanos: u64) {
+        self.rows.fetch_add(rows, Ordering::Relaxed);
+        self.nanos.fetch_add(nanos, Ordering::Relaxed);
+    }
+
+    /// The work for each row, or `None` before [`MIN_OBSERVED_ROWS`] rows.
+    pub fn ns_per_row(&self) -> Option<f64> {
+        let rows = self.rows.load(Ordering::Relaxed);
+        (rows >= MIN_OBSERVED_ROWS)
+            .then(|| self.nanos.load(Ordering::Relaxed) as f64 / rows as f64)
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
+    #[test]
+    fn removed_row_work_needs_min_observed_rows() {
+        let work = RemovedRowWork::new();
+        work.record(MIN_OBSERVED_ROWS - 1, 0);
+        work.record(0, 3 * (MIN_OBSERVED_ROWS - 1));
+        assert_eq!(work.ns_per_row(), None);
+        work.record(1, 3);
+        assert_eq!(work.ns_per_row(), Some(3.0));
+    }
+
     #[test]
     fn filter_cost_derived_values() {
         let empty = FilterCost::default();
diff --git a/datafusion/physical-expr/src/optional_filter_gate.rs b/datafusion/physical-expr/src/optional_filter_gate.rs
index 5da48bcdb80af..15787f101a382 100644
--- a/datafusion/physical-expr/src/optional_filter_gate.rs
+++ b/datafusion/physical-expr/src/optional_filter_gate.rs
@@ -73,12 +73,20 @@
 //!    ```text
 //!    cost_ns   = evaluation time + rows_in * measured overhead
 //!    saving_ns = (rows_in - rows_out) * saving_ns_per_row
-//!    saving_ns_per_row = min_saving_ns_per_row + measured saving
+//!    saving_ns_per_row = producer work + measured saving
 //!    ```
 //!
-//!    `min_saving_ns_per_row` comes from the configuration. It is the work
-//!    that a removed row saves after the filter, for example a hash table
-//!    probe in a join. The *measured saving* and the *measured overhead* are
+//!    The *producer work* is the work that a removed row saves after the
+//!    filter, in the operator that produced the filter, for example the
+//!    hash and the hash table lookup of a probe row in a hash join. The
+//!    producer measures it ([`RemovedRowWork`], see
+//!    [`DynamicFilterPhysicalExpr::removed_row_work`]): a hash join with a
+//!    small build side does 2 to 8 ns of work for each probe row (TPC-DS
+//!    SF1 star joins), a join with a large build side much more. Until the
+//!    producer has measured [`MIN_OBSERVED_ROWS`] rows, the gate uses
+//!    `min_saving_ns_per_row` from the configuration (a prior). With more
+//!    than one dynamic filter in the filter, the smallest measured work is
+//!    used. The *measured saving* and the *measured overhead* are
 //!    optional: a consumer that can measure more work that a removed row
 //!    saves (the Parquet scan measures the decode time of the columns that
 //!    the filter does not read), or a fixed cost for each evaluated row in
@@ -146,9 +154,11 @@ use std::time::Duration;
 use datafusion_common::config::ExecutionOptions;
 use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
 
-use crate::expressions::DynamicFilterTracking;
+use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion};
+
+use crate::expressions::{DynamicFilterPhysicalExpr, DynamicFilterTracking};
 use crate::filter_stats::{
-    Clock, FilterCost, MIN_OBSERVED_ROWS, SystemClock, duration_nanos,
+    Clock, FilterCost, MIN_OBSERVED_ROWS, RemovedRowWork, SystemClock, duration_nanos,
 };
 
 /// A running filter is paused by the cost rule only if its cost is larger
@@ -172,7 +182,8 @@ pub struct OptionalFilterGateConfig {
     /// `initial_pause_batches` are used as `initial_pause_batches`.
     pub max_pause_batches: usize,
     /// Work, in nanoseconds, that each row removed by the filter saves after
-    /// the filter, at the least. The gate adds the saving that the consumer
+    /// the filter, until the producer of the filter has measured it (see
+    /// [`RemovedRowWork`]). The gate adds the saving that the consumer
     /// measures (see [`MeasuredRowSaving`]). The gate pauses a filter whose
     /// evaluation time is larger than the saving of the rows that it
     /// removes. Negative values are used as 0.
@@ -421,9 +432,11 @@ pub struct OptionalFilterGate {
     config: OptionalFilterGateConfig,
     /// The clock that consumers use to measure the evaluation time.
     clock: Arc<dyn Clock>,
-    /// The saving that the consumer measures, added to
-    /// `config.min_saving_ns_per_row`.
+    /// The saving that the consumer measures, added to the producer work.
     measured_saving: Option<Arc<MeasuredRowSaving>>,
+    /// The work that the producers of the dynamic filters in `filter` do
+    /// for each removed row, see [`RemovedRowWork`].
+    producer_work: Vec<Arc<RemovedRowWork>>,
     state: GateState,
     /// True while the current window is a probe after a pause. The cost
     /// rule then uses [`RESUME_COST_MARGIN`].
@@ -457,12 +470,22 @@ impl OptionalFilterGate {
     pub fn new(filter: Arc<dyn PhysicalExpr>, config: OptionalFilterGateConfig) -> Self {
         let config = config.normalized();
         let tracking = DynamicFilterTracking::classify(&filter);
+        let mut producer_work = vec![];
+        filter
+            .apply(|expr| {
+                if let Some(dynamic) = expr.downcast_ref::<DynamicFilterPhysicalExpr>() {
+                    producer_work.push(Arc::clone(dynamic.removed_row_work()));
+                }
+                Ok(TreeNodeRecursion::Continue)
+            })
+            .expect("the closure is infallible");
         Self {
             filter,
             tracking,
             config,
             clock: SystemClock::shared(),
             measured_saving: None,
+            producer_work,
             state: GateState::new_window(),
             probing: false,
             backoff: config.initial_pause_batches,
@@ -513,13 +536,26 @@ impl OptionalFilterGate {
     }
 
     /// The work, in nanoseconds, that the gate assumes each removed row
-    /// saves now: the configured minimum plus the measured saving.
+    /// saves now: the work of the producer (measured, or the configured
+    /// `min_saving_ns_per_row` before the measurement) plus the measured
+    /// saving of the consumer.
     pub fn saving_ns_per_row(&self) -> f64 {
         let measured = self
             .measured_saving
             .as_ref()
             .map_or(0.0, |saving| saving.ns_per_row());
-        self.config.min_saving_ns_per_row + measured
+        self.producer_work_ns_per_row() + measured
+    }
+
+    /// The work of the producer for each removed row: the smallest measured
+    /// [`RemovedRowWork`] of the dynamic filters in the filter, or
+    /// `min_saving_ns_per_row` if none is measured yet.
+    fn producer_work_ns_per_row(&self) -> f64 {
+        self.producer_work
+            .iter()
+            .filter_map(|work| work.ns_per_row())
+            .reduce(f64::min)
+            .unwrap_or(self.config.min_saving_ns_per_row)
     }
 
     /// The work, in nanoseconds for each evaluated row, that the gate adds
@@ -1374,6 +1410,32 @@ mod tests {
         assert!(!shared_gate(filter, &shared).is_paused());
     }
 
+    /// The producer of a dynamic filter measures its work for each row that
+    /// the filter removes: once measured, it replaces the configured
+    /// `min_saving_ns_per_row`.
+    #[test]
+    fn producer_work_replaces_configured_saving() {
+        let (dynamic, filter) = dynamic_filter();
+        let mut gate = gate_with(filter);
+        assert_eq!(gate.saving_ns_per_row(), 20.0);
+        // Removes 80% at 5 ns for each row: 5 < 0.8 * 20 * 1.1, it stays on.
+        for _ in 0..4 {
+            assert_eq!(feed_timed(&mut gate, 0.2, 5.0), GateDecision::Evaluate);
+        }
+        assert!(!gate.is_paused());
+
+        // The producer measures 4 ns for each removed row: 0.8 * 4 < 5.
+        let work = dynamic.removed_row_work();
+        work.record(MIN_OBSERVED_ROWS, 4 * MIN_OBSERVED_ROWS);
+        assert_eq!(gate.saving_ns_per_row(), 4.0);
+        assert_eq!(feed_timed(&mut gate, 0.2, 5.0), GateDecision::Evaluate);
+        assert_eq!(feed_timed(&mut gate, 0.2, 5.0), GateDecision::Evaluate);
+        assert!(gate.is_paused());
+
+        // A filter without dynamic filters uses the configuration.
+        assert_eq!(new_gate().saving_ns_per_row(), 20.0);
+    }
+
     /// A clock that moves by a fixed step each time it is read.
     #[derive(Debug)]
     struct SteppingClock {

From ded1dd654377b5e56ac329c4669d4ae12be36c48 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 18:03:42 -0500
Subject: [PATCH 14/42] feat: add optional_filter_mode config

Add `OptionalFilterMode` and the
`datafusion.execution.optional_filter_mode` option (default `always`):

- `always`: evaluate optional filters like any other pushed-down filter
  (today's behavior).
- `adaptive`: evaluate each optional filter behind an
  `OptionalFilterGate`, which pauses it while it costs more than it saves
  or removes no rows.
- `pruning_only`: use optional filters only for statistics pruning.

The Parquet scan and `FilterExec` read this option. This commit alone
does not change behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/common/src/config.rs               | 70 +++++++++++++++++++
 .../test_files/information_schema.slt         |  2 +
 docs/source/user-guide/configs.md             |  1 +
 3 files changed, 73 insertions(+)

diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index 24d52eb080fad..f05f4034f424f 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -883,6 +883,61 @@ impl Display for MapKeyDedupPolicy {
     }
 }
 
+/// How DataFusion evaluates optional filters.
+///
+/// Optional filters are filters that are not needed for correctness, such as
+/// dynamic filters pushed down by hash joins and TopK. See
+/// [`ExecutionOptions::optional_filter_mode`].
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
+pub enum OptionalFilterMode {
+    /// Evaluate optional filters like any other pushed-down filter.
+    #[default]
+    Always,
+    /// Pause optional filters that cost more than they save. Try them again
+    /// at intervals to find out if they became worth their cost.
+    Adaptive,
+    /// Use optional filters only for statistics pruning (for example of
+    /// files, row groups and pages). Never evaluate them row by row.
+    PruningOnly,
+}
+
+impl FromStr for OptionalFilterMode {
+    type Err = DataFusionError;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        match s.to_ascii_lowercase().as_str() {
+            "always" => Ok(Self::Always),
+            "adaptive" => Ok(Self::Adaptive),
+            "pruning_only" => Ok(Self::PruningOnly),
+            other => Err(DataFusionError::Configuration(format!(
+                "Invalid optional filter mode: {other}. Expected one of: always, adaptive, pruning_only"
+            ))),
+        }
+    }
+}
+
+impl ConfigField for OptionalFilterMode {
+    fn visit<V: Visit>(&self, v: &mut V, key: &str, description: &'static str) {
+        v.some(key, self, description)
+    }
+
+    fn set(&mut self, _: &str, value: &str) -> Result<()> {
+        *self = OptionalFilterMode::from_str(value)?;
+        Ok(())
+    }
+}
+
+impl Display for OptionalFilterMode {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        let str = match self {
+            Self::Always => "always",
+            Self::Adaptive => "adaptive",
+            Self::PruningOnly => "pruning_only",
+        };
+        write!(f, "{str}")
+    }
+}
+
 impl From<SpillCompression> for Option<CompressionType> {
     fn from(c: SpillCompression) -> Self {
         match c {
@@ -1185,6 +1240,21 @@ config_namespace! {
         /// Disabled by default, set to a number greater than 0 for enabling it.
         pub hash_join_buffering_capacity: usize, default = 0
 
+        /// Controls how DataFusion evaluates filters that are not needed for
+        /// correctness, such as the dynamic filters that hash joins and TopK
+        /// push down into scans. `always` evaluates these filters like any
+        /// other pushed-down filter. `adaptive` pauses these filters when they
+        /// cost more than they save (see
+        /// `datafusion.execution.optional_filter_min_saving_ns_per_row`) or when
+        /// they remove no rows, and tries them again at intervals. `pruning_only`
+        /// uses these filters only to prune files, row groups and pages with
+        /// statistics, and never evaluates them row by row.
+        ///
+        /// This option is most important when
+        /// `datafusion.execution.parquet.pushdown_filters` is true, because then
+        /// the Parquet reader evaluates pushed-down filters row by row.
+        pub optional_filter_mode: OptionalFilterMode, default = OptionalFilterMode::Always
+
         /// The assumed work, in nanoseconds, that each row removed by an
         /// optional filter saves downstream. Optional filters are filters that
         /// are not needed for correctness, such as the dynamic filters that
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt b/datafusion/sqllogictest/test_files/information_schema.slt
index 1ecc7a97d310e..3f6ea66b6f1d0 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -232,6 +232,7 @@ datafusion.execution.meta_fetch_concurrency 32
 datafusion.execution.minimum_parallel_output_files 4
 datafusion.execution.objectstore_writer_buffer_size 10485760
 datafusion.execution.optional_filter_min_saving_ns_per_row 20
+datafusion.execution.optional_filter_mode always
 datafusion.execution.parquet.allow_single_file_parallelism true
 datafusion.execution.parquet.binary_as_string false
 datafusion.execution.parquet.bloom_filter_fpp NULL
@@ -394,6 +395,7 @@ datafusion.execution.meta_fetch_concurrency 32 Number of files to read in parall
 datafusion.execution.minimum_parallel_output_files 4 Guarantees a minimum level of output files running in parallel. RecordBatches will be distributed in round robin fashion to each parallel writer. Each writer is closed and a new file opened once soft_max_rows_per_output_file is reached.
 datafusion.execution.objectstore_writer_buffer_size 10485760 Size (bytes) of data buffer DataFusion uses when writing output files. This affects the size of the data chunks that are uploaded to remote object stores (e.g. AWS S3). If very large (>= 100 GiB) output files are being written, it may be necessary to increase this size to avoid errors from the remote end point.
 datafusion.execution.optional_filter_min_saving_ns_per_row 20 The assumed work, in nanoseconds, that each row removed by an optional filter saves downstream. Optional filters are filters that are not needed for correctness, such as the dynamic filters that hash joins and TopK push down into scans. When an operator evaluates optional filters adaptively, it pauses an optional filter whose evaluation costs more than the work that it saves. Consumers that can measure the saving (the Parquet scan) add their measured decode cost. The default is about the cost of a hash table probe for one row. The best value depends on the hardware.
+datafusion.execution.optional_filter_mode always Controls how DataFusion evaluates filters that are not needed for correctness, such as the dynamic filters that hash joins and TopK push down into scans. `always` evaluates these filters like any other pushed-down filter. `adaptive` pauses these filters when they cost more than they save (see `datafusion.execution.optional_filter_min_saving_ns_per_row`) or when they remove no rows, and tries them again at intervals. `pruning_only` uses these filters only to prune files, row groups and pages with statistics, and never evaluates them row by row. This option is most important when `datafusion.execution.parquet.pushdown_filters` is true, because then the Parquet reader evaluates pushed-down filters row by row.
 datafusion.execution.parquet.allow_single_file_parallelism true (writing) Controls whether DataFusion will attempt to speed up writing parquet files by serializing them in parallel. Each column in each row group in each output file are serialized in parallel leveraging a maximum possible core count of n_files\*n_row_groups\*n_columns.
 datafusion.execution.parquet.binary_as_string false (reading) If true, parquet reader will read columns of `Binary/LargeBinary` with `Utf8`, and `BinaryView` with `Utf8View`. Parquet files generated by some legacy writers do not correctly set the UTF8 flag for strings, causing string columns to be loaded as BLOB instead. The parquet reader has special optimizations for `Utf8` validation, so reading such columns as strings is significantly faster than reading them as binary and then casting to string.
 datafusion.execution.parquet.bloom_filter_fpp NULL (writing) Sets bloom filter false positive probability. If NULL, uses default parquet writer setting
diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md
index 178e3d109466c..3c87a2888e86f 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -145,6 +145,7 @@ The following configuration settings are available:
 | datafusion.execution.objectstore_writer_buffer_size                     | 10485760                  | Size (bytes) of data buffer DataFusion uses when writing output files. This affects the size of the data chunks that are uploaded to remote object stores (e.g. AWS S3). If very large (>= 100 GiB) output files are being written, it may be necessary to increase this size to avoid errors from the remote end point.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | datafusion.execution.enable_ansi_mode                                   | false                     | Whether to enable ANSI SQL mode. The flag is experimental and relevant only for DataFusion Spark built-in functions When `enable_ansi_mode` is set to `true`, the query engine follows ANSI SQL semantics for expressions, casting, and error handling. This means: - **Strict type coercion rules:** implicit casts between incompatible types are disallowed. - **Standard SQL arithmetic behavior:** operations such as division by zero, numeric overflow, or invalid casts raise runtime errors rather than returning `NULL` or adjusted values. - **Consistent ANSI behavior** for string concatenation, comparisons, and `NULL` handling. When `enable_ansi_mode` is `false` (the default), the engine uses a more permissive, non-ANSI mode designed for user convenience and backward compatibility. In this mode: - Implicit casts between types are allowed (e.g., string to integer when possible). - Arithmetic operations are more lenient — for example, `abs()` on the minimum representable integer value returns the input value instead of raising overflow. - Division by zero or invalid casts may return `NULL` instead of failing. # Default `false` — ANSI SQL mode is disabled by default.                          |
 | datafusion.execution.hash_join_buffering_capacity                       | 0                         | How many bytes to buffer in the probe side of hash joins while the build side is concurrently being built. Without this, hash joins will wait until the full materialization of the build side before polling the probe side. This is useful in scenarios where the query is not completely CPU bounded, allowing to do some early work concurrently and reducing the latency of the query. Note that when hash join buffering is enabled, the probe side will start eagerly polling data, not giving time for the producer side of dynamic filters to produce any meaningful predicate. Queries with dynamic filters might see performance degradation. Disabled by default, set to a number greater than 0 for enabling it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| datafusion.execution.optional_filter_mode                               | always                    | Controls how DataFusion evaluates filters that are not needed for correctness, such as the dynamic filters that hash joins and TopK push down into scans. `always` evaluates these filters like any other pushed-down filter. `adaptive` pauses these filters when they cost more than they save (see `datafusion.execution.optional_filter_min_saving_ns_per_row`) or when they remove no rows, and tries them again at intervals. `pruning_only` uses these filters only to prune files, row groups and pages with statistics, and never evaluates them row by row. This option is most important when `datafusion.execution.parquet.pushdown_filters` is true, because then the Parquet reader evaluates pushed-down filters row by row.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | datafusion.execution.optional_filter_min_saving_ns_per_row              | 20                        | The assumed work, in nanoseconds, that each row removed by an optional filter saves downstream. Optional filters are filters that are not needed for correctness, such as the dynamic filters that hash joins and TopK push down into scans. When an operator evaluates optional filters adaptively, it pauses an optional filter whose evaluation costs more than the work that it saves. Consumers that can measure the saving (the Parquet scan) add their measured decode cost. The default is about the cost of a hash table probe for one row. The best value depends on the hardware.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
 | datafusion.optimizer.enable_distinct_aggregation_soft_limit             | true                      | When set to true, the optimizer will push a limit operation into grouped aggregations which have no aggregate expressions, as a soft limit, emitting groups once the limit is reached, before all rows in the group are read.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | datafusion.optimizer.enable_round_robin_repartition                     | true                      | When set to true, the physical plan optimizer will try to add round robin repartitioning to increase parallelism to leverage more CPU cores                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |

From 0a51a930ac4b2af3373c48d1f0a525cc8f0b4604 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 18:03:59 -0500
Subject: [PATCH 15/42] feat: let the Parquet scan skip optional filters that
 are not worth their cost

Make the Parquet scan the first consumer of optional filters (conjuncts
wrapped in `OptionalFilterPhysicalExpr`) when row level filter pushdown
(`datafusion.execution.parquet.pushdown_filters`) is enabled. The
`datafusion.execution.optional_filter_mode` option controls the row filter:

- `always` (default): optional conjuncts are normal RowFilter predicates.
  Behavior does not change.
- `adaptive`: each optional conjunct is a separate RowFilter predicate,
  after all required predicates, with an `OptionalFilterGate`. When the
  gate skips a batch, the predicate lets all rows pass without evaluation.
  Each file has its own gates; gates do not share state.
- `pruning_only`: optional conjuncts are not in the RowFilter.

In `adaptive` mode, the gate times each evaluation of the predicate and
pauses a filter that removes no rows or that costs more than it saves.
The saving of a removed row is the configured minimum
(`datafusion.execution.optional_filter_min_saving_ns_per_row`) plus the
decode time of the output columns that the filter does not read. The
scan estimates this decode time from the compressed size of those column
chunks (file metadata) and a decode speed in ns per compressed byte that
it measures over the whole scan (time to produce each output batch from
the decoder, which does not include the row filter).

`ParquetSource::try_pushdown_filters` reads the mode and the minimum
saving from the session configuration.

In all modes, required conjuncts do not change, and statistics pruning
(files, row groups, pages) uses optional conjuncts as before. An optional
conjunct that cannot be pushed down for a file is dropped for that file.

Add the lazily registered metrics `optional_filter_rows_skipped`,
`optional_filter_pauses` and `optional_filter_eval_time`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/core/tests/parquet/mod.rs          |   1 +
 .../core/tests/parquet/optional_filters.rs    | 400 ++++++++
 datafusion/datasource-parquet/src/metrics.rs  |  88 ++
 datafusion/datasource-parquet/src/mod.rs      |   1 +
 .../datasource-parquet/src/opener/mod.rs      |  90 +-
 .../datasource-parquet/src/optional_filter.rs | 246 +++++
 .../datasource-parquet/src/push_decoder.rs    |  46 +-
 .../datasource-parquet/src/row_filter.rs      | 897 +++++++++++++++++-
 datafusion/datasource-parquet/src/source.rs   |  86 ++
 .../test_files/optional_filters.slt           | 135 +++
 docs/source/user-guide/explain-usage.md       |   3 +
 11 files changed, 1909 insertions(+), 84 deletions(-)
 create mode 100644 datafusion/core/tests/parquet/optional_filters.rs
 create mode 100644 datafusion/datasource-parquet/src/optional_filter.rs
 create mode 100644 datafusion/sqllogictest/test_files/optional_filters.slt

diff --git a/datafusion/core/tests/parquet/mod.rs b/datafusion/core/tests/parquet/mod.rs
index 308d7176c455e..77153c42b7ff5 100644
--- a/datafusion/core/tests/parquet/mod.rs
+++ b/datafusion/core/tests/parquet/mod.rs
@@ -53,6 +53,7 @@ mod expr_adapter;
 mod external_access_plan;
 mod file_statistics;
 mod filter_pushdown;
+mod optional_filters;
 mod ordering;
 mod page_pruning;
 mod row_group_pruning;
diff --git a/datafusion/core/tests/parquet/optional_filters.rs b/datafusion/core/tests/parquet/optional_filters.rs
new file mode 100644
index 0000000000000..bff9366d4fa1f
--- /dev/null
+++ b/datafusion/core/tests/parquet/optional_filters.rs
@@ -0,0 +1,400 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! End-to-end tests for *optional filters* in the Parquet scan with row
+//! level filter pushdown (`pushdown_filters = true`).
+//!
+//! An optional filter is a predicate conjunct wrapped in an
+//! `OptionalFilterPhysicalExpr`. The scan handles it as
+//! `datafusion.execution.optional_filter_mode` says:
+//!
+//! * `always`: like all other filters.
+//! * `adaptive`: an `OptionalFilterGate` skips it while it removes no rows,
+//!   or while it costs more than it saves.
+//! * `pruning_only`: only statistics pruning uses it.
+//!
+//! No operator makes optional filters yet, thus these tests push an
+//! `Optional(...)` predicate into the `ParquetSource` directly, with
+//! `FileSource::try_pushdown_filters` and the session configuration.
+
+use std::sync::Arc;
+
+use arrow::array::{Array, Int32Array, Int64Array, RecordBatch};
+use arrow::compute::concat_batches;
+use arrow_schema::{DataType, Field, Schema, SchemaRef};
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::object_store::ObjectStoreUrl;
+use datafusion::datasource::physical_plan::ParquetSource;
+use datafusion::physical_plan::filter::FilterExec;
+use datafusion::physical_plan::{ExecutionPlan, collect, execute_stream};
+use datafusion::prelude::{SessionConfig, SessionContext};
+use datafusion_common::ScalarValue;
+use datafusion_common::config::{ConfigOptions, OptionalFilterMode};
+use datafusion_datasource::file::FileSource;
+use datafusion_datasource::file_scan_config::FileScanConfigBuilder;
+use datafusion_datasource::source::DataSourceExec;
+use datafusion_expr::Operator;
+use datafusion_physical_expr::PhysicalExpr;
+use datafusion_physical_expr::expressions::{
+    BinaryExpr, Column, DynamicFilterPhysicalExpr, OptionalFilterPhysicalExpr, lit,
+};
+use futures::StreamExt;
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::WriterProperties;
+use tempfile::NamedTempFile;
+
+use crate::parquet::utils::MetricsFinder;
+
+const ROW_GROUPS: usize = 10;
+const ROWS_PER_ROW_GROUP: usize = 2000;
+/// Each row filter evaluation sees one batch of this many rows.
+const BATCH_SIZE: usize = 100;
+
+/// A file with `ROW_GROUPS` row groups. Column `a` is `i % 100` (each row
+/// group has all values `0..100`, thus statistics cannot prune it), `rg` is
+/// the row group index and `v` is the row number.
+fn write_file() -> (NamedTempFile, SchemaRef) {
+    let schema = Arc::new(Schema::new(vec![
+        Field::new("a", DataType::Int32, false),
+        Field::new("rg", DataType::Int32, false),
+        Field::new("v", DataType::Int64, false),
+    ]));
+    let file = NamedTempFile::new().unwrap();
+    let props = WriterProperties::builder()
+        .set_max_row_group_row_count(Some(ROWS_PER_ROW_GROUP))
+        .build();
+    let mut writer =
+        ArrowWriter::try_new(file.reopen().unwrap(), Arc::clone(&schema), Some(props))
+            .unwrap();
+    for rg in 0..ROW_GROUPS {
+        let start = rg * ROWS_PER_ROW_GROUP;
+        let rows = start..start + ROWS_PER_ROW_GROUP;
+        let a: Int32Array = rows.clone().map(|i| (i % 100) as i32).collect();
+        let rg_col = Int32Array::from(vec![rg as i32; ROWS_PER_ROW_GROUP]);
+        let v: Int64Array = rows.map(|i| i as i64).collect();
+        let batch = RecordBatch::try_new(
+            Arc::clone(&schema),
+            vec![Arc::new(a), Arc::new(rg_col), Arc::new(v)],
+        )
+        .unwrap();
+        writer.write(&batch).unwrap();
+    }
+    let metadata = writer.close().unwrap();
+    assert_eq!(metadata.num_row_groups(), ROW_GROUPS);
+    (file, schema)
+}
+
+fn col_a(schema: &Schema) -> Arc<dyn PhysicalExpr> {
+    Arc::new(Column::new_with_schema("a", schema).unwrap())
+}
+
+fn a_op(schema: &Schema, op: Operator, value: i32) -> Arc<dyn PhysicalExpr> {
+    Arc::new(BinaryExpr::new(
+        col_a(schema),
+        op,
+        lit(ScalarValue::Int32(Some(value))),
+    ))
+}
+
+/// `a % 100 < 100`: true for all rows. Statistics cannot prove this, thus
+/// the scan evaluates the filter for each row (it does not skip the row
+/// filter for fully matched row groups).
+fn removes_no_rows(schema: &Schema) -> Arc<dyn PhysicalExpr> {
+    let a_mod_100: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
+        col_a(schema),
+        Operator::Modulo,
+        lit(ScalarValue::Int32(Some(100))),
+    ));
+    Arc::new(BinaryExpr::new(
+        a_mod_100,
+        Operator::Lt,
+        lit(ScalarValue::Int32(Some(100))),
+    ))
+}
+
+fn optional(inner: Arc<dyn PhysicalExpr>) -> Arc<dyn PhysicalExpr> {
+    Arc::new(OptionalFilterPhysicalExpr::new(inner))
+}
+
+/// A scan of all columns of `file`. The gates of the optional filters
+/// assume that a removed row saves a very large amount of work, thus the
+/// cost check (which uses the wall clock) never pauses a filter that
+/// removes rows, and the tests are deterministic. See [`scan_with`].
+fn scan(
+    file: &NamedTempFile,
+    schema: &SchemaRef,
+    predicate: Arc<dyn PhysicalExpr>,
+    mode: OptionalFilterMode,
+) -> Arc<dyn ExecutionPlan> {
+    scan_with(file, schema, predicate, mode, 1e9, None)
+}
+
+/// A scan of the columns `projection` (all columns if `None`) of `file`,
+/// with `min_saving_ns_per_row` for the gates of the optional filters.
+fn scan_with(
+    file: &NamedTempFile,
+    schema: &SchemaRef,
+    predicate: Arc<dyn PhysicalExpr>,
+    mode: OptionalFilterMode,
+    min_saving_ns_per_row: f64,
+    projection: Option<Vec<usize>>,
+) -> Arc<dyn ExecutionPlan> {
+    let mut options = ConfigOptions::default();
+    options.execution.parquet.pushdown_filters = true;
+    options.execution.optional_filter_mode = mode;
+    options.execution.optional_filter_min_saving_ns_per_row = min_saving_ns_per_row;
+    let source = ParquetSource::new(Arc::clone(schema))
+        .try_pushdown_filters(vec![predicate], &options)
+        .unwrap()
+        .updated_node
+        .expect("the scan accepts the predicate");
+    let path = file.path().to_str().unwrap().to_string();
+    let size = std::fs::metadata(&path).unwrap().len();
+    let config = FileScanConfigBuilder::new(ObjectStoreUrl::local_filesystem(), source)
+        .with_file(PartitionedFile::new(path, size))
+        .with_projection_indices(projection)
+        .unwrap()
+        .build();
+    DataSourceExec::from_data_source(config)
+}
+
+fn session() -> SessionContext {
+    SessionContext::new_with_config(SessionConfig::new().with_batch_size(BATCH_SIZE))
+}
+
+fn metric(plan: &dyn ExecutionPlan, name: &str) -> usize {
+    MetricsFinder::find_metrics(plan)
+        .unwrap()
+        .sum_by_name(name)
+        .map_or(0, |v| v.as_usize())
+}
+
+/// Sorted values of column `v`.
+fn values(batches: &[RecordBatch]) -> Vec<i64> {
+    let mut values: Vec<i64> = batches
+        .iter()
+        .flat_map(|batch| {
+            let v = batch
+                .column_by_name("v")
+                .unwrap()
+                .as_any()
+                .downcast_ref::<Int64Array>()
+                .unwrap();
+            v.values().to_vec()
+        })
+        .collect();
+    values.sort_unstable();
+    values
+}
+
+/// An optional filter that removes no rows is paused by the gate in
+/// `adaptive` mode. The consumer of the scan (here a
+/// `FilterExec`, like a hash join for a join dynamic filter) applies the
+/// filter again, thus the results are the same in all modes.
+#[tokio::test]
+async fn optional_filter_that_removes_no_rows_is_paused() {
+    let (file, schema) = write_file();
+    let total_rows = ROW_GROUPS * ROWS_PER_ROW_GROUP;
+    let removed_rows = 0;
+
+    let mut results = vec![];
+    for mode in [
+        OptionalFilterMode::Always,
+        OptionalFilterMode::Adaptive,
+        OptionalFilterMode::PruningOnly,
+    ] {
+        let filter = removes_no_rows(&schema);
+        let scan = scan(&file, &schema, optional(Arc::clone(&filter)), mode);
+        let plan: Arc<dyn ExecutionPlan> =
+            Arc::new(FilterExec::try_new(filter, Arc::clone(&scan)).unwrap());
+        let batches = collect(plan, session().task_ctx()).await.unwrap();
+        let values = values(&batches);
+        assert_eq!(values.len(), total_rows - removed_rows, "mode {mode}");
+
+        let pruned = metric(scan.as_ref(), "pushdown_rows_pruned");
+        let skipped = metric(scan.as_ref(), "optional_filter_rows_skipped");
+        let pauses = metric(scan.as_ref(), "optional_filter_pauses");
+        match mode {
+            OptionalFilterMode::Always => {
+                assert_eq!(pruned, removed_rows);
+                assert_eq!(skipped, 0);
+                assert_eq!(pauses, 0);
+            }
+            OptionalFilterMode::Adaptive => {
+                assert!(skipped > total_rows / 2, "skipped {skipped} rows");
+                assert!(pauses > 0);
+                assert_eq!(pruned, removed_rows);
+                // Each row is either skipped or evaluated.
+                let matched = metric(scan.as_ref(), "pushdown_rows_matched");
+                assert_eq!(matched + pruned, total_rows);
+            }
+            OptionalFilterMode::PruningOnly => {
+                // No row filter at all.
+                assert_eq!(pruned, 0);
+                assert_eq!(metric(scan.as_ref(), "pushdown_rows_matched"), 0);
+                assert_eq!(skipped, 0);
+                assert_eq!(pauses, 0);
+            }
+        }
+        results.push(values);
+    }
+    assert_eq!(results[0], results[1]);
+    assert_eq!(results[0], results[2]);
+}
+
+/// A selective optional filter is never paused, and the scan output is the
+/// same as in `always` mode.
+#[tokio::test]
+async fn selective_optional_filter_is_not_paused() {
+    let (file, schema) = write_file();
+    let predicate = optional(a_op(&schema, Operator::Lt, 10));
+    let mut results = vec![];
+    for mode in [OptionalFilterMode::Always, OptionalFilterMode::Adaptive] {
+        let scan = scan(&file, &schema, Arc::clone(&predicate), mode);
+        let batches = collect(Arc::clone(&scan), session().task_ctx())
+            .await
+            .unwrap();
+        assert_eq!(metric(scan.as_ref(), "optional_filter_rows_skipped"), 0);
+        assert_eq!(metric(scan.as_ref(), "optional_filter_pauses"), 0);
+        results.push(values(&batches));
+    }
+    assert_eq!(results[0].len(), ROW_GROUPS * ROWS_PER_ROW_GROUP / 10);
+    assert_eq!(results[0], results[1]);
+}
+
+/// A selective optional filter is paused when it costs more than it saves.
+/// The scan reads only column `a`, which the filter reads too: a removed row
+/// saves no decode time. With `min_saving_ns_per_row = 0`, a removed row
+/// saves nothing, thus any evaluation time is too much and the gate pauses
+/// `a < 10` although it keeps only 10% of the rows. The consumer of the scan
+/// applies the filter again, thus the result does not change.
+#[tokio::test]
+async fn selective_optional_filter_is_paused_when_it_costs_more_than_it_saves() {
+    let (file, schema) = write_file();
+    let filter = a_op(&schema, Operator::Lt, 10);
+    let scan = scan_with(
+        &file,
+        &schema,
+        optional(Arc::clone(&filter)),
+        OptionalFilterMode::Adaptive,
+        0.0,
+        Some(vec![0]),
+    );
+    let plan: Arc<dyn ExecutionPlan> =
+        Arc::new(FilterExec::try_new(filter, Arc::clone(&scan)).unwrap());
+    let batches = collect(plan, session().task_ctx()).await.unwrap();
+    let rows: usize = batches.iter().map(|b| b.num_rows()).sum();
+    assert_eq!(rows, ROW_GROUPS * ROWS_PER_ROW_GROUP / 10);
+    assert!(metric(scan.as_ref(), "optional_filter_pauses") > 0);
+    assert!(metric(scan.as_ref(), "optional_filter_rows_skipped") > 0);
+    assert!(metric(scan.as_ref(), "optional_filter_eval_time") > 0);
+}
+
+/// A required conjunct stays a normal row filter predicate in all modes.
+#[tokio::test]
+async fn required_conjunct_is_unchanged() {
+    let (file, schema) = write_file();
+    // a < 50 AND Optional(a != 99)
+    let predicate: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
+        a_op(&schema, Operator::Lt, 50),
+        Operator::And,
+        optional(a_op(&schema, Operator::NotEq, 99)),
+    ));
+    for mode in [
+        OptionalFilterMode::Always,
+        OptionalFilterMode::Adaptive,
+        OptionalFilterMode::PruningOnly,
+    ] {
+        let scan = scan(&file, &schema, Arc::clone(&predicate), mode);
+        let batches = collect(Arc::clone(&scan), session().task_ctx())
+            .await
+            .unwrap();
+        assert_eq!(
+            values(&batches).len(),
+            ROW_GROUPS * ROWS_PER_ROW_GROUP / 2,
+            "mode {mode}"
+        );
+    }
+}
+
+/// An optional dynamic filter that changes during the scan: the gate paused
+/// the first version of the filter (which removes no rows), and evaluates the filter
+/// again when the filter changes.
+#[tokio::test]
+async fn optional_dynamic_filter_update_restarts_evaluation() {
+    let (file, schema) = write_file();
+    let dynamic = Arc::new(DynamicFilterPhysicalExpr::new(
+        vec![col_a(&schema)],
+        removes_no_rows(&schema),
+    ));
+    let scan = scan(
+        &file,
+        &schema,
+        optional(Arc::clone(&dynamic) as Arc<dyn PhysicalExpr>),
+        OptionalFilterMode::Adaptive,
+    );
+    let mut stream = execute_stream(Arc::clone(&scan), session().task_ctx()).unwrap();
+
+    // Read until the scan is in the third row group. The gate paused the
+    // filter in the first row group.
+    let mut before_update = vec![];
+    let mut max_rg_before_update = 0;
+    while max_rg_before_update < 2 {
+        let batch = stream.next().await.unwrap().unwrap();
+        max_rg_before_update = max_rg_before_update.max(max_rg(&batch));
+        before_update.push(batch);
+    }
+    assert!(metric(scan.as_ref(), "optional_filter_rows_skipped") > 0);
+    assert!(metric(scan.as_ref(), "optional_filter_pauses") > 0);
+
+    // The filter becomes selective.
+    dynamic.update(a_op(&schema, Operator::Lt, 10)).unwrap();
+
+    let mut after_update = vec![];
+    while let Some(batch) = stream.next().await {
+        after_update.push(batch.unwrap());
+    }
+    let after_update = concat_batches(&after_update[0].schema(), &after_update).unwrap();
+    // The scan evaluates the filter of a row group before it returns the
+    // first batch of that row group. Thus all row groups after the current
+    // one use the new filter.
+    let a = column_i32(&after_update, "a");
+    let rg = column_i32(&after_update, "rg");
+    let mut checked_rows = 0;
+    for (a, rg) in a.iter().zip(rg.iter()) {
+        if *rg > max_rg_before_update {
+            assert!(*a < 10, "row with a = {a} in row group {rg} passed");
+            checked_rows += 1;
+        }
+    }
+    let later_row_groups = ROW_GROUPS - 1 - max_rg_before_update as usize;
+    assert_eq!(checked_rows, later_row_groups * ROWS_PER_ROW_GROUP / 10);
+}
+
+fn column_i32<'a>(batch: &'a RecordBatch, name: &str) -> &'a [i32] {
+    batch
+        .column_by_name(name)
+        .unwrap()
+        .as_any()
+        .downcast_ref::<Int32Array>()
+        .unwrap()
+        .values()
+}
+
+fn max_rg(batch: &RecordBatch) -> i32 {
+    column_i32(batch, "rg").iter().copied().max().unwrap_or(0)
+}
diff --git a/datafusion/datasource-parquet/src/metrics.rs b/datafusion/datasource-parquet/src/metrics.rs
index 4abadd2c4e13a..e73b3de165985 100644
--- a/datafusion/datasource-parquet/src/metrics.rs
+++ b/datafusion/datasource-parquet/src/metrics.rs
@@ -16,6 +16,7 @@
 // under the License.
 
 use std::sync::Arc;
+use std::time::Duration;
 
 use datafusion_physical_plan::metrics::{
     Count, ExecutionPlanMetricsSet, Gauge, Label, MetricBuilder, MetricCategory,
@@ -491,3 +492,90 @@ impl RowFilterSkippedFullyMatchedMetric {
         count.add(1);
     }
 }
+
+/// Lazily-registered counters for optional filters that an
+/// [`OptionalFilterGate`](datafusion_physical_expr::optional_filter_gate::OptionalFilterGate)
+/// evaluates in the Parquet `RowFilter` (when
+/// `datafusion.execution.optional_filter_mode` is `adaptive`):
+///
+/// * `optional_filter_rows_skipped`: rows for which the gate skipped an
+///   optional filter (all these rows passed that filter without evaluation).
+/// * `optional_filter_pauses`: number of times a gate paused an optional
+///   filter (because it removed no rows or cost more than it saved).
+/// * `optional_filter_eval_time`: time spent to evaluate optional filters.
+///   `row_pushdown_eval_time` includes this time.
+///
+/// Like [`RowFilterSkippedFullyMatchedMetric`], each counter is registered
+/// only when it first fires, so scans that never skip an optional filter do
+/// not show zero-valued counters in `EXPLAIN ANALYZE`.
+#[derive(Debug)]
+pub(crate) struct OptionalFilterMetrics {
+    metrics: ExecutionPlanMetricsSet,
+    partition: usize,
+    filename: String,
+    rows_skipped: Option<Count>,
+    pauses: Option<Count>,
+    eval_time: Option<Time>,
+}
+
+impl OptionalFilterMetrics {
+    pub(crate) fn new(
+        metrics: &ExecutionPlanMetricsSet,
+        partition: usize,
+        filename: &str,
+    ) -> Self {
+        Self {
+            metrics: metrics.clone(),
+            partition,
+            filename: filename.to_string(),
+            rows_skipped: None,
+            pauses: None,
+            eval_time: None,
+        }
+    }
+
+    fn counter(&self, name: &'static str) -> Count {
+        MetricBuilder::new(&self.metrics)
+            .with_new_label("filename", self.filename.clone())
+            .with_type(MetricType::Summary)
+            .with_category(MetricCategory::Rows)
+            .counter(name, self.partition)
+    }
+
+    /// Record `n` rows for which an optional filter was skipped.
+    pub(crate) fn add_rows_skipped(&mut self, n: usize) {
+        if n == 0 {
+            return;
+        }
+        if self.rows_skipped.is_none() {
+            self.rows_skipped = Some(self.counter("optional_filter_rows_skipped"));
+        }
+        if let Some(count) = &self.rows_skipped {
+            count.add(n);
+        }
+    }
+
+    /// Record `elapsed` time spent to evaluate an optional filter.
+    pub(crate) fn add_eval_time(&mut self, elapsed: Duration) {
+        let eval_time = self.eval_time.get_or_insert_with(|| {
+            MetricBuilder::new(&self.metrics)
+                .with_new_label("filename", self.filename.clone())
+                .with_type(MetricType::Summary)
+                .subset_time("optional_filter_eval_time", self.partition)
+        });
+        eval_time.add_duration(elapsed);
+    }
+
+    /// Record `n` new pauses of an optional filter.
+    pub(crate) fn add_pauses(&mut self, n: usize) {
+        if n == 0 {
+            return;
+        }
+        if self.pauses.is_none() {
+            self.pauses = Some(self.counter("optional_filter_pauses"));
+        }
+        if let Some(count) = &self.pauses {
+            count.add(n);
+        }
+    }
+}
diff --git a/datafusion/datasource-parquet/src/mod.rs b/datafusion/datasource-parquet/src/mod.rs
index 80d2205a1fab0..490af617ad742 100644
--- a/datafusion/datasource-parquet/src/mod.rs
+++ b/datafusion/datasource-parquet/src/mod.rs
@@ -32,6 +32,7 @@ pub mod metadata;
 mod metrics;
 mod nested_schema_pruning;
 mod opener;
+mod optional_filter;
 mod page_filter;
 mod projection_read_plan;
 mod pruning;
diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index a2d5c7d495345..7abbb4cf82244 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -26,11 +26,13 @@ use self::encryption::EncryptionContext;
 use crate::access_plan::PreparedAccessPlan;
 use crate::decoder_projection::DecoderProjection;
 use crate::metrics::{ByteProgress, RowFilterSkippedFullyMatchedMetric};
+use crate::optional_filter::OptionalFilterOptions;
 use crate::page_filter::PagePruningAccessPlanFilter;
 use crate::push_decoder::{
     DecoderBuilderConfig, InitialDecoderState, PushDecoderStreamState, RgPlanEntry,
     RowFilterContext, RowGroupPruner,
 };
+use crate::row_filter::OptionalFilterRowFilterContext;
 use crate::row_group_filter::{RowGroupAccessPlanFilter, row_group_in_range};
 use crate::{
     BloomFilterStatistics, Int96Coercer, ParquetAccessPlan, ParquetFileMetrics,
@@ -309,6 +311,9 @@ pub(super) struct ParquetMorselizer {
     /// Per-scan virtual-column state (validation already performed). `None`
     /// when no virtual columns are requested — the common path.
     pub(crate) virtual_state: Option<Arc<VirtualColumnsState>>,
+    /// How the row filter handles optional conjuncts of the predicate, when
+    /// `pushdown_filters` is true.
+    pub(crate) optional_filters: OptionalFilterOptions,
 }
 
 impl fmt::Debug for ParquetMorselizer {
@@ -501,6 +506,7 @@ struct PreparedParquetOpen {
     reverse_row_groups: bool,
     sort_order_for_reorder: Option<LexOrdering>,
     preserve_order: bool,
+    optional_filters: OptionalFilterOptions,
     #[cfg(feature = "parquet_encryption")]
     file_decryption_properties: Option<Arc<FileDecryptionProperties>>,
 }
@@ -561,7 +567,8 @@ impl DecoderReadPlans {
         // Either way every required conjunct is applied; nothing is silently
         // dropped. Optional conjuncts (see `split_optional`) are not needed
         // for correctness. They never go to the post-scan filter: they are
-        // row filter predicates or they are used only for statistics pruning.
+        // row filter predicates (as `optional_filter_mode` says) or they are
+        // used only for statistics pruning.
         // ---------------------------------------------------------------
         // A pruning-only predicate is applied by a `FilterExec` above the
         // scan: the scan uses it only to prune.
@@ -569,7 +576,24 @@ impl DecoderReadPlans {
             .predicate
             .as_ref()
             .filter(|_| !prepared.pruning_only_predicate);
-        let (row_filter_context, post_scan_conjuncts) =
+        let build_projection = |post_scan_conjuncts: &[Arc<dyn PhysicalExpr>]| {
+            // Build the decoder projection (mask + per-batch transform +
+            // optional post-scan filter) in a single call. Encapsulating it
+            // behind `DecoderProjection` keeps the opener's orchestration body
+            // focused on filter / decoder / stream wiring. The file-column
+            // projection excludes virtual columns and respects nested field
+            // projections.
+            DecoderProjection::try_new(
+                &prepared.projection,
+                post_scan_conjuncts,
+                &prepared.physical_file_schema,
+                metadata.parquet_schema(),
+                &prepared.output_schema,
+                prepared.virtual_state.as_deref(),
+                &prepared.file_metrics,
+            )
+        };
+        let (row_filter_context, projection) =
             match (prepared.pushdown_filters, applied_predicate) {
                 // Pushdown enabled: precompute the candidate list once per file.
                 // Both the initial `RowFilter` and any per-RG rebuilds (via
@@ -577,38 +601,46 @@ impl DecoderReadPlans {
                 // (`reassign_expr_columns`) and column resolution only run once —
                 // not once per row group. Only what the `RowFilter` could not
                 // place falls through to post-scan.
-                (true, Some(predicate)) => RowFilterContext::try_new(
-                    predicate,
-                    &prepared.physical_file_schema,
-                    metadata.metadata(),
-                    prepared.reorder_predicates,
-                    prepared.file_metrics.clone(),
-                    prepared.max_predicate_cache_size,
-                ),
+                (true, Some(predicate)) => {
+                    // The gates of optional filters estimate their saving from
+                    // the output columns, thus build the projection without
+                    // post-scan conjuncts first. It is used as-is when the
+                    // `RowFilter` can place every conjunct (the usual case).
+                    let output_projection = build_projection(&[])?;
+                    let (row_filter_context, rejected) = RowFilterContext::try_new(
+                        predicate,
+                        &prepared.physical_file_schema,
+                        metadata.metadata(),
+                        prepared.reorder_predicates,
+                        prepared.file_metrics.clone(),
+                        prepared.max_predicate_cache_size,
+                        Some(OptionalFilterRowFilterContext {
+                            options: &prepared.optional_filters,
+                            metrics: &prepared.metrics,
+                            partition: prepared.partition_index,
+                            filename: &prepared.file_name,
+                            output_projection: Some(output_projection.projection_mask()),
+                        }),
+                    );
+                    let projection = if rejected.is_empty() {
+                        output_projection
+                    } else {
+                        build_projection(&rejected)?
+                    };
+                    (row_filter_context, projection)
+                }
                 // Pushdown disabled: the required conjuncts run post-scan
                 // (in-scan equivalent of a `FilterExec`). Optional conjuncts
                 // (for example hash join dynamic filters) are not needed for
                 // correctness and are expensive to evaluate for each row,
                 // thus they are used only for statistics pruning, as before
                 // the scan accepted the filters.
-                (false, Some(predicate)) => (None, split_optional(predicate).0),
-                (_, None) => (None, Vec::new()),
+                (false, Some(predicate)) => {
+                    let (required, _optional) = split_optional(predicate);
+                    (None, build_projection(&required)?)
+                }
+                (_, None) => (None, build_projection(&[])?),
             };
-
-        // Build the decoder projection (mask + per-batch transform + optional
-        // post-scan filter) in a single call. Encapsulating it behind
-        // `DecoderProjection` keeps the opener's orchestration body focused on
-        // filter / decoder / stream wiring. The file-column projection
-        // excludes virtual columns and respects nested field projections.
-        let projection = DecoderProjection::try_new(
-            &prepared.projection,
-            &post_scan_conjuncts,
-            &prepared.physical_file_schema,
-            metadata.parquet_schema(),
-            &prepared.output_schema,
-            prepared.virtual_state.as_deref(),
-            &prepared.file_metrics,
-        )?;
         Ok(Self {
             projection,
             row_filter_context,
@@ -1036,6 +1068,7 @@ impl ParquetMorselizer {
             reverse_row_groups: self.reverse_row_groups,
             sort_order_for_reorder: self.sort_order_for_reorder.clone(),
             preserve_order: self.preserve_order,
+            optional_filters: self.optional_filters.clone(),
             #[cfg(feature = "parquet_encryption")]
             file_decryption_properties: None,
         })
@@ -2246,6 +2279,7 @@ mod test {
         max_in_list_size: usize,
         reverse_row_groups: bool,
         preserve_order: bool,
+        optional_filters: OptionalFilterOptions,
     }
 
     #[test]
@@ -2469,6 +2503,7 @@ mod test {
                 max_in_list_size: MAX_IN_LIST_SIZE,
                 reverse_row_groups: false,
                 preserve_order: false,
+                optional_filters: OptionalFilterOptions::default(),
             }
         }
 
@@ -2656,6 +2691,7 @@ mod test {
                 reverse_row_groups: self.reverse_row_groups,
                 sort_order_for_reorder: None,
                 virtual_state,
+                optional_filters: self.optional_filters,
             })
         }
     }
diff --git a/datafusion/datasource-parquet/src/optional_filter.rs b/datafusion/datasource-parquet/src/optional_filter.rs
new file mode 100644
index 0000000000000..4318ea68bb75f
--- /dev/null
+++ b/datafusion/datasource-parquet/src/optional_filter.rs
@@ -0,0 +1,246 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! How the Parquet scan handles *optional filters* in its row filter.
+//!
+//! An optional filter is a conjunct of the scan predicate that is wrapped in
+//! an [`OptionalFilterPhysicalExpr`]: the scan can skip it and the query
+//! result stays the same (for example, a hash join dynamic filter, which the
+//! join applies again). See [`OptionalFilterMode`] for the three modes.
+//!
+//! Statistics pruning (file, row group and page pruning) always uses the
+//! complete predicate, thus optional filters. Only the row-level
+//! `RowFilter` (used when `pushdown_filters` is true) changes with the mode.
+//!
+//! # The saving of a removed row
+//!
+//! In [`OptionalFilterMode::Adaptive`], the [`OptionalFilterGate`] of an
+//! optional filter pauses the filter when its evaluation costs more than the
+//! work that the rows it removes save. The gate adds the configured minimum
+//! saving (`datafusion.execution.optional_filter_min_saving_ns_per_row`, for
+//! the work after the scan) to the saving that the scan measures: the time
+//! to decode the output columns that the filter does not read, for each
+//! row. The scan does not decode these columns for a removed row.
+//!
+//! The scan estimates this decode time from the compressed size of the
+//! column chunks and a measured decode speed:
+//!
+//! ```text
+//! measured saving = ns_per_byte * (compressed bytes for each row of the
+//!                                  output columns that the filter does not read)
+//! ns_per_byte     = time to decode output batches
+//!                   / (output rows * compressed bytes for each row of the
+//!                      output columns)
+//! ```
+//!
+//! The bytes for each row come from the metadata of the file (the average
+//! over its row groups). `ns_per_byte` is measured over all files and
+//! partitions of the scan ([`DecodeCost`]): the scan times each call that
+//! decodes an output batch. This time does not include the row filter,
+//! because the Parquet decoder evaluates the row filter of a row group
+//! before it decodes the output columns. Before the first measurement,
+//! `ns_per_byte` is [`DEFAULT_DECODE_NS_PER_BYTE`].
+//!
+//! This is an estimate: the decode time of a row depends on the encoding and
+//! on the rows that are selected (a sparse selection costs more for each
+//! row), not only on the compressed size. But it is cheap (a clock read and
+//! a few atomic operations for each output batch), it adapts to the data and
+//! the hardware, and it separates the two important cases: a filter that
+//! reads all the output columns (for example a hash join filter on the join
+//! keys) saves only the work after the scan, and a filter on one column of
+//! a wide table (for example a TopK filter with `SELECT *`) saves the decode
+//! of all the other columns.
+//!
+//! [`OptionalFilterPhysicalExpr`]: datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr
+//! [`OptionalFilterGate`]: datafusion_physical_expr::optional_filter_gate::OptionalFilterGate
+
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::Duration;
+
+use datafusion_common::config::OptionalFilterMode;
+use datafusion_physical_expr::filter_stats::duration_nanos;
+use datafusion_physical_expr::optional_filter_gate::{
+    MeasuredRowSaving, OptionalFilterGateConfig,
+};
+use parquet::arrow::ProjectionMask;
+use parquet::file::metadata::ParquetMetaData;
+
+/// The decode time for each compressed byte that the scan assumes before it
+/// measures it (see the [module documentation](self)). The TPC-DS SF1 scans
+/// on one core of a laptop take 4 to 8 ns for each compressed byte (scan
+/// compute time divided by bytes scanned); this is the low end. The
+/// measurement replaces it after the first output batch of the scan.
+///
+/// The value is important: the decoder evaluates the row filter of a whole
+/// row group before it decodes the first output batch of that row group,
+/// thus the gates of the first row group of a scan decide with this value.
+pub(crate) const DEFAULT_DECODE_NS_PER_BYTE: f64 = 4.0;
+
+/// The measured decode speed of the output columns of one scan, shared by
+/// all partitions and files of the scan. See the [module
+/// documentation](self).
+#[derive(Debug, Default)]
+pub(crate) struct DecodeCost {
+    /// Time to decode output batches, in nanoseconds.
+    nanos: AtomicU64,
+    /// Sum, over the output batches, of the rows times the compressed bytes
+    /// for each row of the output columns.
+    bytes: AtomicU64,
+}
+
+impl DecodeCost {
+    /// Adds the decode time of one output batch, and its estimated
+    /// compressed size.
+    pub(crate) fn record(&self, nanos: u64, bytes: u64) {
+        self.nanos.fetch_add(nanos, Ordering::Relaxed);
+        self.bytes.fetch_add(bytes, Ordering::Relaxed);
+    }
+
+    /// The decode time for each compressed byte, or
+    /// [`DEFAULT_DECODE_NS_PER_BYTE`] before the first measurement.
+    pub(crate) fn ns_per_byte(&self) -> f64 {
+        let bytes = self.bytes.load(Ordering::Relaxed);
+        if bytes == 0 {
+            return DEFAULT_DECODE_NS_PER_BYTE;
+        }
+        self.nanos.load(Ordering::Relaxed) as f64 / bytes as f64
+    }
+}
+
+/// Average compressed bytes for each row of the leaf columns for which
+/// `include` is true, over all row groups of the file. 0 if the file has no
+/// rows.
+pub(crate) fn compressed_bytes_per_row(
+    metadata: &ParquetMetaData,
+    include: impl Fn(usize) -> bool,
+) -> f64 {
+    let mut rows: i64 = 0;
+    let mut bytes: i64 = 0;
+    for row_group in metadata.row_groups() {
+        rows += row_group.num_rows();
+        bytes += row_group
+            .columns()
+            .iter()
+            .enumerate()
+            .filter(|(leaf, _)| include(*leaf))
+            .map(|(_, column)| column.compressed_size())
+            .sum::<i64>();
+    }
+    if rows <= 0 {
+        0.0
+    } else {
+        bytes.max(0) as f64 / rows as f64
+    }
+}
+
+/// The measured saving of one gated optional filter of one file, see the
+/// [module documentation](self).
+#[derive(Debug, Clone)]
+pub(crate) struct OptionalFilterSaving {
+    /// The value that the gate of the filter reads.
+    pub(crate) saving: Arc<MeasuredRowSaving>,
+    /// Compressed bytes for each row of the output columns that the filter
+    /// does not read.
+    pub(crate) unread_bytes_per_row: f64,
+}
+
+impl OptionalFilterSaving {
+    /// A saving for a filter that does not read `unread_bytes_per_row`
+    /// compressed bytes of the output columns, with the current decode
+    /// speed of `decode_cost`.
+    pub(crate) fn new(unread_bytes_per_row: f64, decode_cost: &DecodeCost) -> Self {
+        let saving = Self {
+            saving: Arc::new(MeasuredRowSaving::new()),
+            unread_bytes_per_row,
+        };
+        saving.update(decode_cost.ns_per_byte());
+        saving
+    }
+
+    fn update(&self, ns_per_byte: f64) {
+        self.saving
+            .set_ns_per_row(ns_per_byte * self.unread_bytes_per_row);
+    }
+}
+
+/// Measures the decode time of the output batches of one file, and updates
+/// the measured saving of the gated optional filters of the file. See the
+/// [module documentation](self).
+#[derive(Debug)]
+pub(crate) struct OptionalFilterSavings {
+    /// The decode speed of the scan.
+    decode_cost: Arc<DecodeCost>,
+    /// Compressed bytes for each row of the output columns of the file.
+    output_bytes_per_row: f64,
+    /// The saving of each gated optional filter of the file.
+    filters: Vec<OptionalFilterSaving>,
+}
+
+impl OptionalFilterSavings {
+    /// Returns `None` if `filters` is empty: then nothing needs the
+    /// measurement.
+    pub(crate) fn try_new(
+        decode_cost: Arc<DecodeCost>,
+        metadata: &ParquetMetaData,
+        output_projection: &ProjectionMask,
+        filters: Vec<OptionalFilterSaving>,
+    ) -> Option<Self> {
+        if filters.is_empty() {
+            return None;
+        }
+        let output_bytes_per_row = compressed_bytes_per_row(metadata, |leaf| {
+            output_projection.leaf_included(leaf)
+        });
+        Some(Self {
+            decode_cost,
+            output_bytes_per_row,
+            filters,
+        })
+    }
+
+    /// Records that the decoder produced an output batch of `rows` rows in
+    /// `elapsed`, and updates the savings.
+    pub(crate) fn record_output_batch(&self, rows: usize, elapsed: Duration) {
+        let bytes = rows as f64 * self.output_bytes_per_row;
+        if bytes < 1.0 {
+            return;
+        }
+        self.decode_cost
+            .record(duration_nanos(elapsed), bytes as u64);
+        let ns_per_byte = self.decode_cost.ns_per_byte();
+        for filter in &self.filters {
+            filter.update(ns_per_byte);
+        }
+    }
+}
+
+/// The optional filter settings of one scan, see the [module
+/// documentation](self).
+///
+/// Each file makes its own gates: gates do not share state between files
+/// or partitions. Only the decode speed ([`DecodeCost`]) is shared by all
+/// partitions and files of the scan.
+#[derive(Debug, Clone, Default)]
+pub(crate) struct OptionalFilterOptions {
+    /// How the row filter handles optional conjuncts.
+    pub(crate) mode: OptionalFilterMode,
+    /// Configuration of each gate in [`OptionalFilterMode::Adaptive`].
+    pub(crate) gate_config: OptionalFilterGateConfig,
+    /// The decode speed of the output columns of the scan.
+    pub(crate) decode_cost: Arc<DecodeCost>,
+}
diff --git a/datafusion/datasource-parquet/src/push_decoder.rs b/datafusion/datasource-parquet/src/push_decoder.rs
index 61464405a06e9..0fb99c71fa406 100644
--- a/datafusion/datasource-parquet/src/push_decoder.rs
+++ b/datafusion/datasource-parquet/src/push_decoder.rs
@@ -56,6 +56,7 @@ use parquet::arrow::push_decoder::{
 };
 use parquet::file::metadata::ParquetMetaData;
 
+use datafusion_common::instant::Instant;
 use datafusion_common::{DataFusionError, Result, internal_err};
 use datafusion_physical_expr::expressions::DynamicFilterTracking;
 use datafusion_physical_expr::utils::split_optional;
@@ -66,8 +67,10 @@ use datafusion_pruning::{PruningPredicate, PruningPredicateBuilder};
 use crate::ParquetFileMetrics;
 use crate::decoder_projection::{DecoderProjection, PostScanSelection};
 use crate::metrics::{ByteProgress, RowFilterSkippedFullyMatchedMetric};
+use crate::optional_filter::OptionalFilterSavings;
 use crate::row_filter::{
-    PrebuiltRowFilterCandidate, prebuild_row_filter_candidates, row_filter_from_prebuilt,
+    OptionalFilterRowFilterContext, PrebuiltRowFilterCandidate,
+    prebuild_row_filter_candidates, row_filter_from_prebuilt,
 };
 use crate::row_group_filter::RowGroupPruningStatistics;
 
@@ -377,6 +380,9 @@ pub(crate) struct RowFilterContext {
     pub(crate) reorder_predicates: bool,
     pub(crate) file_metrics: ParquetFileMetrics,
     pub(crate) max_predicate_cache_size: Option<usize>,
+    /// Measures the decode time of the output batches for the gates of the
+    /// optional filters. `None` if the file has no gated optional filter.
+    pub(crate) optional_savings: Option<OptionalFilterSavings>,
 }
 
 impl RowFilterContext {
@@ -396,18 +402,34 @@ impl RowFilterContext {
         reorder_predicates: bool,
         file_metrics: ParquetFileMetrics,
         max_predicate_cache_size: Option<usize>,
+        optional: Option<OptionalFilterRowFilterContext<'_>>,
     ) -> (Option<Self>, Vec<Arc<dyn PhysicalExpr>>) {
         match prebuild_row_filter_candidates(
             predicate,
             physical_file_schema,
             file_metadata.as_ref(),
+            optional,
         ) {
             Ok((prebuilt, rejected)) => {
-                let context = prebuilt.map(|prebuilt| Self {
-                    prebuilt: PrebuiltRowFilterCandidateList::new(prebuilt),
-                    reorder_predicates,
-                    file_metrics,
-                    max_predicate_cache_size,
+                let context = prebuilt.map(|prebuilt| {
+                    let optional_savings = optional.and_then(|optional| {
+                        OptionalFilterSavings::try_new(
+                            Arc::clone(&optional.options.decode_cost),
+                            file_metadata,
+                            optional.output_projection?,
+                            prebuilt
+                                .iter()
+                                .filter_map(|c| c.optional_saving().cloned())
+                                .collect(),
+                        )
+                    });
+                    Self {
+                        prebuilt: PrebuiltRowFilterCandidateList::new(prebuilt),
+                        reorder_predicates,
+                        file_metrics,
+                        max_predicate_cache_size,
+                        optional_savings,
+                    }
                 });
                 (context, rejected)
             }
@@ -512,8 +534,20 @@ impl PushDecoderStreamState {
 
             // Step 1: drain a batch from the active reader if any.
             if let Some(reader) = self.active_reader.as_mut() {
+                // The gates of optional filters need the decode time of the
+                // output columns (the reader does not evaluate the row
+                // filter: the decoder did that before it returned the reader).
+                let optional_savings = self
+                    .row_filter_context
+                    .as_ref()
+                    .and_then(|ctx| ctx.optional_savings.as_ref());
+                let start = optional_savings.map(|_| Instant::now());
                 match reader.next() {
                     Some(Ok(batch)) => {
+                        if let (Some(savings), Some(start)) = (optional_savings, start) {
+                            savings
+                                .record_output_batch(batch.num_rows(), start.elapsed());
+                        }
                         self.copy_arrow_reader_metrics();
 
                         // Apply the in-scan post-scan filter (if any). The
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 8cf0a2468c628..f3ac046ebab60 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -66,8 +66,10 @@
 //! - `WHERE s IS NOT NULL`  — not pushed down (references the whole struct)
 
 use std::sync::Arc;
+use std::time::Duration;
 
 use arrow::array::BooleanArray;
+use arrow::buffer::BooleanBuffer;
 use arrow::datatypes::{Schema, SchemaRef};
 use arrow::error::{ArrowError, Result as ArrowResult};
 use arrow::record_batch::RecordBatch;
@@ -77,14 +79,23 @@ use parquet::file::metadata::ParquetMetaData;
 
 use datafusion_common::Result;
 use datafusion_common::cast::as_boolean_array;
+use datafusion_common::config::OptionalFilterMode;
 use datafusion_common::tree_node::TreeNode;
-use datafusion_physical_expr::utils::{is_optional_filter, reassign_expr_columns};
+use datafusion_physical_expr::optional_filter_gate::{GateDecision, OptionalFilterGate};
+use datafusion_physical_expr::utils::{
+    is_optional_filter, reassign_expr_columns, split_optional,
+};
 use datafusion_physical_expr::{PhysicalExpr, split_conjunction};
 
-use datafusion_physical_plan::metrics;
+use datafusion_physical_plan::metrics::{self, ExecutionPlanMetricsSet};
+use parking_lot::Mutex;
 
 use super::ParquetFileMetrics;
 use super::supported_predicates::supports_list_predicates;
+use crate::metrics::OptionalFilterMetrics;
+use crate::optional_filter::{
+    OptionalFilterOptions, OptionalFilterSaving, compressed_bytes_per_row,
+};
 use crate::projection_read_plan::{
     ParquetReadPlan, PushdownChecker, PushdownColumns, assemble_read_plan,
     build_read_plan_with_cast_clipping,
@@ -118,6 +129,50 @@ pub(crate) struct DatafusionArrowPredicate {
     rows_matched: metrics::Count,
     /// how long was spent evaluating this predicate
     time: metrics::Time,
+    /// The gate of an optional filter in [`OptionalFilterMode::Adaptive`].
+    /// `None` for all other filters.
+    gate: Option<SharedOptionalFilterGate>,
+}
+
+/// The [`OptionalFilterGate`] of one optional filter in one file, with its
+/// metrics.
+///
+/// A file can build more than one `RowFilter` (see
+/// [`RowFilterContext::build_row_filter`](crate::push_decoder::RowFilterContext)),
+/// but only one is in use at a time. All of them share this state, thus the
+/// gate keeps its state across row groups.
+#[derive(Debug)]
+pub(crate) struct OptionalFilterGateState {
+    gate: OptionalFilterGate,
+    metrics: OptionalFilterMetrics,
+    /// Value of [`OptionalFilterGate::pauses`] already added to `metrics`.
+    reported_pauses: usize,
+}
+
+type SharedOptionalFilterGate = Arc<Mutex<OptionalFilterGateState>>;
+
+impl OptionalFilterGateState {
+    fn begin_batch(&mut self, num_rows: usize) -> GateDecision {
+        let decision = self.gate.begin_batch();
+        if decision == GateDecision::Skip {
+            self.metrics.add_rows_skipped(num_rows);
+        }
+        self.report_pauses();
+        decision
+    }
+
+    fn record(&mut self, rows_in: usize, rows_out: usize, elapsed: Duration) {
+        self.gate.record(rows_in, rows_out, elapsed);
+        self.metrics.add_eval_time(elapsed);
+        self.report_pauses();
+    }
+
+    fn report_pauses(&mut self) {
+        let pauses = self.gate.pauses();
+        self.metrics
+            .add_pauses(pauses.saturating_sub(self.reported_pauses));
+        self.reported_pauses = pauses;
+    }
 }
 
 impl DatafusionArrowPredicate {
@@ -142,6 +197,7 @@ impl DatafusionArrowPredicate {
             rows_pruned,
             rows_matched,
             time,
+            gate: None,
         })
     }
 }
@@ -154,16 +210,33 @@ impl ArrowPredicate for DatafusionArrowPredicate {
     fn evaluate(&mut self, batch: RecordBatch) -> ArrowResult<BooleanArray> {
         // scoped timer updates on drop
         let mut timer = self.time.timer();
+        let num_rows = batch.num_rows();
+
+        // An optional filter that the gate skips lets all rows pass.
+        let mut gate = self.gate.as_ref().map(|gate| gate.lock());
+        if let Some(gate) = gate.as_mut()
+            && gate.begin_batch(num_rows) == GateDecision::Skip
+        {
+            self.rows_matched.add(num_rows);
+            timer.stop();
+            return Ok(BooleanArray::new(BooleanBuffer::new_set(num_rows), None));
+        }
+        // The gate measures the evaluation time with its clock.
+        let start = gate.as_ref().map(|gate| gate.gate.clock().now_nanos());
 
         self.physical_expr
             .evaluate(&batch)
-            .and_then(|v| v.into_array(batch.num_rows()))
+            .and_then(|v| v.into_array(num_rows))
             .and_then(|array| {
                 let bool_arr = as_boolean_array(&array)?.clone();
                 let num_matched = bool_arr.true_count();
                 let num_pruned = bool_arr.len() - num_matched;
                 self.rows_pruned.add(num_pruned);
                 self.rows_matched.add(num_matched);
+                if let (Some(gate), Some(start)) = (gate.as_mut(), start) {
+                    let elapsed = gate.gate.clock().now_nanos().saturating_sub(start);
+                    gate.record(num_rows, num_matched, Duration::from_nanos(elapsed));
+                }
                 timer.stop();
                 Ok(bool_arr)
             })
@@ -444,8 +517,11 @@ pub fn build_row_filter(
     // that splits conjuncts, orders candidates, and wires metrics — callers
     // that build once per file go through the same code as the per-row-group
     // rebuild path in `RowFilterContext`.
+    //
+    // Optional conjuncts are used like all other conjuncts here (the
+    // `always` optional filter mode).
     let (prebuilt, rejected) =
-        prebuild_row_filter_candidates(expr, file_schema, metadata)?;
+        prebuild_row_filter_candidates(expr, file_schema, metadata, None)?;
     let row_filter = prebuilt.map(|prebuilt| {
         row_filter_from_prebuilt(&prebuilt, reorder_predicates, file_metrics)
     });
@@ -474,12 +550,49 @@ pub(crate) struct PrebuiltRowFilterCandidate {
     /// `reorder_predicates` is enabled. Stable across row groups within a
     /// file, so we cache it once.
     required_bytes: usize,
+    /// The gate of an optional filter in [`OptionalFilterMode::Adaptive`],
+    /// shared by all `RowFilter`s built for the file. `None` for all other
+    /// filters.
+    gate: Option<SharedOptionalFilterGate>,
+    /// The measured saving that the gate reads, if the output projection of
+    /// the file is known. See [`crate::optional_filter`].
+    saving: Option<OptionalFilterSaving>,
 }
 
 impl PrebuiltRowFilterCandidate {
     pub(crate) fn reads_leaf(&self, leaf_idx: usize) -> bool {
         self.projection_mask.leaf_included(leaf_idx)
     }
+
+    /// The measured saving of a gated optional filter.
+    pub(crate) fn optional_saving(&self) -> Option<&OptionalFilterSaving> {
+        self.saving.as_ref()
+    }
+
+    /// True if this is a gated optional filter.
+    pub(crate) fn is_optional(&self) -> bool {
+        self.gate.is_some()
+    }
+}
+
+/// What [`prebuild_row_filter_candidates`] needs to handle optional
+/// conjuncts (conjuncts wrapped in an `OptionalFilterPhysicalExpr`) for one
+/// file.
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct OptionalFilterRowFilterContext<'a> {
+    /// The optional filter settings of the scan.
+    pub(crate) options: &'a OptionalFilterOptions,
+    /// Metrics of the scan, for the lazily-registered gate metrics.
+    pub(crate) metrics: &'a ExecutionPlanMetricsSet,
+    /// Execution partition index.
+    pub(crate) partition: usize,
+    /// Name of the file.
+    pub(crate) filename: &'a str,
+    /// The columns that the decoder produces for the file. The gates use it
+    /// to estimate the decode time that a removed row saves (see
+    /// [`crate::optional_filter`]). If `None`, the gates use only the
+    /// configured minimum saving.
+    pub(crate) output_projection: Option<&'a ProjectionMask>,
 }
 
 /// Precompute the list of [`PrebuiltRowFilterCandidate`]s for a predicate.
@@ -493,44 +606,74 @@ impl PrebuiltRowFilterCandidate {
 ///
 /// The first element is `None` when the predicate has no push-downable
 /// conjuncts, in which case callers should skip installing a `RowFilter`
-/// entirely. The second element holds the conjuncts that cannot be evaluated
-/// as an `ArrowPredicate` on this file; see [`build_row_filter`] for why the
-/// caller must apply them elsewhere.
+/// entirely. The second element holds the required conjuncts that cannot be
+/// evaluated as an `ArrowPredicate` on this file; see [`build_row_filter`]
+/// for why the caller must apply them elsewhere.
+///
+/// # Optional conjuncts
+///
+/// `optional` controls the conjuncts of the root `AND` chain that are
+/// wrapped in an `OptionalFilterPhysicalExpr` (see
+/// [`split_optional`]). The other (required) conjuncts are always used.
 ///
-/// An optional conjunct (an `OptionalFilterPhysicalExpr` on the root `AND`
-/// chain) that cannot be evaluated as an `ArrowPredicate` for this file is
-/// not used. It is not in the rejected conjuncts either, thus the caller does
+/// * `None` or [`OptionalFilterMode::Always`]: optional conjuncts are used
+///   like required conjuncts (the `OptionalFilterPhysicalExpr` wrapper is
+///   transparent).
+/// * [`OptionalFilterMode::PruningOnly`]: optional conjuncts are not used.
+///   Statistics pruning still uses them, because it uses the complete
+///   predicate.
+/// * [`OptionalFilterMode::Adaptive`]: each optional conjunct is a separate
+///   candidate with an [`OptionalFilterGate`], which skips the conjunct while
+///   it costs more than it saves. Each file has its own gates. Optional
+///   candidates come after all required
+///   candidates, see [`row_filter_from_prebuilt`].
+///
+/// An optional conjunct that cannot be evaluated as an `ArrowPredicate` for
+/// this file (for example because of schema evolution) is not used, in all
+/// modes. It is not in the rejected conjuncts either, thus the caller does
 /// not evaluate it after the scan. This is always safe.
 #[expect(clippy::type_complexity)]
 pub(crate) fn prebuild_row_filter_candidates(
     expr: &Arc<dyn PhysicalExpr>,
     file_schema: &SchemaRef,
     metadata: &ParquetMetaData,
+    optional: Option<OptionalFilterRowFilterContext<'_>>,
 ) -> Result<(
     Option<Vec<PrebuiltRowFilterCandidate>>,
     Vec<Arc<dyn PhysicalExpr>>,
 )> {
+    let mode = optional.map_or(OptionalFilterMode::Always, |o| o.options.mode);
+
     // Split into conjuncts:
     // `a = 1 AND b = 2 AND c = 3` -> [`a = 1`, `b = 2`, `c = 3`]
-    let predicates = split_conjunction(expr);
+    let (required, optional_conjuncts) = match mode {
+        OptionalFilterMode::Always => (
+            split_conjunction(expr)
+                .into_iter()
+                .map(Arc::clone)
+                .collect(),
+            vec![],
+        ),
+        OptionalFilterMode::PruningOnly => (split_optional(expr).0, vec![]),
+        OptionalFilterMode::Adaptive => split_optional(expr),
+    };
 
-    // Partition conjuncts into those that can be evaluated as ArrowPredicates
-    // and those that cannot. Rejected conjuncts are returned to the caller so
-    // they are never silently dropped.
-    let mut candidates: Vec<FilterCandidate> = Vec::with_capacity(predicates.len());
+    // Required conjuncts that cannot be evaluated as ArrowPredicates are
+    // returned to the caller so they are never silently dropped.
+    let mut prebuilt = Vec::with_capacity(required.len() + optional_conjuncts.len());
     let mut rejected: Vec<Arc<dyn PhysicalExpr>> = Vec::new();
-    for predicate in predicates {
-        // An optional conjunct that cannot be pushed down for this file is not
-        // needed for correctness: do not use it, and do not send it to the
-        // post-scan filter.
-        if is_optional_filter(predicate) {
-            match FilterCandidateBuilder::new(
-                Arc::clone(predicate),
-                Arc::clone(file_schema),
-            )
-            .build(metadata)
+    for conjunct in required {
+        // In the `always` mode, `required` also holds the optional conjuncts
+        // (still wrapped). An optional conjunct that cannot be pushed down
+        // for this file is not needed for correctness: do not use it, and do
+        // not send it to the post-scan filter.
+        if is_optional_filter(&conjunct) {
+            match FilterCandidateBuilder::new(conjunct, Arc::clone(file_schema))
+                .build(metadata)
             {
-                Ok(Some(candidate)) => candidates.push(candidate),
+                Ok(Some(candidate)) => {
+                    prebuilt.push(PrebuiltRowFilterCandidate::try_new(candidate)?)
+                }
                 Ok(None) => {}
                 Err(e) => {
                     log::debug!(
@@ -540,40 +683,122 @@ pub(crate) fn prebuild_row_filter_candidates(
             }
             continue;
         }
-        match FilterCandidateBuilder::new(Arc::clone(predicate), Arc::clone(file_schema))
+        match FilterCandidateBuilder::new(Arc::clone(&conjunct), Arc::clone(file_schema))
             .build(metadata)?
         {
-            Some(candidate) => candidates.push(candidate),
-            None => rejected.push(Arc::clone(predicate)),
+            Some(candidate) => {
+                prebuilt.push(PrebuiltRowFilterCandidate::try_new(candidate)?)
+            }
+            None => rejected.push(conjunct),
         }
     }
 
-    if candidates.is_empty() {
+    if let Some(optional) = optional {
+        for conjunct in optional_conjuncts {
+            // An optional conjunct that cannot be pushed down for this file
+            // (or that fails to build) is not needed for correctness, thus it
+            // is simply not used.
+            let candidate =
+                match FilterCandidateBuilder::new(conjunct, Arc::clone(file_schema))
+                    .build(metadata)
+                {
+                    Ok(Some(candidate)) => candidate,
+                    Ok(None) => continue,
+                    Err(e) => {
+                        log::debug!(
+                            "Ignoring optional filter that cannot be pushed down: {e}"
+                        );
+                        continue;
+                    }
+                };
+            let mut candidate = PrebuiltRowFilterCandidate::try_new(candidate)?;
+            // The gate watches the dynamic filters in the same (live)
+            // expression that the predicate evaluates, thus it sees their
+            // updates.
+            let mut gate = OptionalFilterGate::new(
+                Arc::clone(&candidate.physical_expr),
+                optional.options.gate_config,
+            );
+            // A removed row saves the decode of the output columns that the
+            // filter does not read.
+            if let Some(output) = optional.output_projection {
+                let unread_bytes_per_row = compressed_bytes_per_row(metadata, |leaf| {
+                    output.leaf_included(leaf)
+                        && !candidate.projection_mask.leaf_included(leaf)
+                });
+                let saving = OptionalFilterSaving::new(
+                    unread_bytes_per_row,
+                    &optional.options.decode_cost,
+                );
+                gate = gate.with_measured_saving(Arc::clone(&saving.saving));
+                candidate.saving = Some(saving);
+            }
+            candidate.gate = Some(Arc::new(Mutex::new(OptionalFilterGateState {
+                gate,
+                metrics: OptionalFilterMetrics::new(
+                    optional.metrics,
+                    optional.partition,
+                    optional.filename,
+                ),
+                reported_pauses: 0,
+            })));
+            prebuilt.push(candidate);
+        }
+    }
+
+    if prebuilt.is_empty() {
         return Ok((None, rejected));
     }
+    Ok((Some(prebuilt), rejected))
+}
 
-    let prebuilt: Vec<PrebuiltRowFilterCandidate> = candidates
-        .into_iter()
-        .map(|candidate| {
-            let physical_expr = reassign_expr_columns(
-                Arc::clone(&candidate.expr),
-                &candidate.read_plan.projected_schema,
-            )?;
-            Ok(PrebuiltRowFilterCandidate {
-                physical_expr,
-                projection_mask: candidate.read_plan.projection_mask.clone(),
-                required_bytes: candidate.required_bytes,
-            })
+impl PrebuiltRowFilterCandidate {
+    /// A candidate without a gate. Its expression is column-reassigned to
+    /// the projected schema of the candidate.
+    fn try_new(candidate: FilterCandidate) -> Result<Self> {
+        let physical_expr =
+            reassign_expr_columns(candidate.expr, &candidate.read_plan.projected_schema)?;
+        Ok(Self {
+            physical_expr,
+            projection_mask: candidate.read_plan.projection_mask,
+            required_bytes: candidate.required_bytes,
+            gate: None,
+            saving: None,
         })
-        .collect::<Result<Vec<_>>>()?;
+    }
+}
 
-    Ok((Some(prebuilt), rejected))
+/// Returns the candidates in evaluation order: the order of `prebuilt`, or
+/// ordered by `required_bytes` if `reorder_predicates` is true. Gated
+/// optional candidates always come last (see [`row_filter_from_prebuilt`]).
+fn order_candidates(
+    prebuilt: &[PrebuiltRowFilterCandidate],
+    reorder_predicates: bool,
+) -> Vec<&PrebuiltRowFilterCandidate> {
+    // Collect references into a working list we can sort without disturbing
+    // the shared cache.
+    let mut ordered: Vec<&PrebuiltRowFilterCandidate> = prebuilt.iter().collect();
+    if reorder_predicates {
+        ordered.sort_unstable_by_key(|c| (c.is_optional(), c.required_bytes));
+    } else {
+        // `prebuild_row_filter_candidates` already puts the optional
+        // candidates last. A stable sort keeps the other order.
+        ordered.sort_by_key(|c| c.is_optional());
+    }
+    ordered
 }
 
 /// Wrap a list of prebuilt candidates into a fresh [`RowFilter`], assigning
 /// per-predicate metric counters and (optionally) reordering by
 /// `required_bytes`. This is the cheap per-row-group rebuild path — no tree
 /// walks, no column resolution, only counter allocation.
+///
+/// Gated optional candidates (see [`prebuild_row_filter_candidates`]) always
+/// come after the required candidates, also when `reorder_predicates` is
+/// true. The required candidates must run in all cases. When they run first,
+/// the optional candidates see only the rows that the required candidates
+/// let pass: the gate then measures the selectivity that the optional filter
+/// adds, and a skipped optional filter costs nothing.
 pub(crate) fn row_filter_from_prebuilt(
     prebuilt: &[PrebuiltRowFilterCandidate],
     reorder_predicates: bool,
@@ -583,13 +808,7 @@ pub(crate) fn row_filter_from_prebuilt(
     let rows_matched = &file_metrics.pushdown_rows_matched;
     let time = &file_metrics.row_pushdown_eval_time;
 
-    // Clone (cheap: Arc bumps + ProjectionMask clone) into a working list we
-    // can sort without disturbing the shared cache.
-    let mut ordered: Vec<&PrebuiltRowFilterCandidate> = prebuilt.iter().collect();
-    if reorder_predicates {
-        ordered.sort_unstable_by_key(|c| c.required_bytes);
-    }
-
+    let ordered = order_candidates(prebuilt, reorder_predicates);
     let total = ordered.len();
     let filters: Vec<Box<dyn ArrowPredicate>> = ordered
         .into_iter()
@@ -608,6 +827,7 @@ pub(crate) fn row_filter_from_prebuilt(
                 rows_pruned: predicate_rows_pruned,
                 rows_matched: predicate_rows_matched,
                 time: time.clone(),
+                gate: candidate.gate.clone(),
             }) as Box<dyn ArrowPredicate>
         })
         .collect();
@@ -2265,3 +2485,578 @@ mod test {
         );
     }
 }
+
+#[cfg(test)]
+mod optional_filter_tests {
+    use super::*;
+    use crate::optional_filter::OptionalFilterOptions;
+    use arrow::array::{Array, Int32Array, StructArray};
+    use arrow::datatypes::{DataType, Field, Fields};
+    use datafusion_common::ScalarValue;
+    use datafusion_expr::Operator;
+    use datafusion_physical_expr::conjunction;
+    use datafusion_physical_expr::expressions::{
+        BinaryExpr, DynamicFilterPhysicalExpr, IsNotNullExpr, OptionalFilterPhysicalExpr,
+        col, lit,
+    };
+    use datafusion_physical_expr::optional_filter_gate::OptionalFilterGateConfig;
+    use datafusion_physical_expr::simplifier::PhysicalExprSimplifier;
+    use datafusion_physical_expr_adapter::{
+        DefaultPhysicalExprAdapterFactory, PhysicalExprAdapterFactory,
+    };
+    use parquet::arrow::ArrowWriter;
+    use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+    use parquet::file::properties::WriterProperties;
+    use tempfile::NamedTempFile;
+
+    const NUM_ROWS: usize = 10_000;
+    const FILENAME: &str = "test.parquet";
+
+    /// A file with `a` (many distinct values: the largest column), `b`
+    /// (`i % 10`), `c` (`(i / 10) % 10`) and a struct column `s`.
+    fn test_file() -> (NamedTempFile, Arc<ParquetMetaData>, SchemaRef) {
+        let s_fields = Fields::from(vec![Field::new("x", DataType::Int32, true)]);
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("a", DataType::Int32, false),
+            Field::new("b", DataType::Int32, false),
+            Field::new("c", DataType::Int32, false),
+            Field::new("s", DataType::Struct(s_fields.clone()), true),
+        ]));
+        let a: Int32Array = (0..NUM_ROWS as i32).map(|i| i * 7919).collect();
+        let b: Int32Array = (0..NUM_ROWS as i32).map(|i| i % 10).collect();
+        let c: Int32Array = (0..NUM_ROWS as i32).map(|i| (i / 10) % 10).collect();
+        let s = StructArray::new(
+            s_fields,
+            vec![Arc::new(Int32Array::from(vec![1; NUM_ROWS])) as _],
+            None,
+        );
+        let batch = RecordBatch::try_new(
+            Arc::clone(&schema),
+            vec![Arc::new(a), Arc::new(b), Arc::new(c), Arc::new(s)],
+        )
+        .unwrap();
+        let file = NamedTempFile::new().unwrap();
+        // Dictionary encoding makes `b` and `c` much smaller than `a`.
+        let props = WriterProperties::builder()
+            .set_dictionary_enabled(true)
+            .build();
+        let mut writer = ArrowWriter::try_new(
+            file.reopen().unwrap(),
+            Arc::clone(&schema),
+            Some(props),
+        )
+        .unwrap();
+        writer.write(&batch).unwrap();
+        writer.close().unwrap();
+        let builder =
+            ParquetRecordBatchReaderBuilder::try_new(file.reopen().unwrap()).unwrap();
+        let metadata = Arc::clone(builder.metadata());
+        let file_schema = Arc::clone(builder.schema());
+        (file, metadata, file_schema)
+    }
+
+    fn optional(inner: Arc<dyn PhysicalExpr>) -> Arc<dyn PhysicalExpr> {
+        Arc::new(OptionalFilterPhysicalExpr::new(inner))
+    }
+
+    fn binary(
+        left: Arc<dyn PhysicalExpr>,
+        op: Operator,
+        right: Arc<dyn PhysicalExpr>,
+    ) -> Arc<dyn PhysicalExpr> {
+        Arc::new(BinaryExpr::new(left, op, right))
+    }
+
+    fn col_op(
+        name: &str,
+        op: Operator,
+        value: i32,
+        schema: &Schema,
+    ) -> Arc<dyn PhysicalExpr> {
+        binary(
+            col(name, schema).unwrap(),
+            op,
+            lit(ScalarValue::Int32(Some(value))),
+        )
+    }
+
+    /// Options with a very large `min_saving_ns_per_row`: the cost check of
+    /// the gates (which uses the wall clock) never pauses a filter that
+    /// removes rows, thus the tests are deterministic. See
+    /// [`options_with_saving`].
+    fn options(mode: OptionalFilterMode) -> OptionalFilterOptions {
+        options_with_saving(mode, 1e9)
+    }
+
+    fn options_with_saving(
+        mode: OptionalFilterMode,
+        min_saving_ns_per_row: f64,
+    ) -> OptionalFilterOptions {
+        OptionalFilterOptions {
+            mode,
+            gate_config: OptionalFilterGateConfig {
+                min_saving_ns_per_row,
+                ..Default::default()
+            },
+            decode_cost: Arc::default(),
+        }
+    }
+
+    fn context<'a>(
+        options: &'a OptionalFilterOptions,
+        metrics: &'a ExecutionPlanMetricsSet,
+    ) -> Option<OptionalFilterRowFilterContext<'a>> {
+        Some(OptionalFilterRowFilterContext {
+            options,
+            metrics,
+            partition: 0,
+            filename: FILENAME,
+            output_projection: None,
+        })
+    }
+
+    fn prebuild(
+        predicate: &Arc<dyn PhysicalExpr>,
+        file_schema: &SchemaRef,
+        metadata: &ParquetMetaData,
+        options: &OptionalFilterOptions,
+        metrics: &ExecutionPlanMetricsSet,
+    ) -> Vec<PrebuiltRowFilterCandidate> {
+        prebuild_row_filter_candidates(
+            predicate,
+            file_schema,
+            metadata,
+            context(options, metrics),
+        )
+        .unwrap()
+        .0
+        .unwrap_or_default()
+    }
+
+    /// `(expression, is_optional)` of each candidate, in evaluation order.
+    fn describe(
+        candidates: &[PrebuiltRowFilterCandidate],
+        reorder: bool,
+    ) -> Vec<(String, bool)> {
+        order_candidates(candidates, reorder)
+            .into_iter()
+            .map(|c| (c.physical_expr.to_string(), c.is_optional()))
+            .collect()
+    }
+
+    fn entry(expr: &str, optional: bool) -> (String, bool) {
+        (expr.to_string(), optional)
+    }
+
+    #[test]
+    fn candidates_for_each_mode() {
+        let (_file, metadata, file_schema) = test_file();
+        let metrics = ExecutionPlanMetricsSet::new();
+        // Optional(b > 3) AND a > 5 AND Optional(c < 10)
+        let predicate = conjunction([
+            optional(col_op("b", Operator::Gt, 3, &file_schema)),
+            col_op("a", Operator::Gt, 5, &file_schema),
+            optional(col_op("c", Operator::Lt, 10, &file_schema)),
+        ]);
+
+        // always: the same candidates as without optional filter handling.
+        let options_always = options(OptionalFilterMode::Always);
+        let candidates = prebuild(
+            &predicate,
+            &file_schema,
+            &metadata,
+            &options_always,
+            &metrics,
+        );
+        assert_eq!(
+            describe(&candidates, false),
+            vec![
+                entry("Optional(b@0 > 3)", false),
+                entry("a@0 > 5", false),
+                entry("Optional(c@0 < 10)", false),
+            ]
+        );
+        let without_context =
+            prebuild_row_filter_candidates(&predicate, &file_schema, &metadata, None)
+                .unwrap()
+                .0
+                .unwrap();
+        assert_eq!(
+            describe(&without_context, false),
+            describe(&candidates, false)
+        );
+
+        // pruning_only: only the required conjunct.
+        let options_pruning = options(OptionalFilterMode::PruningOnly);
+        let candidates = prebuild(
+            &predicate,
+            &file_schema,
+            &metadata,
+            &options_pruning,
+            &metrics,
+        );
+        assert_eq!(describe(&candidates, false), vec![entry("a@0 > 5", false)]);
+
+        // pruning_only with optional conjuncts only: no row filter.
+        let only_optional = optional(col_op("b", Operator::Gt, 3, &file_schema));
+        let prebuilt = prebuild_row_filter_candidates(
+            &only_optional,
+            &file_schema,
+            &metadata,
+            context(&options_pruning, &metrics),
+        )
+        .unwrap();
+        assert!(prebuilt.0.is_none());
+        assert!(prebuilt.1.is_empty());
+
+        // adaptive: required first, then each optional inner with a gate.
+        let options_adaptive = options(OptionalFilterMode::Adaptive);
+        let candidates = prebuild(
+            &predicate,
+            &file_schema,
+            &metadata,
+            &options_adaptive,
+            &metrics,
+        );
+        assert_eq!(
+            describe(&candidates, false),
+            vec![
+                entry("a@0 > 5", false),
+                entry("b@0 > 3", true),
+                entry("c@0 < 10", true),
+            ]
+        );
+    }
+
+    #[test]
+    fn optional_candidates_stay_last_when_reordered() {
+        let (_file, metadata, file_schema) = test_file();
+        let metrics = ExecutionPlanMetricsSet::new();
+        // `a` is the largest column, thus a reorder puts it last among the
+        // required conjuncts. The optional conjunct on the small column `b`
+        // still comes after all required conjuncts.
+        let predicate = conjunction([
+            optional(col_op("b", Operator::Gt, 3, &file_schema)),
+            col_op("a", Operator::Gt, 5, &file_schema),
+            col_op("c", Operator::Lt, 10, &file_schema),
+        ]);
+        let options = options(OptionalFilterMode::Adaptive);
+        let candidates =
+            prebuild(&predicate, &file_schema, &metadata, &options, &metrics);
+        assert_eq!(
+            describe(&candidates, true),
+            vec![
+                entry("c@0 < 10", false),
+                entry("a@0 > 5", false),
+                entry("b@0 > 3", true),
+            ]
+        );
+        assert_eq!(
+            describe(&candidates, false),
+            vec![
+                entry("a@0 > 5", false),
+                entry("c@0 < 10", false),
+                entry("b@0 > 3", true),
+            ]
+        );
+    }
+
+    #[test]
+    fn rejected_optional_conjunct_is_dropped() {
+        let (_file, metadata, file_schema) = test_file();
+        let metrics = ExecutionPlanMetricsSet::new();
+        // A filter on a whole struct column cannot be pushed down.
+        let struct_filter: Arc<dyn PhysicalExpr> =
+            Arc::new(IsNotNullExpr::new(col("s", &file_schema).unwrap()));
+        let predicate = conjunction([
+            col_op("a", Operator::Gt, 5, &file_schema),
+            optional(struct_filter),
+            optional(col_op("b", Operator::Gt, 3, &file_schema)),
+        ]);
+        let options = options(OptionalFilterMode::Adaptive);
+        let candidates =
+            prebuild(&predicate, &file_schema, &metadata, &options, &metrics);
+        assert_eq!(
+            describe(&candidates, false),
+            vec![entry("a@0 > 5", false), entry("b@0 > 3", true)]
+        );
+    }
+
+    /// Read the file with a `RowFilter` from `candidates`, in batches of 100
+    /// rows. Returns the number of output rows.
+    fn read_with_row_filter(
+        file: &NamedTempFile,
+        candidates: &[PrebuiltRowFilterCandidate],
+        file_metrics: &ParquetFileMetrics,
+    ) -> usize {
+        let row_filter = row_filter_from_prebuilt(candidates, false, file_metrics);
+        let reader = ParquetRecordBatchReaderBuilder::try_new(file.reopen().unwrap())
+            .unwrap()
+            .with_batch_size(100)
+            .with_row_filter(row_filter)
+            .build()
+            .unwrap();
+        reader.map(|batch| batch.unwrap().num_rows()).sum()
+    }
+
+    fn metric(metrics: &ExecutionPlanMetricsSet, name: &str) -> usize {
+        metrics
+            .clone_inner()
+            .sum_by_name(name)
+            .map_or(0, |v| v.as_usize())
+    }
+
+    #[test]
+    fn adaptive_pauses_optional_filter_that_removes_no_rows() {
+        let (file, metadata, file_schema) = test_file();
+        // All rows pass `b >= 0`: the filter removes no rows, thus the gate
+        // pauses it.
+        let predicate = optional(col_op("b", Operator::GtEq, 0, &file_schema));
+
+        let metrics = ExecutionPlanMetricsSet::new();
+        let file_metrics = ParquetFileMetrics::new(0, FILENAME, &metrics);
+        let options_always = options(OptionalFilterMode::Always);
+        let candidates = prebuild(
+            &predicate,
+            &file_schema,
+            &metadata,
+            &options_always,
+            &metrics,
+        );
+        let rows_always = read_with_row_filter(&file, &candidates, &file_metrics);
+        assert_eq!(rows_always, NUM_ROWS);
+        assert_eq!(metric(&metrics, "optional_filter_rows_skipped"), 0);
+        assert_eq!(metric(&metrics, "optional_filter_pauses"), 0);
+
+        let metrics = ExecutionPlanMetricsSet::new();
+        let file_metrics = ParquetFileMetrics::new(0, FILENAME, &metrics);
+        let options_adaptive = options(OptionalFilterMode::Adaptive);
+        let candidates = prebuild(
+            &predicate,
+            &file_schema,
+            &metadata,
+            &options_adaptive,
+            &metrics,
+        );
+        let rows_adaptive = read_with_row_filter(&file, &candidates, &file_metrics);
+
+        let skipped = metric(&metrics, "optional_filter_rows_skipped");
+        assert!(skipped > 0, "expected skipped rows");
+        assert!(metric(&metrics, "optional_filter_pauses") > 0);
+        // Skipped rows pass without evaluation.
+        assert_eq!(rows_adaptive, rows_always);
+        assert_eq!(file_metrics.pushdown_rows_matched.value(), rows_adaptive);
+
+        // Gates do not share state: the gate for the next file starts to
+        // evaluate the filter.
+        let candidates = prebuild(
+            &predicate,
+            &file_schema,
+            &metadata,
+            &options_adaptive,
+            &metrics,
+        );
+        let gate = candidates[0].gate.as_ref().unwrap();
+        assert!(!gate.lock().gate.is_paused());
+    }
+
+    #[test]
+    fn adaptive_keeps_selective_optional_filter() {
+        let (file, metadata, file_schema) = test_file();
+        // 10% of the rows pass `b = 1`.
+        let predicate = optional(col_op("b", Operator::Eq, 1, &file_schema));
+        let metrics = ExecutionPlanMetricsSet::new();
+        let file_metrics = ParquetFileMetrics::new(0, FILENAME, &metrics);
+        let options = options(OptionalFilterMode::Adaptive);
+        let candidates =
+            prebuild(&predicate, &file_schema, &metadata, &options, &metrics);
+        let rows = read_with_row_filter(&file, &candidates, &file_metrics);
+        assert_eq!(rows, NUM_ROWS / 10);
+        assert_eq!(metric(&metrics, "optional_filter_rows_skipped"), 0);
+        assert_eq!(metric(&metrics, "optional_filter_pauses"), 0);
+    }
+
+    /// The cost check pauses a selective optional filter that costs more
+    /// than it saves. With no output projection (thus no measured saving)
+    /// and `min_saving_ns_per_row = 0`, a removed row saves nothing and any
+    /// evaluation time is too much: the gate pauses `b = 1` although it
+    /// keeps only 10% of the rows.
+    #[test]
+    fn adaptive_pauses_expensive_optional_filter() {
+        let (file, metadata, file_schema) = test_file();
+        let predicate = optional(col_op("b", Operator::Eq, 1, &file_schema));
+        let metrics = ExecutionPlanMetricsSet::new();
+        let file_metrics = ParquetFileMetrics::new(0, FILENAME, &metrics);
+        let options = options_with_saving(OptionalFilterMode::Adaptive, 0.0);
+        let candidates =
+            prebuild(&predicate, &file_schema, &metadata, &options, &metrics);
+        let rows = read_with_row_filter(&file, &candidates, &file_metrics);
+        let skipped = metric(&metrics, "optional_filter_rows_skipped");
+        assert!(skipped > 0, "expected skipped rows");
+        assert!(metric(&metrics, "optional_filter_pauses") > 0);
+        assert!(metric(&metrics, "optional_filter_eval_time") > 0);
+        // Skipped rows pass without evaluation.
+        assert_eq!(rows, NUM_ROWS / 10 + skipped * 9 / 10);
+    }
+
+    /// With the output projection of the file, the gate of an optional
+    /// filter adds the decode time of the output columns that the filter
+    /// does not read to the configured minimum saving.
+    #[test]
+    fn saving_includes_decode_of_unread_output_columns() {
+        use crate::optional_filter::{
+            DEFAULT_DECODE_NS_PER_BYTE, OptionalFilterSavings, compressed_bytes_per_row,
+        };
+        let (_file, metadata, file_schema) = test_file();
+        // The output columns are `a`, `b` and `c` (leaves 0, 1 and 2). The
+        // optional filter reads `b`.
+        let output =
+            ProjectionMask::leaves(metadata.file_metadata().schema_descr(), [0, 1, 2]);
+        let predicate = optional(col_op("b", Operator::Eq, 1, &file_schema));
+        let metrics = ExecutionPlanMetricsSet::new();
+        let options = options_with_saving(OptionalFilterMode::Adaptive, 5.0);
+        let mut context = context(&options, &metrics).unwrap();
+        context.output_projection = Some(&output);
+        let candidates = prebuild_row_filter_candidates(
+            &predicate,
+            &file_schema,
+            &metadata,
+            Some(context),
+        )
+        .unwrap()
+        .0
+        .unwrap();
+        let saving = candidates[0].optional_saving().unwrap();
+        let rows = NUM_ROWS as f64;
+        let column_bytes =
+            |leaf: usize| metadata.row_group(0).column(leaf).compressed_size() as f64;
+        let unread = (column_bytes(0) + column_bytes(2)) / rows;
+        assert!(unread > 0.0);
+        assert_eq!(saving.unread_bytes_per_row, unread);
+        assert_eq!(
+            compressed_bytes_per_row(&metadata, |leaf| leaf == 0 || leaf == 2),
+            unread
+        );
+        // Before any measurement: the default decode speed.
+        let gate = candidates[0].gate.as_ref().unwrap();
+        let expected = 5.0 + DEFAULT_DECODE_NS_PER_BYTE * unread;
+        assert!((gate.lock().gate.saving_ns_per_row() - expected).abs() < 1e-9);
+
+        // Output batches of 1000 rows that take 4 ns for each compressed
+        // byte of the output columns.
+        let output_bytes = (column_bytes(0) + column_bytes(1) + column_bytes(2)) / rows;
+        let savings = OptionalFilterSavings::try_new(
+            Arc::clone(&options.decode_cost),
+            &metadata,
+            &output,
+            vec![saving.clone()],
+        )
+        .unwrap();
+        let nanos = (4.0 * 1000.0 * output_bytes) as u64;
+        savings.record_output_batch(1000, Duration::from_nanos(nanos));
+        let ns_per_byte = options.decode_cost.ns_per_byte();
+        assert!((ns_per_byte - 4.0).abs() < 0.01, "{ns_per_byte}");
+        let expected = 5.0 + ns_per_byte * unread;
+        assert!((gate.lock().gate.saving_ns_per_row() - expected).abs() < 1e-9);
+
+        // No gated optional filter: nothing to measure.
+        assert!(
+            OptionalFilterSavings::try_new(
+                Arc::clone(&options.decode_cost),
+                &metadata,
+                &output,
+                vec![],
+            )
+            .is_none()
+        );
+    }
+
+    /// The per-file rewrite (schema adaptation and simplification) keeps
+    /// the `Optional` wrapper on the root AND chain, and a dynamic filter in
+    /// it stays live in the row filter: the gate sees its updates.
+    #[test]
+    fn optional_dynamic_filter_survives_rewrite_and_stays_live() {
+        let (_file, metadata, file_schema) = test_file();
+        // The table has `b` as Int64 and the file has Int32: the adapter
+        // inserts a cast.
+        let table_schema = Arc::new(Schema::new(vec![
+            Field::new("a", DataType::Int32, false),
+            Field::new("b", DataType::Int64, false),
+        ]));
+        let b = col("b", &table_schema).unwrap();
+        let b_gt = |value: i64| {
+            binary(
+                Arc::clone(&b),
+                Operator::Gt,
+                lit(ScalarValue::Int64(Some(value))),
+            )
+        };
+        let dynamic = Arc::new(DynamicFilterPhysicalExpr::new(
+            vec![Arc::clone(&b)],
+            b_gt(0),
+        ));
+        let predicate = conjunction([
+            col_op("a", Operator::Gt, 5, &table_schema),
+            optional(Arc::clone(&dynamic) as Arc<dyn PhysicalExpr>),
+        ]);
+
+        let rewriter = DefaultPhysicalExprAdapterFactory
+            .create(Arc::clone(&table_schema), Arc::clone(&file_schema))
+            .unwrap();
+        let simplifier = PhysicalExprSimplifier::new(&file_schema);
+        let rewritten = simplifier
+            .simplify(rewriter.rewrite(Arc::clone(&predicate)).unwrap())
+            .unwrap();
+        let (required, optional_filters) = split_optional(&rewritten);
+        assert_eq!(required.len(), 1);
+        assert_eq!(optional_filters.len(), 1);
+
+        let metrics = ExecutionPlanMetricsSet::new();
+        let options = options(OptionalFilterMode::Adaptive);
+        let candidates =
+            prebuild(&rewritten, &file_schema, &metadata, &options, &metrics);
+        assert_eq!(candidates.len(), 2);
+        let optional_candidate = candidates.iter().find(|c| c.is_optional()).unwrap();
+        let gate = optional_candidate.gate.as_ref().unwrap();
+
+        // Pause the gate with non-selective batches.
+        {
+            let mut state = gate.lock();
+            for _ in 0..10 {
+                if state.gate.is_paused() {
+                    break;
+                }
+                assert_eq!(state.gate.begin_batch(), GateDecision::Evaluate);
+                state.gate.record(100, 100, Duration::ZERO);
+            }
+            assert!(state.gate.is_paused());
+        }
+
+        // The gate sees an update of the original dynamic filter: the pause
+        // ends at once. The row filter evaluates the new expression.
+        dynamic.update(b_gt(4)).unwrap();
+        {
+            let mut state = gate.lock();
+            assert_eq!(state.gate.begin_batch(), GateDecision::Evaluate);
+            assert!(!state.gate.is_paused());
+        }
+
+        let batch = RecordBatch::try_new(
+            Arc::new(Schema::new(vec![Field::new("b", DataType::Int32, false)])),
+            vec![Arc::new(Int32Array::from(vec![3, 4, 5]))],
+        )
+        .unwrap();
+        let result = optional_candidate
+            .physical_expr
+            .evaluate(&batch)
+            .unwrap()
+            .into_array(3)
+            .unwrap();
+        let result = as_boolean_array(&result).unwrap();
+        assert_eq!(result.null_count(), 0);
+        assert_eq!(
+            result.iter().collect::<Vec<_>>(),
+            vec![Some(false), Some(false), Some(true)]
+        );
+    }
+}
diff --git a/datafusion/datasource-parquet/src/source.rs b/datafusion/datasource-parquet/src/source.rs
index 686ca4f1ad5a4..ec853d525b8b3 100644
--- a/datafusion/datasource-parquet/src/source.rs
+++ b/datafusion/datasource-parquet/src/source.rs
@@ -26,6 +26,7 @@ use crate::ParquetFileSchemaProvider;
 use crate::opener::ParquetMorselizer;
 use crate::opener::build_pruning_predicates;
 use crate::opener::build_virtual_columns_state;
+use crate::optional_filter::{DecodeCost, OptionalFilterOptions};
 use crate::row_filter::can_expr_be_pushed_down_with_schemas;
 use arrow_schema::Fields;
 use arrow_schema::extension::ExtensionType;
@@ -33,6 +34,7 @@ use arrow_schema::{DataType, Field};
 use datafusion_common::config::ConfigOptions;
 #[cfg(feature = "parquet_encryption")]
 use datafusion_common::config::EncryptionFactoryOptions;
+use datafusion_common::config::OptionalFilterMode;
 use datafusion_datasource::as_file_source;
 use datafusion_datasource::file_stream::FileOpener;
 use datafusion_datasource::morsel::Morselizer;
@@ -49,6 +51,7 @@ use datafusion_datasource::file::FileSource;
 use datafusion_datasource::file_scan_config::FileScanConfig;
 use datafusion_functions::core::file_row_index::FileRowIndexFunc;
 use datafusion_physical_expr::expressions::{Column, DynamicFilterTracking};
+use datafusion_physical_expr::optional_filter_gate::OptionalFilterGateConfig;
 use datafusion_physical_expr::projection::ProjectionExprs;
 use datafusion_physical_expr::{EquivalenceProperties, conjunction};
 use datafusion_physical_expr_adapter::DefaultPhysicalExprAdapterFactory;
@@ -327,6 +330,19 @@ pub struct ParquetSource {
     /// Sort order driving `PreparedAccessPlan::reorder_by_statistics`
     /// in the opener.
     sort_order_for_reorder: Option<LexOrdering>,
+    /// How the row filter handles optional conjuncts of the predicate (see
+    /// `OptionalFilterPhysicalExpr`) when `pushdown_filters` is true.
+    /// [`FileSource::try_pushdown_filters`] sets it from
+    /// `datafusion.execution.optional_filter_mode`.
+    optional_filter_mode: OptionalFilterMode,
+    /// Configuration of the gates in [`OptionalFilterMode::Adaptive`].
+    /// [`FileSource::try_pushdown_filters`] sets it from
+    /// `datafusion.execution.optional_filter_min_saving_ns_per_row`.
+    optional_filter_gate_config: OptionalFilterGateConfig,
+    /// The measured decode speed of the output columns, shared by all
+    /// partitions and files of this scan. The gates of the optional filters
+    /// use it. Replaced each time the predicate changes.
+    optional_filter_decode_cost: Arc<DecodeCost>,
 }
 
 impl ParquetSource {
@@ -355,6 +371,9 @@ impl ParquetSource {
             encryption_factory: None,
             reverse_row_groups: false,
             sort_order_for_reorder: None,
+            optional_filter_mode: OptionalFilterMode::default(),
+            optional_filter_gate_config: OptionalFilterGateConfig::default(),
+            optional_filter_decode_cost: Arc::default(),
         }
     }
 
@@ -387,6 +406,7 @@ impl ParquetSource {
     pub fn with_predicate(&self, predicate: Arc<dyn PhysicalExpr>) -> Self {
         let mut conf = self.clone();
         conf.predicate = Some(Arc::clone(&predicate));
+        conf.optional_filter_decode_cost = Arc::default();
         conf
     }
 
@@ -685,6 +705,11 @@ impl FileSource for ParquetSource {
             reverse_row_groups: self.reverse_row_groups,
             sort_order_for_reorder: self.sort_order_for_reorder.clone(),
             virtual_state,
+            optional_filters: OptionalFilterOptions {
+                mode: self.optional_filter_mode,
+                gate_config: self.optional_filter_gate_config,
+                decode_cost: Arc::clone(&self.optional_filter_decode_cost),
+            },
         }))
     }
 
@@ -911,6 +936,12 @@ impl FileSource for ParquetSource {
             .with_updated_node(Arc::new(source) as _));
         }
         source = source.with_pushdown_filters(pushdown_filters);
+        // Optional filters (for example dynamic filters from joins) are
+        // handled as the session configuration says. The predicate changed,
+        // thus the decode speed measurement starts again.
+        source.optional_filter_mode = config.execution.optional_filter_mode;
+        source.optional_filter_gate_config = (&config.execution).into();
+        source.optional_filter_decode_cost = Arc::default();
         let source = Arc::new(source);
         // The parquet scan always accepts pushable filters: report each
         // pushable filter as accepted (`Yes`) so the parent `FilterExec` is
@@ -1173,6 +1204,12 @@ impl FileSource for ParquetSource {
                 encryption_factory: _,
             reverse_row_groups,
             sort_order_for_reorder,
+            // Not serialized: the decoded source uses the default
+            // (`always`) optional filter mode.
+            optional_filter_mode: _,
+            optional_filter_gate_config: _,
+            // Runtime state, recreated when the source is decoded.
+            optional_filter_decode_cost: _,
         } = self;
 
         if schema_provider.is_some() {
@@ -2297,4 +2334,53 @@ mod tests {
              pushed down"
         );
     }
+
+    #[test]
+    fn try_pushdown_filters_reads_optional_filter_config() {
+        use arrow::datatypes::{DataType, Field, Schema};
+        use datafusion_expr::{col, lit as logical_lit};
+        use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
+        use datafusion_physical_expr::planner::logical2physical;
+        use datafusion_physical_plan::filter_pushdown::PushedDown;
+
+        let schema = Arc::new(Schema::new(vec![Field::new(
+            "value",
+            DataType::Int64,
+            false,
+        )]));
+        let source = ParquetSource::new(Arc::clone(&schema));
+        assert_eq!(source.optional_filter_mode, OptionalFilterMode::Always);
+
+        let filter = logical2physical(&col("value").eq(logical_lit(1i64)), &schema);
+        let optional: Arc<dyn PhysicalExpr> =
+            Arc::new(OptionalFilterPhysicalExpr::new(filter));
+
+        let mut config = ConfigOptions::default();
+        config.execution.parquet.pushdown_filters = true;
+        config.execution.optional_filter_mode = OptionalFilterMode::Adaptive;
+        config.execution.optional_filter_min_saving_ns_per_row = 7.5;
+        let prop = source
+            .try_pushdown_filters(vec![optional], &config)
+            .expect("try_pushdown_filters must not error");
+        assert!(matches!(prop.filters[0], PushedDown::Yes));
+
+        let updated = prop.updated_node.expect("source should be updated");
+        let updated = (updated.as_ref() as &dyn std::any::Any)
+            .downcast_ref::<ParquetSource>()
+            .expect("ParquetSource");
+        assert_eq!(updated.optional_filter_mode, OptionalFilterMode::Adaptive);
+        assert_eq!(
+            updated.optional_filter_gate_config.min_saving_ns_per_row,
+            7.5
+        );
+        assert_eq!(
+            updated.predicate.as_ref().unwrap().to_string(),
+            "Optional(value@0 = 1)"
+        );
+        // The predicate changed, thus the decode speed measurement is new.
+        assert!(!Arc::ptr_eq(
+            &updated.optional_filter_decode_cost,
+            &source.optional_filter_decode_cost
+        ));
+    }
 }
diff --git a/datafusion/sqllogictest/test_files/optional_filters.slt b/datafusion/sqllogictest/test_files/optional_filters.slt
new file mode 100644
index 0000000000000..d2f5bba4145ac
--- /dev/null
+++ b/datafusion/sqllogictest/test_files/optional_filters.slt
@@ -0,0 +1,135 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Tests for `datafusion.execution.optional_filter_mode` with a Parquet scan
+# and row level filter pushdown.
+#
+# Optional filters are filters that are not needed for correctness, such as
+# the dynamic filters that hash joins and TopK push down into the scan. The
+# queries below must give the same results in each mode.
+#
+# The Rust integration tests in `datafusion/core/tests/parquet/optional_filters.rs`
+# examine the metrics (`optional_filter_rows_skipped` and
+# `optional_filter_pauses`) of each mode, because SQL cannot make an optional
+# filter directly.
+
+statement ok
+set datafusion.execution.parquet.pushdown_filters = true;
+
+statement ok
+COPY (SELECT value AS k, value % 100 AS a FROM generate_series(1, 20000) AS t(value))
+TO 'test_files/scratch/optional_filters/probe.parquet'
+STORED AS PARQUET
+OPTIONS ('format.max_row_group_size' '2000');
+
+statement ok
+COPY (SELECT value AS k, value * 10 AS b FROM generate_series(1, 20000, 7) AS t(value))
+TO 'test_files/scratch/optional_filters/build.parquet'
+STORED AS PARQUET;
+
+statement ok
+CREATE EXTERNAL TABLE probe
+STORED AS PARQUET
+LOCATION 'test_files/scratch/optional_filters/probe.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE build
+STORED AS PARQUET
+LOCATION 'test_files/scratch/optional_filters/build.parquet';
+
+# The mode can be set, and an invalid mode is an error.
+statement error Invalid optional filter mode: sometimes\. Expected one of: always, adaptive, pruning_only
+set datafusion.execution.optional_filter_mode = 'sometimes';
+
+#######
+# always (the default)
+#######
+
+statement ok
+set datafusion.execution.optional_filter_mode = 'always';
+
+query IIII
+SELECT count(*), sum(probe.k), sum(probe.a), sum(build.b)
+FROM probe JOIN build ON probe.k = build.k
+WHERE probe.a < 90;
+----
+2572 25717303 114403 257173030
+
+query II
+SELECT k, a FROM probe WHERE a > 50 ORDER BY k DESC LIMIT 3;
+----
+19999 99
+19998 98
+19997 97
+
+#######
+# adaptive
+#######
+
+statement ok
+set datafusion.execution.optional_filter_mode = 'adaptive';
+
+query IIII
+SELECT count(*), sum(probe.k), sum(probe.a), sum(build.b)
+FROM probe JOIN build ON probe.k = build.k
+WHERE probe.a < 90;
+----
+2572 25717303 114403 257173030
+
+query II
+SELECT k, a FROM probe WHERE a > 50 ORDER BY k DESC LIMIT 3;
+----
+19999 99
+19998 98
+19997 97
+
+#######
+# pruning_only
+#######
+
+statement ok
+set datafusion.execution.optional_filter_mode = 'pruning_only';
+
+query IIII
+SELECT count(*), sum(probe.k), sum(probe.a), sum(build.b)
+FROM probe JOIN build ON probe.k = build.k
+WHERE probe.a < 90;
+----
+2572 25717303 114403 257173030
+
+query II
+SELECT k, a FROM probe WHERE a > 50 ORDER BY k DESC LIMIT 3;
+----
+19999 99
+19998 98
+19997 97
+
+#######
+# Clean up
+#######
+
+statement ok
+reset datafusion.execution.optional_filter_mode;
+
+statement ok
+reset datafusion.execution.parquet.pushdown_filters;
+
+statement ok
+DROP TABLE probe;
+
+statement ok
+DROP TABLE build;
diff --git a/docs/source/user-guide/explain-usage.md b/docs/source/user-guide/explain-usage.md
index b073313c6a0a4..7a916b21fd59d 100644
--- a/docs/source/user-guide/explain-usage.md
+++ b/docs/source/user-guide/explain-usage.md
@@ -236,6 +236,9 @@ When predicate pushdown is enabled, `DataSourceExec` with `ParquetSource` gains
 - `limit_pruned_rows`: number of rows skipped by limit pruning when fully matched page ranges contain enough rows to satisfy the limit.
 - `pushdown_rows_matched`: rows that were tested by any of the above filters, and passed all of them.
 - `pushdown_rows_pruned`: rows that were tested by any of the above filters, and did not pass at least one of them.
+- `optional_filter_rows_skipped`: rows for which the scan skipped an optional filter (a filter that is not needed for correctness, such as a join dynamic filter) because the filter removed no rows or cost more than it saved. Shown only when `datafusion.execution.optional_filter_mode` is `adaptive` and the scan skipped a filter.
+- `optional_filter_pauses`: number of times the scan paused an optional filter. Shown only when `datafusion.execution.optional_filter_mode` is `adaptive` and the scan paused a filter.
+- `optional_filter_eval_time`: time spent to evaluate optional filters (included in `row_pushdown_eval_time`). Shown only when `datafusion.execution.optional_filter_mode` is `adaptive` and the scan evaluated an optional filter.
 - `predicate_evaluation_errors`: number of times evaluating the filter expression failed (expected to be zero in normal operation)
 - `num_predicate_creation_errors`: number of errors creating predicates (expected to be zero in normal operation)
 - `bloom_filter_eval_time`: time spent parsing and evaluating Bloom Filters

From de1b263c744f76f21008e977aed274b7fb7804d1 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 22:02:02 -0500
Subject: [PATCH 16/42] test: end-to-end tests for optional filters in the
 Parquet post-scan path

The Parquet scan evaluates rejected and non-pushed-down required conjuncts after the decode (#22384). Optional conjuncts never go to this post-scan filter. These tests show this behaviour through `ParquetSource::try_pushdown_filters` and all `optional_filter_mode` values:

- `optional_filter_is_not_evaluated_post_scan`: with `pushdown_filters = false`, only the required conjuncts run post-scan. The optional conjuncts are used only for statistics pruning.
- `rejected_optional_filter_is_not_evaluated_post_scan`: with `pushdown_filters = true`, an optional conjunct that the row filter cannot evaluate (a whole-struct `IS NOT NULL`) is not used. The same conjunct as a required conjunct runs post-scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../core/tests/parquet/optional_filters.rs    | 218 +++++++++++++++++-
 1 file changed, 212 insertions(+), 6 deletions(-)

diff --git a/datafusion/core/tests/parquet/optional_filters.rs b/datafusion/core/tests/parquet/optional_filters.rs
index bff9366d4fa1f..8d5e10ae55225 100644
--- a/datafusion/core/tests/parquet/optional_filters.rs
+++ b/datafusion/core/tests/parquet/optional_filters.rs
@@ -15,8 +15,7 @@
 // specific language governing permissions and limitations
 // under the License.
 
-//! End-to-end tests for *optional filters* in the Parquet scan with row
-//! level filter pushdown (`pushdown_filters = true`).
+//! End-to-end tests for *optional filters* in the Parquet scan.
 //!
 //! An optional filter is a predicate conjunct wrapped in an
 //! `OptionalFilterPhysicalExpr`. The scan handles it as
@@ -27,15 +26,19 @@
 //!   or while it costs more than it saves.
 //! * `pruning_only`: only statistics pruning uses it.
 //!
+//! With `pushdown_filters = false`, the scan uses optional filters only for
+//! statistics pruning, in all modes: it never evaluates them for each row.
+//!
 //! No operator makes optional filters yet, thus these tests push an
 //! `Optional(...)` predicate into the `ParquetSource` directly, with
 //! `FileSource::try_pushdown_filters` and the session configuration.
 
 use std::sync::Arc;
 
-use arrow::array::{Array, Int32Array, Int64Array, RecordBatch};
+use arrow::array::{Array, Int32Array, Int64Array, RecordBatch, StructArray};
+use arrow::buffer::NullBuffer;
 use arrow::compute::concat_batches;
-use arrow_schema::{DataType, Field, Schema, SchemaRef};
+use arrow_schema::{DataType, Field, Fields, Schema, SchemaRef};
 use datafusion::datasource::listing::PartitionedFile;
 use datafusion::datasource::object_store::ObjectStoreUrl;
 use datafusion::datasource::physical_plan::ParquetSource;
@@ -50,7 +53,8 @@ use datafusion_datasource::source::DataSourceExec;
 use datafusion_expr::Operator;
 use datafusion_physical_expr::PhysicalExpr;
 use datafusion_physical_expr::expressions::{
-    BinaryExpr, Column, DynamicFilterPhysicalExpr, OptionalFilterPhysicalExpr, lit,
+    BinaryExpr, Column, DynamicFilterPhysicalExpr, IsNotNullExpr,
+    OptionalFilterPhysicalExpr, lit,
 };
 use futures::StreamExt;
 use parquet::arrow::ArrowWriter;
@@ -152,9 +156,30 @@ fn scan_with(
     mode: OptionalFilterMode,
     min_saving_ns_per_row: f64,
     projection: Option<Vec<usize>>,
+) -> Arc<dyn ExecutionPlan> {
+    scan_with_pushdown(
+        file,
+        schema,
+        predicate,
+        mode,
+        min_saving_ns_per_row,
+        projection,
+        true,
+    )
+}
+
+/// Like [`scan_with`], with `pushdown_filters` set to `pushdown`.
+fn scan_with_pushdown(
+    file: &NamedTempFile,
+    schema: &SchemaRef,
+    predicate: Arc<dyn PhysicalExpr>,
+    mode: OptionalFilterMode,
+    min_saving_ns_per_row: f64,
+    projection: Option<Vec<usize>>,
+    pushdown: bool,
 ) -> Arc<dyn ExecutionPlan> {
     let mut options = ConfigOptions::default();
-    options.execution.parquet.pushdown_filters = true;
+    options.execution.parquet.pushdown_filters = pushdown;
     options.execution.optional_filter_mode = mode;
     options.execution.optional_filter_min_saving_ns_per_row = min_saving_ns_per_row;
     let source = ParquetSource::new(Arc::clone(schema))
@@ -385,6 +410,187 @@ async fn optional_dynamic_filter_update_restarts_evaluation() {
     assert_eq!(checked_rows, later_row_groups * ROWS_PER_ROW_GROUP / 10);
 }
 
+/// With `pushdown_filters = false`, the scan evaluates the required
+/// conjuncts after the decode (the post-scan filter), but not the optional
+/// conjuncts, in all modes. The consumer of the scan applies the optional
+/// filter again, thus the results do not change.
+#[tokio::test]
+async fn optional_filter_is_not_evaluated_post_scan() {
+    let (file, schema) = write_file();
+    let total_rows = ROW_GROUPS * ROWS_PER_ROW_GROUP;
+    let optional_filter = a_op(&schema, Operator::Lt, 10);
+    // a < 50 AND Optional(a < 10)
+    let predicate: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
+        a_op(&schema, Operator::Lt, 50),
+        Operator::And,
+        optional(Arc::clone(&optional_filter)),
+    ));
+    for mode in [
+        OptionalFilterMode::Always,
+        OptionalFilterMode::Adaptive,
+        OptionalFilterMode::PruningOnly,
+    ] {
+        let scan = scan_with_pushdown(
+            &file,
+            &schema,
+            Arc::clone(&predicate),
+            mode,
+            1e9,
+            None,
+            false,
+        );
+        let plan: Arc<dyn ExecutionPlan> = Arc::new(
+            FilterExec::try_new(Arc::clone(&optional_filter), Arc::clone(&scan)).unwrap(),
+        );
+        let batches = collect(plan, session().task_ctx()).await.unwrap();
+        assert_eq!(values(&batches).len(), total_rows / 10, "mode {mode}");
+
+        // The post-scan filter has only the required conjunct `a < 50`: it
+        // keeps half of the rows. `Optional(a < 10)` would keep 10%.
+        let matched = metric(scan.as_ref(), "post_scan_rows_matched");
+        let pruned = metric(scan.as_ref(), "post_scan_rows_pruned");
+        assert_eq!(
+            (matched, pruned),
+            (total_rows / 2, total_rows / 2),
+            "mode {mode}"
+        );
+        // No row filter.
+        assert_eq!(metric(scan.as_ref(), "pushdown_rows_matched"), 0);
+        assert_eq!(metric(scan.as_ref(), "optional_filter_rows_skipped"), 0);
+    }
+
+    // With only optional conjuncts, the scan has no post-scan filter.
+    let scan = scan_with_pushdown(
+        &file,
+        &schema,
+        optional(Arc::clone(&optional_filter)),
+        OptionalFilterMode::Always,
+        1e9,
+        None,
+        false,
+    );
+    let batches = collect(Arc::clone(&scan), session().task_ctx())
+        .await
+        .unwrap();
+    assert_eq!(values(&batches).len(), total_rows);
+    assert_eq!(metric(scan.as_ref(), "post_scan_rows_matched"), 0);
+    assert_eq!(metric(scan.as_ref(), "post_scan_rows_pruned"), 0);
+}
+
+/// A file with the columns `a` (`i % 100`) and `s` (a struct that is null
+/// for each even row). The row filter cannot evaluate `s IS NOT NULL`,
+/// because the predicate reads the complete struct.
+fn write_struct_file() -> (NamedTempFile, SchemaRef) {
+    let s_fields = Fields::from(vec![Field::new("x", DataType::Int32, true)]);
+    let schema = Arc::new(Schema::new(vec![
+        Field::new("a", DataType::Int32, false),
+        Field::new("s", DataType::Struct(s_fields.clone()), true),
+    ]));
+    let rows = ROW_GROUPS * ROWS_PER_ROW_GROUP;
+    let a: Int32Array = (0..rows).map(|i| (i % 100) as i32).collect();
+    let s = StructArray::new(
+        s_fields,
+        vec![Arc::new(Int32Array::from(vec![1; rows])) as _],
+        Some(NullBuffer::from_iter((0..rows).map(|i| i % 2 == 1))),
+    );
+    let batch = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(a), Arc::new(s)])
+        .unwrap();
+    let file = NamedTempFile::new().unwrap();
+    let mut writer =
+        ArrowWriter::try_new(file.reopen().unwrap(), Arc::clone(&schema), None).unwrap();
+    writer.write(&batch).unwrap();
+    writer.close().unwrap();
+    (file, schema)
+}
+
+/// A scan of `file` with `pushdown_filters = true`. The table provider sets
+/// `table_predicate` on the source, and the planner pushes `a < 50`. The
+/// planner does not push a predicate that reads a complete struct, but a
+/// table provider can set it.
+fn struct_scan(
+    file: &NamedTempFile,
+    schema: &SchemaRef,
+    table_predicate: Arc<dyn PhysicalExpr>,
+    mode: OptionalFilterMode,
+) -> Arc<dyn ExecutionPlan> {
+    let mut options = ConfigOptions::default();
+    options.execution.parquet.pushdown_filters = true;
+    options.execution.optional_filter_mode = mode;
+    options.execution.optional_filter_min_saving_ns_per_row = 1e9;
+    let source = ParquetSource::new(Arc::clone(schema))
+        .with_predicate(table_predicate)
+        .try_pushdown_filters(vec![a_op(schema, Operator::Lt, 50)], &options)
+        .unwrap()
+        .updated_node
+        .expect("the scan accepts the predicate");
+    let path = file.path().to_str().unwrap().to_string();
+    let size = std::fs::metadata(&path).unwrap().len();
+    let config = FileScanConfigBuilder::new(ObjectStoreUrl::local_filesystem(), source)
+        .with_file(PartitionedFile::new(path, size))
+        .build();
+    DataSourceExec::from_data_source(config)
+}
+
+/// With `pushdown_filters = true`, an optional conjunct that the row filter
+/// cannot evaluate for a file is not used for that file, in all modes: it
+/// does not go to the post-scan filter. A required conjunct that the row
+/// filter cannot evaluate still runs post-scan.
+#[tokio::test]
+async fn rejected_optional_filter_is_not_evaluated_post_scan() {
+    let (file, schema) = write_struct_file();
+    let total_rows = ROW_GROUPS * ROWS_PER_ROW_GROUP;
+    let s_is_not_null: Arc<dyn PhysicalExpr> = Arc::new(IsNotNullExpr::new(Arc::new(
+        Column::new_with_schema("s", &schema).unwrap(),
+    )));
+    let rows = |batches: &[RecordBatch]| -> usize {
+        batches.iter().map(RecordBatch::num_rows).sum()
+    };
+
+    for mode in [
+        OptionalFilterMode::Always,
+        OptionalFilterMode::Adaptive,
+        OptionalFilterMode::PruningOnly,
+    ] {
+        // Optional(s IS NOT NULL) AND a < 50
+        let scan =
+            struct_scan(&file, &schema, optional(Arc::clone(&s_is_not_null)), mode);
+        let batches = collect(Arc::clone(&scan), session().task_ctx())
+            .await
+            .unwrap();
+        // The row filter has only `a < 50`.
+        assert_eq!(rows(&batches), total_rows / 2, "mode {mode}");
+        assert_eq!(
+            metric(scan.as_ref(), "pushdown_rows_matched"),
+            total_rows / 2,
+            "mode {mode}"
+        );
+        // No post-scan filter.
+        assert_eq!(metric(scan.as_ref(), "post_scan_rows_matched"), 0);
+        assert_eq!(metric(scan.as_ref(), "post_scan_rows_pruned"), 0);
+    }
+
+    // Control: the same conjunct as a required conjunct runs post-scan.
+    // s IS NOT NULL AND a < 50
+    let scan = struct_scan(
+        &file,
+        &schema,
+        Arc::clone(&s_is_not_null),
+        OptionalFilterMode::Always,
+    );
+    let batches = collect(Arc::clone(&scan), session().task_ctx())
+        .await
+        .unwrap();
+    assert_eq!(rows(&batches), total_rows / 4);
+    assert_eq!(
+        metric(scan.as_ref(), "post_scan_rows_matched"),
+        total_rows / 4
+    );
+    assert_eq!(
+        metric(scan.as_ref(), "post_scan_rows_pruned"),
+        total_rows / 4
+    );
+}
+
 fn column_i32<'a>(batch: &'a RecordBatch, name: &str) -> &'a [i32] {
     batch
         .column_by_name(name)

From e6162939eb634ceb1785b1dcf68808fd3fd2a80b Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 14:53:49 -0500
Subject: [PATCH 17/42] feat(parquet): adaptive placement of scan filter
 conjuncts

Add `datafusion.execution.adaptive_filter_placement` (default `false`). With `pushdown_filters = true`, the Parquet scan decides for each conjunct where to evaluate it, at file open and again at each row group boundary:

- Required conjunct: `RowFilter` (late materialization) or the post-scan filter from #22384.
- Optional conjunct in the `adaptive` optional filter mode: `RowFilter`, or `Skip` while its gate is paused. A skipped conjunct is not in the `RowFilter`, thus its columns are not decoded. The scan counts down the pause of the gate with the batches of the skipped row groups.

The decision for a required conjunct compares the decode time that a row filter saves with the extra fetch latency of a row filter stage:

    benefit = skippable fraction * unread output bytes per row * decode ns per byte
    cost    = mean fetch latency / rows of the next row group

The skippable fraction counts rows in 64-row windows where no row passes (the decoder only skips long runs of removed rows). The measurements are pooled over all files and partitions of the scan. Before enough rows are measured, a conjunct that reads all output columns starts in the post-scan filter.

When the placement changes, the stream rebuilds the decoder with `ParquetPushDecoder::into_builder` (new `RowFilter` and projection mask) and builds a new `DecoderProjection`. A file with adaptive placement always uses the batch coalescer and the stream-level `LIMIT`. The placement does not change for a file with a live row selection, or when the new post-scan conjuncts change the narrowed batch schema.

The logic is in the new `filter_placement` module: `model` (pure decision), `stats` (pooled measurements) and `FilePlacement` (per-file state).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/common/src/config.rs               |  10 +
 .../src/decoder_projection.rs                 | 115 +++-
 .../src/filter_placement/mod.rs               | 339 +++++++++++
 .../src/filter_placement/model.rs             | 257 +++++++++
 .../src/filter_placement/stats.rs             | 365 ++++++++++++
 datafusion/datasource-parquet/src/mod.rs      |   1 +
 .../datasource-parquet/src/opener/mod.rs      | 542 +++++++++++++++---
 .../datasource-parquet/src/push_decoder.rs    | 156 ++++-
 .../datasource-parquet/src/row_filter.rs      | 217 +++++--
 datafusion/datasource-parquet/src/source.rs   |  22 +
 .../test_files/information_schema.slt         |   2 +
 docs/source/user-guide/configs.md             |   1 +
 12 files changed, 1887 insertions(+), 140 deletions(-)
 create mode 100644 datafusion/datasource-parquet/src/filter_placement/mod.rs
 create mode 100644 datafusion/datasource-parquet/src/filter_placement/model.rs
 create mode 100644 datafusion/datasource-parquet/src/filter_placement/stats.rs

diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index f05f4034f424f..a861c520f1c64 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -1265,6 +1265,16 @@ config_namespace! {
         /// The default is about the cost of a hash table probe for one row. The
         /// best value depends on the hardware.
         pub optional_filter_min_saving_ns_per_row: f64, default = 20.0
+
+        /// When true and `datafusion.execution.parquet.pushdown_filters` is
+        /// true, the Parquet scan decides for each filter conjunct if it is a
+        /// row filter (late materialization) or a filter on the decoded
+        /// batches. The scan measures the rows that each conjunct removes, the
+        /// decode time and the fetch latency, and it changes the placement at
+        /// row group boundaries. When `datafusion.execution.optional_filter_mode`
+        /// is `adaptive`, an optional filter that is paused is also removed
+        /// from the row filter, thus its columns are not decoded.
+        pub adaptive_filter_placement: bool, default = false
     }
 }
 
diff --git a/datafusion/datasource-parquet/src/decoder_projection.rs b/datafusion/datasource-parquet/src/decoder_projection.rs
index ac8499fa3f9c9..a9b9309088883 100644
--- a/datafusion/datasource-parquet/src/decoder_projection.rs
+++ b/datafusion/datasource-parquet/src/decoder_projection.rs
@@ -48,9 +48,10 @@ use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
 use datafusion_physical_plan::metrics::{Count, Time};
 
 use parquet::arrow::ProjectionMask;
-use parquet::schema::types::SchemaDescriptor;
+use parquet::schema::types::{SchemaDescPtr, SchemaDescriptor};
 
 use crate::ParquetFileMetrics;
+use crate::filter_placement::ConjunctStats;
 use crate::opener::{VirtualColumnsState, append_fields};
 use crate::projection_read_plan::build_projection_read_plan;
 
@@ -132,8 +133,10 @@ pub(crate) enum PostScanSelection {
 /// into [`ParquetFileMetrics`] for `EXPLAIN ANALYZE`.
 pub(crate) struct PostScanFilter {
     /// The `AND` conjuncts of the predicate, rebased onto the decoder's stream
-    /// schema, in the order they are evaluated. Never empty.
-    conjuncts: Vec<Arc<dyn PhysicalExpr>>,
+    /// schema, in the order they are evaluated, each with the measurements
+    /// of the adaptive filter placement (if the placement manages it). Never
+    /// empty.
+    conjuncts: Vec<PostScanConjunct>,
     rows_pruned: Count,
     rows_matched: Count,
     eval_time: Time,
@@ -165,7 +168,14 @@ impl PostScanFilter {
         let mut acc: Option<BooleanArray> = None;
 
         let last = self.conjuncts.len() - 1;
-        for (position, conjunct) in self.conjuncts.iter().enumerate() {
+        for (
+            position,
+            PostScanConjunct {
+                expr: conjunct,
+                stats,
+            },
+        ) in self.conjuncts.iter().enumerate()
+        {
             let rows_in = working.num_rows();
             let array = conjunct.evaluate(&working)?.into_array(rows_in)?;
             let Ok(mask) = as_boolean_array(array.as_ref()) else {
@@ -179,6 +189,9 @@ impl PostScanFilter {
                 Some(_) => prep_null_mask_filter(mask),
                 None => mask.clone(),
             };
+            if let Some(stats) = stats {
+                stats.record_evaluation(&mask);
+            }
             // An all-true conjunct leaves the accumulated selection untouched.
             if mask.true_count() == rows_in {
                 continue;
@@ -228,6 +241,62 @@ impl PostScanFilter {
     }
 }
 
+/// A conjunct for the post-scan filter, see [`DecoderProjection::try_new`].
+#[derive(Debug, Clone)]
+pub(crate) struct PostScanConjunct {
+    /// The conjunct, in terms of the physical file schema.
+    pub(crate) expr: Arc<dyn PhysicalExpr>,
+    /// The measurements of the adaptive filter placement, if it manages the
+    /// conjunct. The post-scan filter records each evaluation.
+    pub(crate) stats: Option<Arc<ConjunctStats>>,
+}
+
+impl From<Arc<dyn PhysicalExpr>> for PostScanConjunct {
+    fn from(expr: Arc<dyn PhysicalExpr>) -> Self {
+        Self { expr, stats: None }
+    }
+}
+
+/// The inputs of [`DecoderProjection::try_new`] that do not change for a
+/// file. The opener builds the first projection of a file with it, and the
+/// stream builds a new projection with it when the adaptive filter
+/// placement changes the post-scan conjuncts at a row group boundary.
+#[derive(Clone)]
+pub(crate) struct DecoderProjectionBuilder {
+    pub(crate) projection: ProjectionExprs,
+    /// Conjuncts that are always in the post-scan filter of the file: the
+    /// required conjuncts that the `RowFilter` rejects, or all required
+    /// conjuncts when `pushdown_filters` is false.
+    pub(crate) fixed_post_scan: Vec<PostScanConjunct>,
+    pub(crate) physical_file_schema: SchemaRef,
+    pub(crate) parquet_schema: SchemaDescPtr,
+    pub(crate) output_schema: SchemaRef,
+    pub(crate) virtual_state: Option<Arc<VirtualColumnsState>>,
+    pub(crate) file_metrics: ParquetFileMetrics,
+}
+
+impl DecoderProjectionBuilder {
+    /// The decoder projection with `placed` (the conjuncts that the adaptive
+    /// filter placement puts in the post-scan filter) and
+    /// [`Self::fixed_post_scan`] in the post-scan filter.
+    pub(crate) fn build(&self, placed: &[PostScanConjunct]) -> Result<DecoderProjection> {
+        let post_scan_conjuncts: Vec<PostScanConjunct> = placed
+            .iter()
+            .chain(&self.fixed_post_scan)
+            .cloned()
+            .collect();
+        DecoderProjection::try_new(
+            &self.projection,
+            &post_scan_conjuncts,
+            &self.physical_file_schema,
+            &self.parquet_schema,
+            &self.output_schema,
+            self.virtual_state.as_deref(),
+            &self.file_metrics,
+        )
+    }
+}
+
 /// Per-file decoder projection: the [`ProjectionMask`] installed on the
 /// parquet decoder, plus the per-batch transform that maps the decoder's
 /// output onto the scan's `output_schema`.
@@ -289,7 +358,7 @@ impl DecoderProjection {
     /// behaviour.
     pub(crate) fn try_new(
         projection: &ProjectionExprs,
-        post_scan_conjuncts: &[Arc<dyn PhysicalExpr>],
+        post_scan_conjuncts: &[PostScanConjunct],
         physical_file_schema: &SchemaRef,
         parquet_schema: &SchemaDescriptor,
         output_schema: &SchemaRef,
@@ -320,12 +389,15 @@ impl DecoderProjection {
         // the post-scan predicate, which is rebased onto the stream schema
         // where the reader has appended the virtual columns.
         let post_scan_for_read_plan: Vec<Arc<dyn PhysicalExpr>> = match virtual_state {
-            None => post_scan_conjuncts.to_vec(),
+            None => post_scan_conjuncts
+                .iter()
+                .map(|conjunct| Arc::clone(&conjunct.expr))
+                .collect(),
             Some(state) => post_scan_conjuncts
                 .iter()
-                .map(|expr| {
+                .map(|conjunct| {
                     replace_columns_with_literals(
-                        Arc::clone(expr),
+                        Arc::clone(&conjunct.expr),
                         state.null_replacements(),
                     )
                 })
@@ -392,13 +464,24 @@ impl DecoderProjection {
             // already conjunct-wise, but a single entry may still be a nested
             // `AND` (e.g. a `RowFilter`-rejected conjunct). The compact-once
             // loop can only compact between the pieces it can see.
-            let rebased = post_scan_conjuncts
-                .iter()
-                .map(|expr| reassign_expr_columns(Arc::clone(expr), &stream_schema))
-                .collect::<Result<Vec<_>>>()?
-                .iter()
-                .flat_map(|expr| split_conjunction(expr).into_iter().map(Arc::clone))
-                .collect::<Vec<_>>();
+            let mut rebased = vec![];
+            for conjunct in post_scan_conjuncts {
+                let expr =
+                    reassign_expr_columns(Arc::clone(&conjunct.expr), &stream_schema)?;
+                let pieces = split_conjunction(&expr);
+                // The measurements belong to the whole conjunct. The adaptive
+                // filter placement only manages conjuncts of the root `AND`
+                // chain, which are never split again.
+                let stats = if pieces.len() == 1 {
+                    conjunct.stats.clone()
+                } else {
+                    None
+                };
+                rebased.extend(pieces.into_iter().map(|piece| PostScanConjunct {
+                    expr: Arc::clone(piece),
+                    stats: stats.clone(),
+                }));
+            }
             Some(PostScanFilter {
                 conjuncts: rebased,
                 rows_pruned: file_metrics.post_scan_rows_pruned.clone(),
@@ -514,7 +597,7 @@ mod tests {
 
     fn filter(conjuncts: Vec<Arc<dyn PhysicalExpr>>) -> PostScanFilter {
         PostScanFilter {
-            conjuncts,
+            conjuncts: conjuncts.into_iter().map(PostScanConjunct::from).collect(),
             rows_pruned: Count::new(),
             rows_matched: Count::new(),
             eval_time: Time::new(),
diff --git a/datafusion/datasource-parquet/src/filter_placement/mod.rs b/datafusion/datasource-parquet/src/filter_placement/mod.rs
new file mode 100644
index 0000000000000..8cb023fa6b932
--- /dev/null
+++ b/datafusion/datasource-parquet/src/filter_placement/mod.rs
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Adaptive placement of the filter conjuncts of a Parquet scan
+//! (`datafusion.execution.adaptive_filter_placement`).
+//!
+//! With `pushdown_filters = true`, the scan can evaluate each conjunct of
+//! its predicate in one of these places:
+//!
+//! | Placement | Where | For |
+//! |---|---|---|
+//! | [`Placement::RowFilter`] | A predicate of the Parquet `RowFilter`: the decoder skips the other columns of the rows that it removes | Required and optional conjuncts |
+//! | [`Placement::PostScan`] | The post-scan filter, on the decoded batches | Required conjuncts |
+//! | [`Placement::Skip`] | Not evaluated, and its columns are not decoded | Optional conjuncts in the `adaptive` optional filter mode, while the gate is paused |
+//!
+//! A row filter is not always better. It saves decode time only when it
+//! removes long runs of rows, and the decoder fetches the columns of each
+//! row filter predicate before the other columns (one more round trip for
+//! each row group). See [`model`] for the decision.
+//!
+//! [`FilePlacement`] decides at file open and again at each row group
+//! boundary. The stream then rebuilds the decoder
+//! (`ParquetPushDecoder::into_builder`) with the new `RowFilter` and
+//! projection mask, and rebuilds the post-scan filter. The measurements are
+//! pooled over all files and partitions of the scan ([`PlacementSites`]).
+//!
+//! Conjuncts that `build_row_filter` rejects for a file always run in the
+//! post-scan filter (required) or are not used (optional). Optional
+//! conjuncts in the `always` mode are always row filters.
+
+mod model;
+mod stats;
+
+use std::sync::Arc;
+use std::time::Duration;
+
+use datafusion_physical_expr::utils::is_optional_filter;
+use datafusion_physical_expr::{PhysicalExpr, split_conjunction};
+use datafusion_physical_expr_common::metrics::Count;
+use parquet::arrow::ProjectionMask;
+use parquet::file::metadata::ParquetMetaData;
+
+use crate::decoder_projection::PostScanConjunct;
+use crate::optional_filter::compressed_bytes_per_row;
+use crate::row_filter::{PrebuiltRowFilterCandidate, SharedOptionalFilterGate};
+
+pub(crate) use model::Placement;
+use model::{RequiredConjunctInputs, place_optional, place_required};
+pub(crate) use stats::{ConjunctStats, PlacementSites};
+
+/// The adaptive placement settings of one scan.
+#[derive(Debug, Clone, Default)]
+pub(crate) struct PlacementOptions {
+    /// `datafusion.execution.adaptive_filter_placement`. Only used when
+    /// `pushdown_filters` is true.
+    pub(crate) enabled: bool,
+    /// Pooled measurements, shared by all partitions and files of the scan.
+    pub(crate) sites: Arc<PlacementSites>,
+    /// Number of conjuncts in the root `AND` chain of the scan predicate
+    /// (before the rewrite for each file).
+    pub(crate) scan_conjunct_count: usize,
+}
+
+impl PlacementOptions {
+    pub(crate) fn new(
+        enabled: bool,
+        sites: Arc<PlacementSites>,
+        predicate: Option<&Arc<dyn PhysicalExpr>>,
+    ) -> Self {
+        let scan_conjunct_count =
+            predicate.map_or(0, |predicate| split_conjunction(predicate).len());
+        Self {
+            enabled,
+            sites,
+            scan_conjunct_count,
+        }
+    }
+}
+
+/// A conjunct whose placement [`FilePlacement`] decides.
+#[derive(Debug)]
+struct ManagedConjunct {
+    /// Index of the row filter candidate of the conjunct.
+    candidate: usize,
+    kind: ConjunctKind,
+    placement: Placement,
+}
+
+#[derive(Debug)]
+enum ConjunctKind {
+    Required {
+        /// The conjunct in terms of the file schema, for the post-scan
+        /// filter.
+        expr: Arc<dyn PhysicalExpr>,
+        stats: Arc<ConjunctStats>,
+        /// Compressed bytes for each row of the output columns that the
+        /// conjunct does not read.
+        unread_output_bytes_per_row: f64,
+    },
+    /// An optional conjunct with a gate (the `adaptive` optional filter
+    /// mode).
+    Optional { gate: SharedOptionalFilterGate },
+}
+
+/// The placement of the conjuncts of one file. See the [module
+/// documentation](self).
+#[derive(Debug)]
+pub(crate) struct FilePlacement {
+    conjuncts: Vec<ManagedConjunct>,
+    sites: Arc<PlacementSites>,
+    metadata: Arc<ParquetMetaData>,
+    batch_size: usize,
+    /// Rows of the row group that the current placement is for.
+    row_group_rows: usize,
+    /// Compressed bytes for each row of the columns that the decoder
+    /// produces with the current projection mask.
+    decoded_bytes_per_row: f64,
+    /// When true, the placement does not change any more.
+    frozen: bool,
+    /// Number of row group boundaries where the placement changed.
+    changes: Count,
+}
+
+impl FilePlacement {
+    /// Returns `None` if no conjunct needs a placement decision.
+    ///
+    /// Adds the pooled measurements of each required conjunct to its
+    /// candidate, so that the row filter records its evaluations. Makes the
+    /// placement for the first row group, which has `first_row_group_rows`
+    /// rows.
+    #[expect(clippy::too_many_arguments)]
+    pub(crate) fn try_new(
+        candidates: &mut [PrebuiltRowFilterCandidate],
+        predicate: &Arc<dyn PhysicalExpr>,
+        options: &PlacementOptions,
+        output_projection: &ProjectionMask,
+        metadata: &Arc<ParquetMetaData>,
+        batch_size: usize,
+        first_row_group_rows: usize,
+        changes: Count,
+    ) -> Option<Self> {
+        if !options.enabled {
+            return None;
+        }
+        let file_conjuncts = split_conjunction(predicate);
+        let mut conjuncts = vec![];
+        for (index, candidate) in candidates.iter_mut().enumerate() {
+            let kind = if let Some(gate) = candidate.gate() {
+                ConjunctKind::Optional {
+                    gate: Arc::clone(gate),
+                }
+            } else if is_optional_filter(candidate.source_expr())
+                || candidate.position() >= file_conjuncts.len()
+            {
+                // An optional conjunct in the `always` mode.
+                continue;
+            } else {
+                let stats = options.sites.stats_for(
+                    &file_conjuncts,
+                    candidate.position(),
+                    options.scan_conjunct_count,
+                );
+                candidate.set_placement_stats(Arc::clone(&stats));
+                let unread_output_bytes_per_row =
+                    compressed_bytes_per_row(metadata, |leaf| {
+                        output_projection.leaf_included(leaf)
+                            && !candidate.reads_leaf(leaf)
+                    });
+                ConjunctKind::Required {
+                    expr: Arc::clone(candidate.source_expr()),
+                    stats,
+                    unread_output_bytes_per_row,
+                }
+            };
+            conjuncts.push(ManagedConjunct {
+                candidate: index,
+                kind,
+                placement: Placement::RowFilter,
+            });
+        }
+        if conjuncts.is_empty() {
+            return None;
+        }
+        let mut placement = Self {
+            conjuncts,
+            sites: Arc::clone(&options.sites),
+            metadata: Arc::clone(metadata),
+            batch_size: batch_size.max(1),
+            row_group_rows: first_row_group_rows,
+            decoded_bytes_per_row: 0.0,
+            frozen: false,
+            changes,
+        };
+        placement.place_all(first_row_group_rows, true);
+        Some(placement)
+    }
+
+    /// Makes the placement for the next row group, which has
+    /// `next_row_group_rows` rows. Call at each row group boundary. Returns
+    /// true if the placement changed: then rebuild the `RowFilter` and the
+    /// decoder projection.
+    pub(crate) fn decide(&mut self, next_row_group_rows: usize) -> bool {
+        if self.frozen {
+            return false;
+        }
+        // A skipped optional conjunct did not see the batches of the last
+        // row group. Count them down on its gate, so that the pause can
+        // end.
+        let batches = self.row_group_rows.div_ceil(self.batch_size);
+        for conjunct in &self.conjuncts {
+            if let (Placement::Skip, ConjunctKind::Optional { gate }) =
+                (conjunct.placement, &conjunct.kind)
+            {
+                gate.lock().skip_batches(batches, self.batch_size);
+            }
+        }
+        self.row_group_rows = next_row_group_rows;
+        let changed = self.place_all(next_row_group_rows, false);
+        if changed {
+            self.changes.add(1);
+        }
+        changed
+    }
+
+    /// Decides the placement of each conjunct. Returns true if a placement
+    /// changed.
+    fn place_all(&mut self, row_group_rows: usize, first: bool) -> bool {
+        let decode_ns_per_byte = self.sites.decode().ns_per_byte();
+        let fetch_ns_per_row =
+            self.sites.fetch().mean_nanos() / row_group_rows.max(1) as f64;
+        let mut changed = false;
+        for conjunct in &mut self.conjuncts {
+            let placement = match &conjunct.kind {
+                ConjunctKind::Required {
+                    stats,
+                    unread_output_bytes_per_row,
+                    ..
+                } => place_required(
+                    &RequiredConjunctInputs {
+                        observation: stats.observation(),
+                        unread_output_bytes_per_row: *unread_output_bytes_per_row,
+                        decode_ns_per_byte,
+                        fetch_ns_per_row,
+                    },
+                    (!first).then_some(conjunct.placement),
+                ),
+                ConjunctKind::Optional { gate } => {
+                    place_optional(gate.lock().is_paused())
+                }
+            };
+            changed |= placement != conjunct.placement;
+            conjunct.placement = placement;
+        }
+        changed
+    }
+
+    /// Number of rows of the row group at `index` in the file.
+    pub(crate) fn row_group_rows(&self, index: usize) -> usize {
+        usize::try_from(self.metadata.row_group(index).num_rows()).unwrap_or(0)
+    }
+
+    /// The current placement of each managed conjunct.
+    pub(crate) fn placements(&self) -> Vec<Placement> {
+        self.conjuncts.iter().map(|c| c.placement).collect()
+    }
+
+    /// Sets the placement of each managed conjunct to `placements` (from
+    /// [`Self::placements`]) and stops all later changes. For a file where
+    /// the scan cannot apply a change.
+    pub(crate) fn restore_and_freeze(&mut self, placements: &[Placement]) {
+        for (conjunct, placement) in self.conjuncts.iter_mut().zip(placements) {
+            conjunct.placement = *placement;
+        }
+        self.frozen = true;
+    }
+
+    /// Stops all later changes.
+    pub(crate) fn freeze(&mut self) {
+        self.frozen = true;
+    }
+
+    /// True if the candidate at `index` is a predicate of the `RowFilter`.
+    pub(crate) fn in_row_filter(&self, index: usize) -> bool {
+        self.conjuncts
+            .iter()
+            .find(|c| c.candidate == index)
+            .is_none_or(|c| c.placement == Placement::RowFilter)
+    }
+
+    /// The managed conjuncts that the post-scan filter evaluates.
+    pub(crate) fn post_scan_conjuncts(&self) -> Vec<PostScanConjunct> {
+        self.conjuncts
+            .iter()
+            .filter(|c| c.placement == Placement::PostScan)
+            .filter_map(|c| match &c.kind {
+                ConjunctKind::Required { expr, stats, .. } => Some(PostScanConjunct {
+                    expr: Arc::clone(expr),
+                    stats: Some(Arc::clone(stats)),
+                }),
+                ConjunctKind::Optional { .. } => None,
+            })
+            .collect()
+    }
+
+    /// Sets the projection mask of the decoder, for the decode time
+    /// measurement.
+    pub(crate) fn set_decoder_mask(&mut self, mask: &ProjectionMask) {
+        self.decoded_bytes_per_row =
+            compressed_bytes_per_row(&self.metadata, |leaf| mask.leaf_included(leaf));
+    }
+
+    /// Records that the decoder produced a batch of `rows` rows in
+    /// `elapsed`.
+    pub(crate) fn record_decode(&self, rows: usize, elapsed: Duration) {
+        let bytes = rows as f64 * self.decoded_bytes_per_row;
+        if bytes >= 1.0 {
+            self.sites.decode().record(elapsed, bytes as u64);
+        }
+    }
+
+    /// Records one fetch of the scan that took `elapsed`.
+    pub(crate) fn record_fetch(&self, elapsed: Duration) {
+        self.sites.fetch().record(elapsed);
+    }
+}
diff --git a/datafusion/datasource-parquet/src/filter_placement/model.rs b/datafusion/datasource-parquet/src/filter_placement/model.rs
new file mode 100644
index 0000000000000..25c23d438f0cb
--- /dev/null
+++ b/datafusion/datasource-parquet/src/filter_placement/model.rs
@@ -0,0 +1,257 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The placement decision for one conjunct. Pure functions, see the
+//! [module documentation](super) for the model.
+
+use super::stats::Observation;
+
+/// Where the scan evaluates one conjunct of its predicate.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum Placement {
+    /// A predicate of the Parquet `RowFilter` (late materialization).
+    RowFilter,
+    /// A conjunct of the post-scan filter, on the decoded batches.
+    PostScan,
+    /// Not evaluated. Only for optional conjuncts.
+    Skip,
+}
+
+/// Minimum number of evaluated rows before the measurements replace the
+/// initial rule.
+pub(crate) const MIN_OBSERVED_ROWS: u64 = 8192;
+
+/// A conjunct that the statistics pruned in at least this fraction of the
+/// row groups starts in the post-scan filter. See [`place_required`].
+pub(crate) const CLUSTERED_PRUNED_FRACTION: f64 = 0.5;
+
+/// A row filter conjunct moves to the post-scan filter only if its cost is
+/// larger than this multiple of its benefit. The same margin as the
+/// optional filter gate.
+const LEAVE_ROW_FILTER_MARGIN: f64 = 1.1;
+
+/// A post-scan conjunct moves to the row filter only if its cost is smaller
+/// than this multiple of its benefit.
+const ENTER_ROW_FILTER_MARGIN: f64 = 0.9;
+
+/// What the decision for a required conjunct uses.
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub(crate) struct RequiredConjunctInputs {
+    /// The pooled measurements of the conjunct.
+    pub(crate) observation: Observation,
+    /// Compressed bytes for each row of the output columns that the
+    /// conjunct does not read. A row filter saves their decode for the rows
+    /// that it removes.
+    pub(crate) unread_output_bytes_per_row: f64,
+    /// Measured decode time for each compressed byte.
+    pub(crate) decode_ns_per_byte: f64,
+    /// The extra fetch latency of a row filter stage, for each row of the
+    /// next row group: the decoder fetches the columns of each row filter
+    /// predicate before the other columns.
+    pub(crate) fetch_ns_per_row: f64,
+}
+
+impl RequiredConjunctInputs {
+    /// Decode time, in nanoseconds for each row, that a row filter saves.
+    /// `None` without enough measurements.
+    pub(crate) fn benefit_ns_per_row(&self) -> Option<f64> {
+        if self.observation.rows_in < MIN_OBSERVED_ROWS {
+            return None;
+        }
+        let skippable = self.observation.skippable_fraction()?;
+        Some(skippable * self.unread_output_bytes_per_row * self.decode_ns_per_byte)
+    }
+}
+
+/// The placement of a required conjunct: [`Placement::RowFilter`] or
+/// [`Placement::PostScan`].
+///
+/// With enough measurements, the conjunct is a row filter if the decode
+/// time that it saves is larger than the extra fetch latency:
+///
+/// ```text
+/// benefit = skippable fraction * unread output bytes * decode ns for each byte
+/// cost    = fetch latency / rows of the next row group
+/// ```
+///
+/// The time to evaluate the conjunct is the same in both placements, thus
+/// it is not in the model. `current` is the placement for the last row
+/// group, `None` at the first decision. A change needs a 10% margin, so that
+/// a conjunct does not move at each row group when the benefit and the cost
+/// are almost equal.
+///
+/// Without enough measurements (the initial rule), the conjunct is a post-scan
+/// filter if the row group statistics pruned it in at least
+/// [`CLUSTERED_PRUNED_FRACTION`] of the row groups (the data is clustered on
+/// its columns, thus the row groups that are left pass most rows), or if
+/// the conjunct reads all output columns (a row filter cannot save decode
+/// time). Otherwise it is a row filter, as without adaptive placement.
+pub(crate) fn place_required(
+    inputs: &RequiredConjunctInputs,
+    current: Option<Placement>,
+) -> Placement {
+    let Some(benefit) = inputs.benefit_ns_per_row() else {
+        if let Some(current) = current {
+            return current;
+        }
+        let clustered = inputs
+            .observation
+            .pruned_fraction()
+            .is_some_and(|pruned| pruned >= CLUSTERED_PRUNED_FRACTION);
+        return if clustered || inputs.unread_output_bytes_per_row <= 0.0 {
+            Placement::PostScan
+        } else {
+            Placement::RowFilter
+        };
+    };
+    let cost = inputs.fetch_ns_per_row;
+    let row_filter = match current {
+        Some(Placement::RowFilter) => cost <= benefit * LEAVE_ROW_FILTER_MARGIN,
+        _ => cost < benefit * ENTER_ROW_FILTER_MARGIN,
+    };
+    // A conjunct that saves nothing is never a row filter.
+    if row_filter && benefit > 0.0 {
+        Placement::RowFilter
+    } else {
+        Placement::PostScan
+    }
+}
+
+/// The placement of an optional conjunct in the `adaptive` optional filter
+/// mode: [`Placement::Skip`] while its gate is paused, otherwise
+/// [`Placement::RowFilter`].
+pub(crate) fn place_optional(gate_paused: bool) -> Placement {
+    if gate_paused {
+        Placement::Skip
+    } else {
+        Placement::RowFilter
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn inputs(
+        rows_in: u64,
+        skippable_rows: u64,
+        pruned: u64,
+        kept: u64,
+        unread_output_bytes_per_row: f64,
+        fetch_ns_per_row: f64,
+    ) -> RequiredConjunctInputs {
+        RequiredConjunctInputs {
+            observation: Observation {
+                rows_in,
+                rows_out: rows_in - skippable_rows,
+                skippable_rows,
+                row_groups_pruned: pruned,
+                row_groups_kept: kept,
+            },
+            unread_output_bytes_per_row,
+            decode_ns_per_byte: 4.0,
+            fetch_ns_per_row,
+        }
+    }
+
+    #[test]
+    fn initial_rule() {
+        // Output columns that the conjunct does not read: row filter.
+        let wide = inputs(0, 0, 0, 0, 10.0, 0.0);
+        assert_eq!(place_required(&wide, None), Placement::RowFilter);
+        // The conjunct reads all output columns: post-scan.
+        let narrow = inputs(0, 0, 0, 0, 0.0, 0.0);
+        assert_eq!(place_required(&narrow, None), Placement::PostScan);
+        // Statistics pruned most row groups: post-scan.
+        let clustered = inputs(0, 0, 6, 4, 10.0, 0.0);
+        assert_eq!(place_required(&clustered, None), Placement::PostScan);
+        let not_clustered = inputs(0, 0, 4, 6, 10.0, 0.0);
+        assert_eq!(place_required(&not_clustered, None), Placement::RowFilter);
+        // Too few measured rows: the initial rule, and no change later.
+        let few = inputs(MIN_OBSERVED_ROWS - 1, 0, 0, 0, 10.0, 0.0);
+        assert_eq!(place_required(&few, None), Placement::RowFilter);
+        assert_eq!(
+            place_required(&few, Some(Placement::PostScan)),
+            Placement::PostScan
+        );
+    }
+
+    #[test]
+    fn measured_benefit() {
+        let rows = 100_000;
+        // Half of the rows are in empty windows: a row filter saves decode
+        // time, and the fetch latency is small.
+        let clustered = inputs(rows, rows / 2, 0, 0, 10.0, 1.0);
+        assert_eq!(place_required(&clustered, None), Placement::RowFilter);
+        assert_eq!(
+            place_required(&clustered, Some(Placement::PostScan)),
+            Placement::RowFilter
+        );
+        // No empty windows (for example a uniform 50% filter): nothing to
+        // save, thus post-scan. Also without any fetch latency.
+        let scattered = inputs(rows, 0, 0, 0, 10.0, 0.0);
+        assert_eq!(
+            place_required(&scattered, Some(Placement::RowFilter)),
+            Placement::PostScan
+        );
+        // The measurements replace the statistics prior.
+        let clustered_by_stats = inputs(rows, rows / 2, 9, 1, 10.0, 0.0);
+        assert_eq!(
+            place_required(&clustered_by_stats, None),
+            Placement::RowFilter
+        );
+        // A high fetch latency (a remote object store) is more than the
+        // saving: benefit = 0.5 * 10 * 4 = 20 ns, cost = 30 ns.
+        let remote = inputs(rows, rows / 2, 0, 0, 10.0, 30.0);
+        assert_eq!(
+            place_required(&remote, Some(Placement::RowFilter)),
+            Placement::PostScan
+        );
+    }
+
+    #[test]
+    fn margin_prevents_flapping() {
+        let rows = 100_000;
+        // benefit = 0.5 * 10 * 4 = 20 ns; cost = 21 ns is inside the margin.
+        let close = inputs(rows, rows / 2, 0, 0, 10.0, 21.0);
+        assert_eq!(
+            place_required(&close, Some(Placement::RowFilter)),
+            Placement::RowFilter
+        );
+        assert_eq!(
+            place_required(&close, Some(Placement::PostScan)),
+            Placement::PostScan
+        );
+        // Outside the margin in both directions.
+        let expensive = inputs(rows, rows / 2, 0, 0, 10.0, 23.0);
+        assert_eq!(
+            place_required(&expensive, Some(Placement::RowFilter)),
+            Placement::PostScan
+        );
+        let cheap = inputs(rows, rows / 2, 0, 0, 10.0, 17.0);
+        assert_eq!(
+            place_required(&cheap, Some(Placement::PostScan)),
+            Placement::RowFilter
+        );
+    }
+
+    #[test]
+    fn optional_follows_the_gate() {
+        assert_eq!(place_optional(true), Placement::Skip);
+        assert_eq!(place_optional(false), Placement::RowFilter);
+    }
+}
diff --git a/datafusion/datasource-parquet/src/filter_placement/stats.rs b/datafusion/datasource-parquet/src/filter_placement/stats.rs
new file mode 100644
index 0000000000000..73466b8d52593
--- /dev/null
+++ b/datafusion/datasource-parquet/src/filter_placement/stats.rs
@@ -0,0 +1,365 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Measurements that the placement decisions use. All partitions and files
+//! of one scan share them.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::Duration;
+
+use arrow::array::{Array, BooleanArray};
+use datafusion_physical_expr::PhysicalExpr;
+use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
+use datafusion_physical_expr::filter_stats::duration_nanos;
+use datafusion_physical_expr::utils::is_optional_filter;
+use parking_lot::Mutex;
+
+use crate::optional_filter::DEFAULT_DECODE_NS_PER_BYTE;
+
+/// Number of rows in one window of [`skippable_rows`].
+///
+/// The Parquet decoder skips the rows that a row filter removes only when
+/// the removed rows make long runs. Its default selection policy
+/// (`RowSelectionPolicy::Auto { threshold: 32 }`) decodes all rows and then
+/// filters them when the runs are shorter than 32 rows on average. A window
+/// of 64 rows is one `u64` of the filter result.
+pub(crate) const SKIP_WINDOW_ROWS: usize = 64;
+
+/// Rows of `result` in windows of [`SKIP_WINDOW_ROWS`] rows where no row
+/// passes (a `null` does not pass). These are the rows that a row filter
+/// lets the decoder skip for the other columns. A partial window at the end
+/// is counted if no row in it passes.
+pub(crate) fn skippable_rows(result: &BooleanArray) -> usize {
+    let values = result.values();
+    let passed = match result.nulls() {
+        Some(nulls) => values & nulls.inner(),
+        None => values.clone(),
+    };
+    let chunks = passed.bit_chunks();
+    let full = chunks.iter().filter(|chunk| *chunk == 0).count() * SKIP_WINDOW_ROWS;
+    let remainder = if chunks.remainder_len() > 0 && chunks.remainder_bits() == 0 {
+        chunks.remainder_len()
+    } else {
+        0
+    };
+    full + remainder
+}
+
+/// What the scan measured for one conjunct: in the row filter or in the
+/// post-scan filter, and in the row group statistics pruning.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub(crate) struct Observation {
+    /// Rows that the conjunct was evaluated on.
+    pub(crate) rows_in: u64,
+    /// Rows that passed the conjunct.
+    pub(crate) rows_out: u64,
+    /// Rows in windows where no row passed, see [`skippable_rows`].
+    pub(crate) skippable_rows: u64,
+    /// Row groups that the conjunct alone pruned with statistics.
+    pub(crate) row_groups_pruned: u64,
+    /// Row groups that the conjunct alone did not prune with statistics.
+    pub(crate) row_groups_kept: u64,
+}
+
+impl Observation {
+    /// Fraction of the evaluated rows that the decoder can skip, or `None`
+    /// if the conjunct was not evaluated on any row.
+    pub(crate) fn skippable_fraction(&self) -> Option<f64> {
+        (self.rows_in > 0).then(|| self.skippable_rows as f64 / self.rows_in as f64)
+    }
+
+    /// Fraction of the row groups that the conjunct pruned with statistics,
+    /// or `None` if there is no statistics pruning result.
+    pub(crate) fn pruned_fraction(&self) -> Option<f64> {
+        let total = self.row_groups_pruned + self.row_groups_kept;
+        (total > 0).then(|| self.row_groups_pruned as f64 / total as f64)
+    }
+}
+
+/// The pooled [`Observation`] of one conjunct. Lock-free.
+#[derive(Debug, Default)]
+pub(crate) struct ConjunctStats {
+    rows_in: AtomicU64,
+    rows_out: AtomicU64,
+    skippable_rows: AtomicU64,
+    row_groups_pruned: AtomicU64,
+    row_groups_kept: AtomicU64,
+}
+
+impl ConjunctStats {
+    /// Records one evaluation of the conjunct. `result` has one value for
+    /// each evaluated row.
+    pub(crate) fn record_evaluation(&self, result: &BooleanArray) {
+        let rows_in = result.len() as u64;
+        if rows_in == 0 {
+            return;
+        }
+        // `true_count` does not count nulls.
+        let rows_out = result.true_count() as u64;
+        let skippable = if rows_out == 0 {
+            rows_in
+        } else if rows_out == rows_in {
+            0
+        } else {
+            skippable_rows(result) as u64
+        };
+        self.rows_in.fetch_add(rows_in, Ordering::Relaxed);
+        self.rows_out.fetch_add(rows_out, Ordering::Relaxed);
+        self.skippable_rows.fetch_add(skippable, Ordering::Relaxed);
+    }
+
+    /// The pooled values. The values are not read at the same instant.
+    pub(crate) fn observation(&self) -> Observation {
+        Observation {
+            rows_in: self.rows_in.load(Ordering::Relaxed),
+            rows_out: self.rows_out.load(Ordering::Relaxed),
+            skippable_rows: self.skippable_rows.load(Ordering::Relaxed),
+            row_groups_pruned: self.row_groups_pruned.load(Ordering::Relaxed),
+            row_groups_kept: self.row_groups_kept.load(Ordering::Relaxed),
+        }
+    }
+}
+
+/// The measured decode speed of the columns that the decoder produces, for
+/// one scan. Lock-free.
+#[derive(Debug, Default)]
+pub(crate) struct DecodeSpeed {
+    nanos: AtomicU64,
+    bytes: AtomicU64,
+}
+
+impl DecodeSpeed {
+    /// Records that the decoder produced `bytes` compressed bytes (estimated
+    /// from the footer) in `elapsed`.
+    pub(crate) fn record(&self, elapsed: Duration, bytes: u64) {
+        self.nanos
+            .fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
+        self.bytes.fetch_add(bytes, Ordering::Relaxed);
+    }
+
+    /// Decode time for each compressed byte. Before the first measurement,
+    /// the same default as the optional filter gates.
+    pub(crate) fn ns_per_byte(&self) -> f64 {
+        let bytes = self.bytes.load(Ordering::Relaxed);
+        if bytes == 0 {
+            return DEFAULT_DECODE_NS_PER_BYTE;
+        }
+        self.nanos.load(Ordering::Relaxed) as f64 / bytes as f64
+    }
+}
+
+/// The mean latency of the fetches (`AsyncFileReader::get_byte_ranges`) of
+/// one scan. Lock-free.
+#[derive(Debug, Default)]
+pub(crate) struct FetchCost {
+    nanos: AtomicU64,
+    fetches: AtomicU64,
+}
+
+impl FetchCost {
+    /// Records one fetch that took `elapsed`.
+    pub(crate) fn record(&self, elapsed: Duration) {
+        self.nanos
+            .fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
+        self.fetches.fetch_add(1, Ordering::Relaxed);
+    }
+
+    /// Mean latency of one fetch in nanoseconds, 0 before the first fetch.
+    pub(crate) fn mean_nanos(&self) -> f64 {
+        let fetches = self.fetches.load(Ordering::Relaxed);
+        if fetches == 0 {
+            return 0.0;
+        }
+        self.nanos.load(Ordering::Relaxed) as f64 / fetches as f64
+    }
+}
+
+/// Identifies one conjunct of the scan predicate in the rewritten predicate
+/// of each file.
+///
+/// The scan rewrites its predicate for each file (schema adaptation,
+/// partition values, simplification):
+///
+/// * A conjunct with a [`PhysicalExpr::expression_id`] (for example a
+///   `DynamicFilterPhysicalExpr`) uses that id. `with_new_children` keeps
+///   the id, thus all rewrites keep it.
+/// * Other conjuncts use their position in the root `AND` chain. A rewrite
+///   does not reorder the conjuncts, but the simplifier can fold a conjunct
+///   to a literal. Thus the position is used only when the file predicate
+///   has as many conjuncts as the scan predicate.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+enum ConjunctKey {
+    ExpressionId(u64),
+    Position(usize),
+}
+
+/// The pooled measurements of one scan, shared by all partitions and files
+/// of that scan. `ParquetSource` makes a new one each time its predicate
+/// changes.
+#[derive(Debug, Default)]
+pub(crate) struct PlacementSites {
+    conjuncts: Mutex<HashMap<ConjunctKey, Arc<ConjunctStats>>>,
+    fetch: FetchCost,
+    decode: DecodeSpeed,
+}
+
+impl PlacementSites {
+    /// The fetch latency of the scan.
+    pub(crate) fn fetch(&self) -> &FetchCost {
+        &self.fetch
+    }
+
+    /// The decode speed of the scan.
+    pub(crate) fn decode(&self) -> &DecodeSpeed {
+        &self.decode
+    }
+
+    /// The statistics of the conjunct at `position` in `conjuncts` (the
+    /// conjuncts of the root `AND` chain of a file predicate). The scan
+    /// predicate has `scan_count` conjuncts. See [`ConjunctSiteKey`].
+    pub(crate) fn stats_for(
+        &self,
+        conjuncts: &[&Arc<dyn PhysicalExpr>],
+        position: usize,
+        scan_count: usize,
+    ) -> Arc<ConjunctStats> {
+        let conjunct = conjuncts[position];
+        // An optional filter keeps the id of the filter that it wraps.
+        let expr = conjunct
+            .downcast_ref::<OptionalFilterPhysicalExpr>()
+            .map_or(conjunct, |optional| optional.inner());
+        debug_assert!(!is_optional_filter(expr));
+        let key = match expr.expression_id() {
+            Some(id) => Some(ConjunctKey::ExpressionId(id)),
+            None => {
+                (conjuncts.len() == scan_count).then_some(ConjunctKey::Position(position))
+            }
+        };
+        match key {
+            Some(key) => Arc::clone(self.conjuncts.lock().entry(key).or_default()),
+            // The conjunct cannot be identified: do not share its statistics.
+            None => Arc::new(ConjunctStats::default()),
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn bools(values: impl IntoIterator<Item = Option<bool>>) -> BooleanArray {
+        values.into_iter().collect()
+    }
+
+    #[test]
+    fn skippable_rows_counts_empty_windows() {
+        // Window 0 (rows 0..64) has one passing row, window 1 has none, the
+        // partial window (rows 128..160) has none.
+        let mut values = vec![Some(false); 160];
+        values[10] = Some(true);
+        assert_eq!(skippable_rows(&bools(values.clone())), 64 + 32);
+
+        // A null does not pass.
+        values[70] = None;
+        assert_eq!(skippable_rows(&bools(values.clone())), 64 + 32);
+
+        // One passing row in the partial window.
+        values[150] = Some(true);
+        assert_eq!(skippable_rows(&bools(values)), 64);
+
+        // Scattered passing rows: no window is empty.
+        let scattered = (0..640).map(|i| Some(i % 50 == 0));
+        assert_eq!(skippable_rows(&bools(scattered)), 0);
+    }
+
+    #[test]
+    fn conjunct_stats_accumulate() {
+        let stats = ConjunctStats::default();
+        let mut values = vec![Some(false); 128];
+        values[0] = Some(true);
+        stats.record_evaluation(&bools(values));
+        stats.record_evaluation(&bools(vec![Some(true); 64]));
+        stats.record_evaluation(&bools(vec![None; 64]));
+        let observation = stats.observation();
+        assert_eq!(
+            observation,
+            Observation {
+                rows_in: 256,
+                rows_out: 65,
+                skippable_rows: 128,
+                row_groups_pruned: 0,
+                row_groups_kept: 0,
+            }
+        );
+        assert_eq!(observation.skippable_fraction(), Some(0.5));
+        assert_eq!(Observation::default().skippable_fraction(), None);
+        assert_eq!(Observation::default().pruned_fraction(), None);
+    }
+
+    #[test]
+    fn decode_speed() {
+        let decode = DecodeSpeed::default();
+        assert_eq!(decode.ns_per_byte(), DEFAULT_DECODE_NS_PER_BYTE);
+        decode.record(Duration::from_nanos(300), 100);
+        decode.record(Duration::from_nanos(100), 100);
+        assert_eq!(decode.ns_per_byte(), 2.0);
+    }
+
+    #[test]
+    fn stats_are_shared_by_position_or_id() {
+        use arrow::datatypes::{DataType, Field, Schema};
+        use datafusion_physical_expr::expressions::{
+            DynamicFilterPhysicalExpr, col, lit,
+        };
+
+        let schema = Schema::new(vec![Field::new("a", DataType::Boolean, true)]);
+        let a = col("a", &schema).unwrap();
+        let dynamic: Arc<dyn PhysicalExpr> = Arc::new(DynamicFilterPhysicalExpr::new(
+            vec![Arc::clone(&a)],
+            lit(true),
+        ));
+        let sites = PlacementSites::default();
+        let file = [&a, &dynamic];
+        // Same position and count: shared.
+        assert!(Arc::ptr_eq(
+            &sites.stats_for(&file, 0, 2),
+            &sites.stats_for(&file, 0, 2)
+        ));
+        // Another count: not shared.
+        assert!(!Arc::ptr_eq(
+            &sites.stats_for(&file, 0, 2),
+            &sites.stats_for(&file, 0, 3)
+        ));
+        // An expression id: shared at all positions and counts.
+        let other_file = [&dynamic];
+        assert!(Arc::ptr_eq(
+            &sites.stats_for(&file, 1, 5),
+            &sites.stats_for(&other_file, 0, 7)
+        ));
+    }
+
+    #[test]
+    fn fetch_cost_mean() {
+        let fetch = FetchCost::default();
+        assert_eq!(fetch.mean_nanos(), 0.0);
+        fetch.record(Duration::from_micros(1));
+        fetch.record(Duration::from_micros(3));
+        assert_eq!(fetch.mean_nanos(), 2_000.0);
+    }
+}
diff --git a/datafusion/datasource-parquet/src/mod.rs b/datafusion/datasource-parquet/src/mod.rs
index 490af617ad742..c5adb1eb349c6 100644
--- a/datafusion/datasource-parquet/src/mod.rs
+++ b/datafusion/datasource-parquet/src/mod.rs
@@ -28,6 +28,7 @@ pub mod access_plan;
 mod bloom_filter;
 mod decoder_projection;
 pub mod file_format;
+mod filter_placement;
 pub mod metadata;
 mod metrics;
 mod nested_schema_pruning;
diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index 7abbb4cf82244..3d8ffc5fb60db 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -24,7 +24,10 @@ use self::early_stop::EarlyStoppingStream;
 #[cfg(feature = "parquet_encryption")]
 use self::encryption::EncryptionContext;
 use crate::access_plan::PreparedAccessPlan;
-use crate::decoder_projection::DecoderProjection;
+use crate::decoder_projection::{
+    DecoderProjection, DecoderProjectionBuilder, PostScanConjunct,
+};
+use crate::filter_placement::{FilePlacement, PlacementOptions};
 use crate::metrics::{ByteProgress, RowFilterSkippedFullyMatchedMetric};
 use crate::optional_filter::OptionalFilterOptions;
 use crate::page_filter::PagePruningAccessPlanFilter;
@@ -32,7 +35,7 @@ use crate::push_decoder::{
     DecoderBuilderConfig, InitialDecoderState, PushDecoderStreamState, RgPlanEntry,
     RowFilterContext, RowGroupPruner,
 };
-use crate::row_filter::OptionalFilterRowFilterContext;
+use crate::row_filter::{OptionalFilterRowFilterContext, PrebuiltRowFilterCandidate};
 use crate::row_group_filter::{RowGroupAccessPlanFilter, row_group_in_range};
 use crate::{
     BloomFilterStatistics, Int96Coercer, ParquetAccessPlan, ParquetFileMetrics,
@@ -79,6 +82,7 @@ use datafusion_execution::parquet_encryption::EncryptionFactory;
 use futures::{FutureExt, StreamExt, future::BoxFuture, stream::BoxStream};
 use log::debug;
 use parquet::arrow::ParquetRecordBatchStreamBuilder;
+use parquet::arrow::ProjectionMask;
 use parquet::arrow::arrow_reader::metrics::ArrowReaderMetrics;
 use parquet::arrow::arrow_reader::{
     ArrowReaderMetadata, ArrowReaderOptions, MaskRunIter,
@@ -87,7 +91,9 @@ use parquet::arrow::async_reader::AsyncFileReader;
 use parquet::arrow::parquet_column;
 use parquet::basic::Type;
 use parquet::bloom_filter::Sbbf;
-use parquet::file::metadata::{PageIndexPolicy, ParquetMetaDataReader, RowGroupMetaData};
+use parquet::file::metadata::{
+    PageIndexPolicy, ParquetMetaData, ParquetMetaDataReader, RowGroupMetaData,
+};
 
 /// Morselizer-level state for virtual columns, precomputed once per scan
 /// partition so each file skips the validator walks, `null_replacements`
@@ -314,6 +320,9 @@ pub(super) struct ParquetMorselizer {
     /// How the row filter handles optional conjuncts of the predicate, when
     /// `pushdown_filters` is true.
     pub(crate) optional_filters: OptionalFilterOptions,
+    /// The adaptive filter placement settings, used when `pushdown_filters`
+    /// is true.
+    pub(crate) filter_placement: PlacementOptions,
 }
 
 impl fmt::Debug for ParquetMorselizer {
@@ -507,6 +516,7 @@ struct PreparedParquetOpen {
     sort_order_for_reorder: Option<LexOrdering>,
     preserve_order: bool,
     optional_filters: OptionalFilterOptions,
+    filter_placement: PlacementOptions,
     #[cfg(feature = "parquet_encryption")]
     file_decryption_properties: Option<Arc<FileDecryptionProperties>>,
 }
@@ -544,6 +554,9 @@ struct RowGroupsPrunedParquetOpen {
 struct DecoderReadPlans {
     projection: DecoderProjection,
     row_filter_context: Option<RowFilterContext>,
+    /// Builds a new projection when the adaptive filter placement changes.
+    /// `Some` exactly when the row filter context has a placement.
+    projection_builder: Option<DecoderProjectionBuilder>,
 }
 
 impl DecoderReadPlans {
@@ -576,75 +589,109 @@ impl DecoderReadPlans {
             .predicate
             .as_ref()
             .filter(|_| !prepared.pruning_only_predicate);
-        let build_projection = |post_scan_conjuncts: &[Arc<dyn PhysicalExpr>]| {
-            // Build the decoder projection (mask + per-batch transform +
-            // optional post-scan filter) in a single call. Encapsulating it
-            // behind `DecoderProjection` keeps the opener's orchestration body
-            // focused on filter / decoder / stream wiring. The file-column
-            // projection excludes virtual columns and respects nested field
-            // projections.
-            DecoderProjection::try_new(
-                &prepared.projection,
-                post_scan_conjuncts,
-                &prepared.physical_file_schema,
-                metadata.parquet_schema(),
-                &prepared.output_schema,
-                prepared.virtual_state.as_deref(),
-                &prepared.file_metrics,
-            )
+        // Build the decoder projection (mask + per-batch transform +
+        // optional post-scan filter). Encapsulating it behind
+        // `DecoderProjection` keeps the opener's orchestration body focused
+        // on filter / decoder / stream wiring. The file-column projection
+        // excludes virtual columns and respects nested field projections.
+        let mut projection_builder = DecoderProjectionBuilder {
+            projection: prepared.projection.clone(),
+            fixed_post_scan: vec![],
+            physical_file_schema: Arc::clone(&prepared.physical_file_schema),
+            parquet_schema: metadata.metadata().file_metadata().schema_descr_ptr(),
+            output_schema: Arc::clone(&prepared.output_schema),
+            virtual_state: prepared.virtual_state.clone(),
+            file_metrics: prepared.file_metrics.clone(),
         };
-        let (row_filter_context, projection) =
-            match (prepared.pushdown_filters, applied_predicate) {
-                // Pushdown enabled: precompute the candidate list once per file.
-                // Both the initial `RowFilter` and any per-RG rebuilds (via
-                // `RowFilterContext::build_row_filter`) reuse it, so tree walks
-                // (`reassign_expr_columns`) and column resolution only run once —
-                // not once per row group. Only what the `RowFilter` could not
-                // place falls through to post-scan.
-                (true, Some(predicate)) => {
-                    // The gates of optional filters estimate their saving from
-                    // the output columns, thus build the projection without
-                    // post-scan conjuncts first. It is used as-is when the
-                    // `RowFilter` can place every conjunct (the usual case).
-                    let output_projection = build_projection(&[])?;
-                    let (row_filter_context, rejected) = RowFilterContext::try_new(
-                        predicate,
-                        &prepared.physical_file_schema,
-                        metadata.metadata(),
-                        prepared.reorder_predicates,
-                        prepared.file_metrics.clone(),
-                        prepared.max_predicate_cache_size,
-                        Some(OptionalFilterRowFilterContext {
-                            options: &prepared.optional_filters,
-                            metrics: &prepared.metrics,
-                            partition: prepared.partition_index,
-                            filename: &prepared.file_name,
-                            output_projection: Some(output_projection.projection_mask()),
-                        }),
-                    );
-                    let projection = if rejected.is_empty() {
-                        output_projection
-                    } else {
-                        build_projection(&rejected)?
-                    };
-                    (row_filter_context, projection)
-                }
-                // Pushdown disabled: the required conjuncts run post-scan
-                // (in-scan equivalent of a `FilterExec`). Optional conjuncts
-                // (for example hash join dynamic filters) are not needed for
-                // correctness and are expensive to evaluate for each row,
-                // thus they are used only for statistics pruning, as before
-                // the scan accepted the filters.
-                (false, Some(predicate)) => {
-                    let (required, _optional) = split_optional(predicate);
-                    (None, build_projection(&required)?)
-                }
-                (_, None) => (None, build_projection(&[])?),
-            };
-        Ok(Self {
-            projection,
-            row_filter_context,
-        })
+        match (prepared.pushdown_filters, applied_predicate) {
+            // Pushdown enabled: precompute the candidate list once per file.
+            // Both the initial `RowFilter` and any per-RG rebuilds (via
+            // `RowFilterContext::build_row_filter`) reuse it, so tree walks
+            // (`reassign_expr_columns`) and column resolution only run once —
+            // not once per row group. Only what the `RowFilter` could not
+            // place falls through to post-scan.
+            (true, Some(predicate)) => {
+                // The gates of optional filters and the adaptive filter
+                // placement estimate their saving from the output columns,
+                // thus build the projection without post-scan conjuncts
+                // first. It is used as-is when the `RowFilter` can place every
+                // conjunct (the usual case).
+                let output_projection = projection_builder.build(&[])?;
+                let (row_filter_context, rejected) = RowFilterContext::try_new(
+                    predicate,
+                    &prepared.physical_file_schema,
+                    metadata.metadata(),
+                    prepared.reorder_predicates,
+                    prepared.file_metrics.clone(),
+                    prepared.max_predicate_cache_size,
+                    Some(OptionalFilterRowFilterContext {
+                        options: &prepared.optional_filters,
+                        metrics: &prepared.metrics,
+                        partition: prepared.partition_index,
+                        filename: &prepared.file_name,
+                        output_projection: Some(output_projection.projection_mask()),
+                    }),
+                );
+                projection_builder.fixed_post_scan =
+                    rejected.into_iter().map(PostScanConjunct::from).collect();
+                let row_filter_context = row_filter_context.map(|ctx| {
+                    ctx.with_placement(|candidates| {
+                        new_file_placement(
+                            prepared,
+                            metadata.metadata(),
+                            predicate,
+                            output_projection.projection_mask(),
+                            candidates,
+                        )
+                    })
+                });
+                let placed = row_filter_context
+                    .as_ref()
+                    .and_then(|ctx| ctx.placement())
+                    .map(FilePlacement::post_scan_conjuncts)
+                    .unwrap_or_default();
+                let projection = if placed.is_empty()
+                    && projection_builder.fixed_post_scan.is_empty()
+                {
+                    output_projection
+                } else {
+                    projection_builder.build(&placed)?
+                };
+                let mut row_filter_context = row_filter_context;
+                let placement = row_filter_context
+                    .as_mut()
+                    .and_then(|ctx| ctx.placement.as_mut());
+                let projection_builder = placement.map(|placement| {
+                    placement.set_decoder_mask(projection.projection_mask());
+                    projection_builder
+                });
+                Ok(Self {
+                    projection,
+                    row_filter_context,
+                    projection_builder,
+                })
+            }
+            // Pushdown disabled: the required conjuncts run post-scan (in-scan
+            // equivalent of a `FilterExec`). Optional conjuncts (for example
+            // hash join dynamic filters) are not needed for correctness and
+            // are expensive to evaluate for each row, thus they are used only
+            // for statistics pruning, as before the scan accepted the filters.
+            (false, Some(predicate)) => {
+                let (required, _optional) = split_optional(predicate);
+                projection_builder.fixed_post_scan =
+                    required.into_iter().map(PostScanConjunct::from).collect();
+                Ok(Self {
+                    projection: projection_builder.build(&[])?,
+                    row_filter_context: None,
+                    projection_builder: None,
+                })
+            }
+            (_, None) => Ok(Self {
+                projection: projection_builder.build(&[])?,
+                row_filter_context: None,
+                projection_builder: None,
+            }),
+        }
     }
 
     fn reads_leaf(&self, leaf_idx: usize) -> bool {
@@ -656,6 +703,38 @@ impl DecoderReadPlans {
     }
 }
 
+/// The adaptive filter placement of a file (see [`crate::filter_placement`]),
+/// or `None` when it is disabled or when no conjunct needs a decision.
+fn new_file_placement(
+    prepared: &PreparedParquetOpen,
+    metadata: &Arc<ParquetMetaData>,
+    predicate: &Arc<dyn PhysicalExpr>,
+    output_projection: &ProjectionMask,
+    candidates: &mut [PrebuiltRowFilterCandidate],
+) -> Option<FilePlacement> {
+    if !prepared.filter_placement.enabled {
+        return None;
+    }
+    // The first decision is made before the scan knows its first row group.
+    let row_groups = metadata.num_row_groups().max(1);
+    let mean_row_group_rows =
+        usize::try_from(metadata.file_metadata().num_rows()).unwrap_or(0) / row_groups;
+    let changes = MetricBuilder::new(&prepared.metrics)
+        .with_new_label("filename", prepared.file_name.clone())
+        .with_type(datafusion_physical_plan::metrics::MetricType::Summary)
+        .counter("filter_placement_changes", prepared.partition_index);
+    FilePlacement::try_new(
+        candidates,
+        predicate,
+        &prepared.filter_placement,
+        output_projection,
+        metadata,
+        prepared.batch_size,
+        mean_row_group_rows,
+        changes,
+    )
+}
+
 /// State of [`ParquetOpenState`]
 ///
 /// Result of loading bloom filters needed for row-group pruning.
@@ -1069,6 +1148,7 @@ impl ParquetMorselizer {
             sort_order_for_reorder: self.sort_order_for_reorder.clone(),
             preserve_order: self.preserve_order,
             optional_filters: self.optional_filters.clone(),
+            filter_placement: self.filter_placement.clone(),
             #[cfg(feature = "parquet_encryption")]
             file_decryption_properties: None,
         })
@@ -1769,6 +1849,7 @@ impl RowGroupsPrunedParquetOpen {
         let DecoderReadPlans {
             projection: decoder_projection,
             row_filter_context: precomputed_context,
+            projection_builder,
         } = match decoder_read_plans {
             Some(plans) => plans,
             None => DecoderReadPlans::try_new(&prepared, &reader_metadata)?,
@@ -1786,8 +1867,10 @@ impl RowGroupsPrunedParquetOpen {
         // Decoder-local LIMIT is only safe when no post-decode work can reject
         // rows. A post-scan filter can — so when one is present the limit is
         // enforced at the stream level via `remaining_limit` and kept out of
-        // the decoder; otherwise it is pushed into the decoder.
-        let has_post_scan_filter = decoder_projection.has_post_scan_filter();
+        // the decoder; otherwise it is pushed into the decoder. With adaptive
+        // filter placement, a post-scan filter can start at any row group.
+        let has_post_scan_filter =
+            decoder_projection.has_post_scan_filter() || projection_builder.is_some();
         let decoder_limit = prepared.limit.filter(|_| !has_post_scan_filter);
         let remaining_limit = prepared.limit.filter(|_| has_post_scan_filter);
 
@@ -1844,7 +1927,17 @@ impl RowGroupsPrunedParquetOpen {
             // installed filter is owned by the decoder and is not
             // recoverable once replaced.
             let first_rg_fully_matched = rg_plan.front().is_some_and(|e| e.fully_matched);
-            let row_filter_context = precomputed_context;
+            let mut row_filter_context = precomputed_context;
+            // A rebuild at a row group boundary cannot keep a live row
+            // selection (#24355), thus the adaptive filter placement keeps
+            // its first decision for this file.
+            if has_row_selection
+                && let Some(placement) = row_filter_context
+                    .as_mut()
+                    .and_then(|ctx| ctx.placement.as_mut())
+            {
+                placement.freeze();
+            }
 
             let mut builder = decoder_config.build(selections, reader_metadata.clone());
             let mut filter_installed = false;
@@ -1973,9 +2066,16 @@ impl RowGroupsPrunedParquetOpen {
             remaining_limit,
             // A post-scan filter can leave only a few rows per decoded batch;
             // reassemble them so the operator above sees full-size batches.
-            batch_coalescer: has_post_scan_filter
-                .then(|| BatchCoalescer::new(filtered_schema, prepared.batch_size)),
+            // Full-size batches (no post-scan filter for a row group, with
+            // adaptive filter placement) are not copied.
+            batch_coalescer: has_post_scan_filter.then(|| {
+                let biggest =
+                    projection_builder.as_ref().map(|_| prepared.batch_size / 2);
+                BatchCoalescer::new(filtered_schema, prepared.batch_size)
+                    .with_biggest_coalesce_batch_size(biggest)
+            }),
             flushed: false,
+            projection_builder,
         }
         .into_stream();
 
@@ -2280,6 +2380,7 @@ mod test {
         reverse_row_groups: bool,
         preserve_order: bool,
         optional_filters: OptionalFilterOptions,
+        filter_placement: bool,
     }
 
     #[test]
@@ -2504,6 +2605,7 @@ mod test {
                 reverse_row_groups: false,
                 preserve_order: false,
                 optional_filters: OptionalFilterOptions::default(),
+                filter_placement: false,
             }
         }
 
@@ -2603,6 +2705,16 @@ mod test {
         }
 
         /// Set whether the scan must preserve file order.
+        fn with_filter_placement(mut self, enable: bool) -> Self {
+            self.filter_placement = enable;
+            self
+        }
+
+        fn with_optional_filters(mut self, options: OptionalFilterOptions) -> Self {
+            self.optional_filters = options;
+            self
+        }
+
         fn with_preserve_order(mut self, enable: bool) -> Self {
             self.preserve_order = enable;
             self
@@ -2653,6 +2765,11 @@ mod test {
                 self.pushdown_filters,
             )?;
 
+            let filter_placement = PlacementOptions::new(
+                self.filter_placement,
+                Arc::default(),
+                self.predicate.as_ref(),
+            );
             Ok(ParquetMorselizer {
                 partition_index: self.partition_index,
                 projection,
@@ -2692,6 +2809,7 @@ mod test {
                 sort_order_for_reorder: None,
                 virtual_state,
                 optional_filters: self.optional_filters,
+                filter_placement,
             })
         }
     }
@@ -5568,6 +5686,278 @@ mod test {
         (schema, file)
     }
 
+    /// End-to-end tests of the adaptive filter placement
+    /// (`crate::filter_placement`).
+    mod filter_placement {
+        use super::*;
+        use arrow::array::{Int32Array, Int64Array};
+        use datafusion_common::config::OptionalFilterMode;
+        use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
+        use datafusion_physical_expr::optional_filter_gate::OptionalFilterGateConfig;
+
+        const ROW_GROUPS: usize = 4;
+        const ROWS_PER_ROW_GROUP: usize = 16 * 1024;
+        const TOTAL_ROWS: usize = ROW_GROUPS * ROWS_PER_ROW_GROUP;
+
+        /// A file with `ROW_GROUPS` row groups and the columns `a` (Int32,
+        /// `a(i)` for row `i`) and `v` (Int64, the row number).
+        async fn write_file(
+            store: &Arc<dyn ObjectStore>,
+            a: impl Fn(usize) -> i32,
+        ) -> (SchemaRef, PartitionedFile) {
+            let schema = Arc::new(Schema::new(vec![
+                Field::new("a", DataType::Int32, false),
+                Field::new("v", DataType::Int64, false),
+            ]));
+            let batch = RecordBatch::try_new(
+                Arc::clone(&schema),
+                vec![
+                    Arc::new(Int32Array::from_iter_values((0..TOTAL_ROWS).map(&a))),
+                    Arc::new(Int64Array::from_iter_values(0..TOTAL_ROWS as i64)),
+                ],
+            )
+            .unwrap();
+            let props = WriterProperties::builder()
+                .set_max_row_group_row_count(Some(ROWS_PER_ROW_GROUP))
+                .build();
+            let size = write_parquet_batches(
+                Arc::clone(store),
+                "placement.parquet",
+                vec![batch],
+                Some(props),
+            )
+            .await;
+            let file = PartitionedFile::new("placement.parquet".to_string(), size as u64);
+            (schema, file)
+        }
+
+        struct Scan {
+            rows: usize,
+            metrics: ExecutionPlanMetricsSet,
+        }
+
+        impl Scan {
+            fn count(&self, name: &str) -> usize {
+                counter_metric_value(&self.metrics, name)
+            }
+
+            /// Rows that the `RowFilter` evaluated.
+            fn row_filter_rows(&self) -> usize {
+                self.count("pushdown_rows_matched") + self.count("pushdown_rows_pruned")
+            }
+
+            /// Rows that the post-scan filter evaluated.
+            fn post_scan_rows(&self) -> usize {
+                self.count("post_scan_rows_matched") + self.count("post_scan_rows_pruned")
+            }
+        }
+
+        async fn scan(
+            store: &Arc<dyn ObjectStore>,
+            schema: &SchemaRef,
+            file: &PartitionedFile,
+            predicate: &Arc<dyn PhysicalExpr>,
+            pushdown_filters: bool,
+            filter_placement: bool,
+            optional_filters: OptionalFilterOptions,
+        ) -> Scan {
+            let metrics = ExecutionPlanMetricsSet::new();
+            let morselizer = ParquetMorselizerBuilder::new()
+                .with_store(Arc::clone(store))
+                .with_schema(Arc::clone(schema))
+                .with_predicate(Arc::clone(predicate))
+                .with_pushdown_filters(pushdown_filters)
+                .with_filter_placement(filter_placement)
+                .with_optional_filters(optional_filters)
+                .with_metrics(metrics.clone())
+                .build();
+            let stream = open_file(&morselizer, file.clone()).await.unwrap();
+            let (_, rows) = count_batches_and_rows(stream).await;
+            Scan { rows, metrics }
+        }
+
+        /// A filter that removes every second row saves no decode time in a
+        /// row filter: the removed rows do not make runs that the decoder can
+        /// skip. The scan starts it as a row filter (the initial rule: the
+        /// filter does not read `v`), measures it in the first row group and
+        /// moves it to the post-scan filter at the first row group boundary.
+        /// The result is the same in all placements.
+        #[tokio::test]
+        async fn scattered_filter_moves_post_scan_at_row_group_boundary() {
+            let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+            let (schema, file) = write_file(&store, |i| (i % 2) as i32).await;
+            let predicate = logical2physical(&col("a").eq(lit(0)), &schema);
+            let options = OptionalFilterOptions::default();
+
+            let adaptive = scan(
+                &store,
+                &schema,
+                &file,
+                &predicate,
+                true,
+                true,
+                options.clone(),
+            )
+            .await;
+            let row_filter = scan(
+                &store,
+                &schema,
+                &file,
+                &predicate,
+                true,
+                false,
+                options.clone(),
+            )
+            .await;
+            let post_scan =
+                scan(&store, &schema, &file, &predicate, false, false, options).await;
+
+            for scan in [&adaptive, &row_filter, &post_scan] {
+                assert_eq!(scan.rows, TOTAL_ROWS / 2);
+            }
+            assert_eq!(adaptive.count("filter_placement_changes"), 1);
+            assert_eq!(adaptive.row_filter_rows(), ROWS_PER_ROW_GROUP);
+            assert_eq!(adaptive.post_scan_rows(), TOTAL_ROWS - ROWS_PER_ROW_GROUP);
+            assert_eq!(row_filter.row_filter_rows(), TOTAL_ROWS);
+            assert_eq!(post_scan.post_scan_rows(), TOTAL_ROWS);
+        }
+
+        /// A filter that removes long runs of rows saves decode time in a row
+        /// filter, thus it stays a row filter.
+        #[tokio::test]
+        async fn clustered_filter_stays_row_filter() {
+            let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+            // `a` increases by 1 every 1000 rows.
+            let (schema, file) = write_file(&store, |i| (i / 1000) as i32).await;
+            // 10000 rows in the first row group and 10000 rows in the third.
+            let predicate = logical2physical(
+                &col("a").lt(lit(10)).or(col("a").between(lit(40), lit(49))),
+                &schema,
+            );
+            let adaptive = scan(
+                &store,
+                &schema,
+                &file,
+                &predicate,
+                true,
+                true,
+                OptionalFilterOptions::default(),
+            )
+            .await;
+            assert_eq!(adaptive.rows, 20_000);
+            assert_eq!(adaptive.count("filter_placement_changes"), 0);
+            assert_eq!(adaptive.row_filter_rows(), TOTAL_ROWS);
+            assert_eq!(adaptive.post_scan_rows(), 0);
+        }
+
+        /// A filter that reads all output columns cannot save decode time in
+        /// a row filter, thus it starts in the post-scan filter.
+        #[tokio::test]
+        async fn filter_on_all_output_columns_starts_post_scan() {
+            let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+            let (schema, file) = write_file(&store, |i| (i / 1000) as i32).await;
+            let predicate = logical2physical(&col("a").lt(lit(10)), &schema);
+            let metrics = ExecutionPlanMetricsSet::new();
+            let morselizer = ParquetMorselizerBuilder::new()
+                .with_store(Arc::clone(&store))
+                .with_schema(Arc::clone(&schema))
+                .with_projection_indices(&[0])
+                .with_predicate(predicate)
+                .with_pushdown_filters(true)
+                .with_filter_placement(true)
+                .with_metrics(metrics.clone())
+                .build();
+            let stream = open_file(&morselizer, file).await.unwrap();
+            let (_, rows) = count_batches_and_rows(stream).await;
+            let scan = Scan { rows, metrics };
+            assert_eq!(scan.rows, 10_000);
+            assert_eq!(scan.row_filter_rows(), 0);
+            assert_eq!(scan.post_scan_rows(), TOTAL_ROWS);
+            assert_eq!(scan.count("filter_placement_changes"), 0);
+        }
+
+        /// An optional filter that removes no rows is paused by its gate. At
+        /// the next row group boundary, the scan removes it from the
+        /// `RowFilter`: it is not evaluated and its column is not decoded for
+        /// the row filter. Without adaptive placement, the paused filter stays
+        /// a `RowFilter` predicate that lets all rows pass.
+        #[tokio::test]
+        async fn paused_optional_filter_is_removed_from_row_filter() {
+            let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+            let (schema, file) = write_file(&store, |i| (i % 100) as i32).await;
+            let inner = logical2physical(&col("a").gt_eq(lit(0)), &schema);
+            let predicate: Arc<dyn PhysicalExpr> =
+                Arc::new(OptionalFilterPhysicalExpr::new(inner));
+            let options = || OptionalFilterOptions {
+                mode: OptionalFilterMode::Adaptive,
+                gate_config: OptionalFilterGateConfig::default(),
+                decode_cost: Arc::default(),
+            };
+
+            let adaptive =
+                scan(&store, &schema, &file, &predicate, true, true, options()).await;
+            let gated =
+                scan(&store, &schema, &file, &predicate, true, false, options()).await;
+
+            for scan in [&adaptive, &gated] {
+                assert_eq!(scan.rows, TOTAL_ROWS);
+                assert!(scan.count("optional_filter_rows_skipped") > 0);
+            }
+            // Without placement, every row goes through the row filter.
+            assert_eq!(gated.row_filter_rows(), TOTAL_ROWS);
+            assert_eq!(gated.count("filter_placement_changes"), 0);
+            // With placement, the row groups where the filter is paused have
+            // no row filter.
+            assert!(adaptive.count("filter_placement_changes") > 0);
+            assert!(
+                adaptive.row_filter_rows() < TOTAL_ROWS,
+                "row filter rows: {}",
+                adaptive.row_filter_rows()
+            );
+            assert_eq!(adaptive.post_scan_rows(), 0);
+        }
+
+        /// With a limit, the results are the same when the placement moves a
+        /// filter to the post-scan filter in the middle of the file.
+        #[tokio::test]
+        async fn limit_is_applied_after_placement_change() {
+            let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+            let (schema, file) = write_file(&store, |i| (i % 2) as i32).await;
+            let predicate = logical2physical(&col("a").eq(lit(0)), &schema);
+            let limit = ROWS_PER_ROW_GROUP / 2 + 100;
+            let metrics = ExecutionPlanMetricsSet::new();
+            let morselizer = ParquetMorselizerBuilder::new()
+                .with_store(Arc::clone(&store))
+                .with_schema(Arc::clone(&schema))
+                .with_predicate(predicate)
+                .with_pushdown_filters(true)
+                .with_filter_placement(true)
+                .with_limit(limit)
+                .with_metrics(metrics.clone())
+                .build();
+            let stream = open_file(&morselizer, file).await.unwrap();
+            let batches = stream.try_collect::<Vec<_>>().await.unwrap();
+            let values: Vec<i64> = batches
+                .iter()
+                .flat_map(|batch| {
+                    batch
+                        .column(1)
+                        .as_any()
+                        .downcast_ref::<Int64Array>()
+                        .unwrap()
+                        .values()
+                        .to_vec()
+                })
+                .collect();
+            let expected: Vec<i64> = (0..limit as i64).map(|i| i * 2).collect();
+            assert_eq!(values, expected);
+            assert_eq!(
+                counter_metric_value(&metrics, "filter_placement_changes"),
+                1
+            );
+        }
+    }
+
     /// Helpers for tests that exercise parquet virtual columns
     /// (e.g. `row_number`) plumbed through `TableSchema`/`ParquetOpener`.
     mod virtual_columns {
diff --git a/datafusion/datasource-parquet/src/push_decoder.rs b/datafusion/datasource-parquet/src/push_decoder.rs
index 0fb99c71fa406..7a520ed017f4d 100644
--- a/datafusion/datasource-parquet/src/push_decoder.rs
+++ b/datafusion/datasource-parquet/src/push_decoder.rs
@@ -37,6 +37,7 @@
 
 use std::collections::VecDeque;
 use std::sync::Arc;
+use std::time::Duration;
 
 use arrow::array::RecordBatch;
 use arrow::compute::BatchCoalescer;
@@ -65,7 +66,10 @@ use datafusion_physical_plan::metrics::{BaselineMetrics, Count, Gauge};
 use datafusion_pruning::{PruningPredicate, PruningPredicateBuilder};
 
 use crate::ParquetFileMetrics;
-use crate::decoder_projection::{DecoderProjection, PostScanSelection};
+use crate::decoder_projection::{
+    DecoderProjection, DecoderProjectionBuilder, PostScanSelection,
+};
+use crate::filter_placement::FilePlacement;
 use crate::metrics::{ByteProgress, RowFilterSkippedFullyMatchedMetric};
 use crate::optional_filter::OptionalFilterSavings;
 use crate::row_filter::{
@@ -345,6 +349,10 @@ pub(crate) struct PushDecoderStreamState {
     /// end-of-input flushing happens exactly once no matter which terminal
     /// path reached it.
     pub(crate) flushed: bool,
+    /// Builds a new [`DecoderProjection`] when the adaptive filter placement
+    /// changes the post-scan conjuncts at a row group boundary. `Some`
+    /// exactly when [`RowFilterContext::placement`] is `Some`.
+    pub(crate) projection_builder: Option<DecoderProjectionBuilder>,
 }
 
 /// A reusable, `Arc`-shared list of prebuilt row-filter candidates.
@@ -383,6 +391,10 @@ pub(crate) struct RowFilterContext {
     /// Measures the decode time of the output batches for the gates of the
     /// optional filters. `None` if the file has no gated optional filter.
     pub(crate) optional_savings: Option<OptionalFilterSavings>,
+    /// The adaptive placement of the conjuncts (see
+    /// [`crate::filter_placement`]). `None` when it is disabled or when no
+    /// conjunct needs a decision.
+    pub(crate) placement: Option<FilePlacement>,
 }
 
 impl RowFilterContext {
@@ -429,6 +441,7 @@ impl RowFilterContext {
                         file_metrics,
                         max_predicate_cache_size,
                         optional_savings,
+                        placement: None,
                     }
                 });
                 (context, rejected)
@@ -447,6 +460,41 @@ impl RowFilterContext {
         }
     }
 
+    /// Adds the adaptive filter placement that `make` returns. `make` gets
+    /// the prebuilt candidates, before any `RowFilter` is built from them.
+    pub(crate) fn with_placement(
+        mut self,
+        make: impl FnOnce(&mut [PrebuiltRowFilterCandidate]) -> Option<FilePlacement>,
+    ) -> Self {
+        let candidates = Arc::get_mut(&mut self.prebuilt.inner)
+            .expect("no RowFilter was built from the candidates yet");
+        self.placement = make(candidates);
+        self
+    }
+
+    /// The adaptive filter placement of the file, if any.
+    pub(crate) fn placement(&self) -> Option<&FilePlacement> {
+        self.placement.as_ref()
+    }
+
+    /// True if [`Self::record_output_batch`] needs the decode time of the
+    /// output batches.
+    pub(crate) fn measures_decode(&self) -> bool {
+        self.optional_savings.is_some() || self.placement.is_some()
+    }
+
+    /// Records that the decoder produced an output batch of `rows` rows in
+    /// `elapsed`, for the gates of the optional filters and for the adaptive
+    /// filter placement.
+    pub(crate) fn record_output_batch(&self, rows: usize, elapsed: Duration) {
+        if let Some(savings) = &self.optional_savings {
+            savings.record_output_batch(rows, elapsed);
+        }
+        if let Some(placement) = &self.placement {
+            placement.record_decode(rows, elapsed);
+        }
+    }
+
     /// Whether any pushed-down predicate reads this Parquet leaf column.
     pub(crate) fn reads_leaf(&self, leaf_idx: usize) -> bool {
         self.prebuilt
@@ -461,9 +509,18 @@ impl RowFilterContext {
     ///
     /// Infallible by construction: [`Self::try_new`] only produces a context
     /// when the prebuilt candidate list is non-empty.
+    ///
+    /// With adaptive filter placement, only the candidates that are placed
+    /// in the `RowFilter` are used.
     pub(crate) fn build_row_filter(&self) -> RowFilter {
+        let placement = self.placement.as_ref();
         row_filter_from_prebuilt(
-            self.prebuilt.as_slice(),
+            self.prebuilt
+                .as_slice()
+                .iter()
+                .enumerate()
+                .filter(|(index, _)| placement.is_none_or(|p| p.in_row_filter(*index)))
+                .map(|(_, candidate)| candidate),
             self.reorder_predicates,
             &self.file_metrics,
         )
@@ -534,19 +591,19 @@ impl PushDecoderStreamState {
 
             // Step 1: drain a batch from the active reader if any.
             if let Some(reader) = self.active_reader.as_mut() {
-                // The gates of optional filters need the decode time of the
-                // output columns (the reader does not evaluate the row
-                // filter: the decoder did that before it returned the reader).
-                let optional_savings = self
+                // The gates of optional filters and the adaptive filter
+                // placement need the decode time of the output columns (the
+                // reader does not evaluate the row filter: the decoder did
+                // that before it returned the reader).
+                let decode_ctx = self
                     .row_filter_context
                     .as_ref()
-                    .and_then(|ctx| ctx.optional_savings.as_ref());
-                let start = optional_savings.map(|_| Instant::now());
+                    .filter(|ctx| ctx.measures_decode());
+                let start = decode_ctx.map(|_| Instant::now());
                 match reader.next() {
                     Some(Ok(batch)) => {
-                        if let (Some(savings), Some(start)) = (optional_savings, start) {
-                            savings
-                                .record_output_batch(batch.num_rows(), start.elapsed());
+                        if let (Some(ctx), Some(start)) = (decode_ctx, start) {
+                            ctx.record_output_batch(batch.num_rows(), start.elapsed());
                         }
                         self.copy_arrow_reader_metrics();
 
@@ -589,6 +646,18 @@ impl PushDecoderStreamState {
                             continue;
                         }
 
+                        // No post-scan filter, but a coalescer: the adaptive
+                        // filter placement can add a post-scan filter at a
+                        // later row group, thus the batches go through the
+                        // coalescer to keep their order, and the limit is
+                        // applied on its output.
+                        if let Some(coalescer) = self.batch_coalescer.as_mut() {
+                            if let Err(e) = coalescer.push_batch(batch) {
+                                return Some((Err(DataFusionError::from(e)), self));
+                            }
+                            continue;
+                        }
+
                         // No post-scan filter: the decoder's batches are
                         // already the right shape, so project and yield
                         // directly. The limit was pushed into the decoder.
@@ -656,11 +725,25 @@ impl PushDecoderStreamState {
                 Ok(DecodeResult::NeedsData(ranges)) => {
                     // I/O, not compute.
                     timer.stop();
+                    // The adaptive filter placement uses the fetch latency.
+                    let fetch_start = self
+                        .row_filter_context
+                        .as_ref()
+                        .and_then(|ctx| ctx.placement())
+                        .map(|_| Instant::now());
                     let data = self
                         .reader
                         .get_byte_ranges(ranges.clone())
                         .await
                         .map_err(DataFusionError::from);
+                    if let (Some(start), Some(placement)) = (
+                        fetch_start,
+                        self.row_filter_context
+                            .as_ref()
+                            .and_then(|ctx| ctx.placement()),
+                    ) {
+                        placement.record_fetch(start.elapsed());
+                    }
                     timer.restart();
                     match data {
                         Ok(data) => {
@@ -793,6 +876,11 @@ impl PushDecoderStreamState {
         &mut self,
         pruned_count: usize,
     ) -> Result<bool, DataFusionError> {
+        // The adaptive filter placement for the next RG. `Some` when it
+        // changed: then the decoder needs the new projection and a new
+        // `RowFilter`.
+        let new_projection = self.update_placement()?;
+
         // `desired_filter` is `Some(true)` when the next RG needs a real
         // filter, `Some(false)` when it is fully-matched (filter is a no-op, so
         // we suppress it), and `None` when there is no pushdown predicate at
@@ -801,10 +889,11 @@ impl PushDecoderStreamState {
             .row_filter_context
             .as_ref()
             .and_then(|_| self.rg_plan.front().map(|e| !e.fully_matched));
-        let filter_needs_toggle =
-            desired_filter.is_some_and(|want| want != self.filter_installed);
+        let filter_needs_toggle = desired_filter.is_some_and(|want| {
+            want != self.filter_installed || (want && new_projection.is_some())
+        });
 
-        if pruned_count == 0 && !filter_needs_toggle {
+        if pruned_count == 0 && !filter_needs_toggle && new_projection.is_none() {
             return Ok(false);
         }
         if self.rg_plan.is_empty() {
@@ -824,6 +913,10 @@ impl PushDecoderStreamState {
                 .collect();
             builder = builder.with_row_group_selections(selections);
         }
+        if let Some(projection) = new_projection {
+            builder = builder.with_projection(projection.projection_mask().clone());
+            self.decoder_projection = projection;
+        }
         if filter_needs_toggle {
             let want_filter = desired_filter.expect("filter_needs_toggle ⇒ desired Some");
             if want_filter {
@@ -847,6 +940,41 @@ impl PushDecoderStreamState {
         Ok(false)
     }
 
+    /// Makes the adaptive filter placement for the next RG
+    /// (`rg_plan.front()`). Returns the decoder projection for the new
+    /// placement if it changed.
+    ///
+    /// The coalescer holds batches with the schema of the current
+    /// projection. If the new post-scan conjuncts change that schema (this
+    /// can happen with nested columns), the file keeps its current placement
+    /// until its end.
+    fn update_placement(&mut self) -> Result<Option<DecoderProjection>> {
+        let (Some(builder), Some(front)) =
+            (self.projection_builder.as_ref(), self.rg_plan.front())
+        else {
+            return Ok(None);
+        };
+        let Some(placement) = self
+            .row_filter_context
+            .as_mut()
+            .and_then(|ctx| ctx.placement.as_mut())
+        else {
+            return Ok(None);
+        };
+        let previous = placement.placements();
+        let rows = placement.row_group_rows(front.rg_index);
+        if !placement.decide(rows) {
+            return Ok(None);
+        }
+        let projection = builder.build(&placement.post_scan_conjuncts())?;
+        if projection.filtered_schema() != self.decoder_projection.filtered_schema() {
+            placement.restore_and_freeze(&previous);
+            return Ok(None);
+        }
+        placement.set_decoder_mask(projection.projection_mask());
+        Ok(Some(projection))
+    }
+
     /// Copies metrics from ArrowReaderMetrics (the metrics collected by the
     /// arrow-rs parquet reader) to the parquet file metrics for DataFusion
     fn copy_arrow_reader_metrics(&self) {
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index f3ac046ebab60..599b1364d43b5 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -81,10 +81,9 @@ use datafusion_common::Result;
 use datafusion_common::cast::as_boolean_array;
 use datafusion_common::config::OptionalFilterMode;
 use datafusion_common::tree_node::TreeNode;
+use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
 use datafusion_physical_expr::optional_filter_gate::{GateDecision, OptionalFilterGate};
-use datafusion_physical_expr::utils::{
-    is_optional_filter, reassign_expr_columns, split_optional,
-};
+use datafusion_physical_expr::utils::{is_optional_filter, reassign_expr_columns};
 use datafusion_physical_expr::{PhysicalExpr, split_conjunction};
 
 use datafusion_physical_plan::metrics::{self, ExecutionPlanMetricsSet};
@@ -92,6 +91,7 @@ use parking_lot::Mutex;
 
 use super::ParquetFileMetrics;
 use super::supported_predicates::supports_list_predicates;
+use crate::filter_placement::ConjunctStats;
 use crate::metrics::OptionalFilterMetrics;
 use crate::optional_filter::{
     OptionalFilterOptions, OptionalFilterSaving, compressed_bytes_per_row,
@@ -132,6 +132,10 @@ pub(crate) struct DatafusionArrowPredicate {
     /// The gate of an optional filter in [`OptionalFilterMode::Adaptive`].
     /// `None` for all other filters.
     gate: Option<SharedOptionalFilterGate>,
+    /// The measurements of the conjunct for the adaptive filter placement
+    /// (see [`crate::filter_placement`]). `None` if the placement does not
+    /// manage the conjunct.
+    placement_stats: Option<Arc<ConjunctStats>>,
 }
 
 /// The [`OptionalFilterGate`] of one optional filter in one file, with its
@@ -149,7 +153,7 @@ pub(crate) struct OptionalFilterGateState {
     reported_pauses: usize,
 }
 
-type SharedOptionalFilterGate = Arc<Mutex<OptionalFilterGateState>>;
+pub(crate) type SharedOptionalFilterGate = Arc<Mutex<OptionalFilterGateState>>;
 
 impl OptionalFilterGateState {
     fn begin_batch(&mut self, num_rows: usize) -> GateDecision {
@@ -167,6 +171,26 @@ impl OptionalFilterGateState {
         self.report_pauses();
     }
 
+    /// True if the gate skips the next batch.
+    pub(crate) fn is_paused(&self) -> bool {
+        self.gate.is_paused()
+    }
+
+    /// Counts down up to `batches` batches of `rows_per_batch` rows that the
+    /// scan did not evaluate the filter on, while the gate is paused. The
+    /// adaptive filter placement calls this when it removed the paused
+    /// filter from the `RowFilter` (see [`crate::filter_placement`]). Stops
+    /// when the gate stops the pause, for example because the filter
+    /// changed.
+    pub(crate) fn skip_batches(&mut self, batches: usize, rows_per_batch: usize) {
+        for _ in 0..batches {
+            if !self.gate.is_paused() {
+                break;
+            }
+            self.begin_batch(rows_per_batch);
+        }
+    }
+
     fn report_pauses(&mut self) {
         let pauses = self.gate.pauses();
         self.metrics
@@ -198,6 +222,7 @@ impl DatafusionArrowPredicate {
             rows_matched,
             time,
             gate: None,
+            placement_stats: None,
         })
     }
 }
@@ -233,6 +258,9 @@ impl ArrowPredicate for DatafusionArrowPredicate {
                 let num_pruned = bool_arr.len() - num_matched;
                 self.rows_pruned.add(num_pruned);
                 self.rows_matched.add(num_matched);
+                if let Some(stats) = &self.placement_stats {
+                    stats.record_evaluation(&bool_arr);
+                }
                 if let (Some(gate), Some(start)) = (gate.as_mut(), start) {
                     let elapsed = gate.gate.clock().now_nanos().saturating_sub(start);
                     gate.record(num_rows, num_matched, Duration::from_nanos(elapsed));
@@ -557,6 +585,14 @@ pub(crate) struct PrebuiltRowFilterCandidate {
     /// The measured saving that the gate reads, if the output projection of
     /// the file is known. See [`crate::optional_filter`].
     saving: Option<OptionalFilterSaving>,
+    /// Position of the conjunct in the root `AND` chain of the file
+    /// predicate.
+    position: usize,
+    /// The conjunct in terms of the file schema (the inner expression for a
+    /// gated optional filter).
+    source_expr: Arc<dyn PhysicalExpr>,
+    /// See [`DatafusionArrowPredicate::placement_stats`].
+    placement_stats: Option<Arc<ConjunctStats>>,
 }
 
 impl PrebuiltRowFilterCandidate {
@@ -573,6 +609,29 @@ impl PrebuiltRowFilterCandidate {
     pub(crate) fn is_optional(&self) -> bool {
         self.gate.is_some()
     }
+
+    /// The gate of a gated optional filter.
+    pub(crate) fn gate(&self) -> Option<&SharedOptionalFilterGate> {
+        self.gate.as_ref()
+    }
+
+    /// Position of the conjunct in the root `AND` chain of the file
+    /// predicate.
+    pub(crate) fn position(&self) -> usize {
+        self.position
+    }
+
+    /// The conjunct in terms of the file schema (the inner expression for a
+    /// gated optional filter).
+    pub(crate) fn source_expr(&self) -> &Arc<dyn PhysicalExpr> {
+        &self.source_expr
+    }
+
+    /// Records the evaluations of this candidate in `stats`, for the
+    /// adaptive filter placement.
+    pub(crate) fn set_placement_stats(&mut self, stats: Arc<ConjunctStats>) {
+        self.placement_stats = Some(stats);
+    }
 }
 
 /// What [`prebuild_row_filter_candidates`] needs to handle optional
@@ -614,7 +673,8 @@ pub(crate) struct OptionalFilterRowFilterContext<'a> {
 ///
 /// `optional` controls the conjuncts of the root `AND` chain that are
 /// wrapped in an `OptionalFilterPhysicalExpr` (see
-/// [`split_optional`]). The other (required) conjuncts are always used.
+/// [`split_optional`](datafusion_physical_expr::utils::split_optional)).
+/// The other (required) conjuncts are always used.
 ///
 /// * `None` or [`OptionalFilterMode::Always`]: optional conjuncts are used
 ///   like required conjuncts (the `OptionalFilterPhysicalExpr` wrapper is
@@ -644,25 +704,27 @@ pub(crate) fn prebuild_row_filter_candidates(
 )> {
     let mode = optional.map_or(OptionalFilterMode::Always, |o| o.options.mode);
 
-    // Split into conjuncts:
+    // Split into conjuncts, with their position in the root `AND` chain:
     // `a = 1 AND b = 2 AND c = 3` -> [`a = 1`, `b = 2`, `c = 3`]
-    let (required, optional_conjuncts) = match mode {
-        OptionalFilterMode::Always => (
-            split_conjunction(expr)
-                .into_iter()
-                .map(Arc::clone)
-                .collect(),
-            vec![],
-        ),
-        OptionalFilterMode::PruningOnly => (split_optional(expr).0, vec![]),
-        OptionalFilterMode::Adaptive => split_optional(expr),
-    };
+    let mut required = vec![];
+    let mut optional_conjuncts = vec![];
+    for (position, conjunct) in split_conjunction(expr).into_iter().enumerate() {
+        match (conjunct.downcast_ref::<OptionalFilterPhysicalExpr>(), mode) {
+            (None, _) | (Some(_), OptionalFilterMode::Always) => {
+                required.push((position, Arc::clone(conjunct)))
+            }
+            (Some(_), OptionalFilterMode::PruningOnly) => {}
+            (Some(optional), OptionalFilterMode::Adaptive) => {
+                optional_conjuncts.push((position, Arc::clone(optional.inner())))
+            }
+        }
+    }
 
     // Required conjuncts that cannot be evaluated as ArrowPredicates are
     // returned to the caller so they are never silently dropped.
     let mut prebuilt = Vec::with_capacity(required.len() + optional_conjuncts.len());
     let mut rejected: Vec<Arc<dyn PhysicalExpr>> = Vec::new();
-    for conjunct in required {
+    for (position, conjunct) in required {
         // In the `always` mode, `required` also holds the optional conjuncts
         // (still wrapped). An optional conjunct that cannot be pushed down
         // for this file is not needed for correctness: do not use it, and do
@@ -671,9 +733,8 @@ pub(crate) fn prebuild_row_filter_candidates(
             match FilterCandidateBuilder::new(conjunct, Arc::clone(file_schema))
                 .build(metadata)
             {
-                Ok(Some(candidate)) => {
-                    prebuilt.push(PrebuiltRowFilterCandidate::try_new(candidate)?)
-                }
+                Ok(Some(candidate)) => prebuilt
+                    .push(PrebuiltRowFilterCandidate::try_new(candidate, position)?),
                 Ok(None) => {}
                 Err(e) => {
                     log::debug!(
@@ -687,14 +748,14 @@ pub(crate) fn prebuild_row_filter_candidates(
             .build(metadata)?
         {
             Some(candidate) => {
-                prebuilt.push(PrebuiltRowFilterCandidate::try_new(candidate)?)
+                prebuilt.push(PrebuiltRowFilterCandidate::try_new(candidate, position)?)
             }
             None => rejected.push(conjunct),
         }
     }
 
     if let Some(optional) = optional {
-        for conjunct in optional_conjuncts {
+        for (position, conjunct) in optional_conjuncts {
             // An optional conjunct that cannot be pushed down for this file
             // (or that fails to build) is not needed for correctness, thus it
             // is simply not used.
@@ -711,7 +772,7 @@ pub(crate) fn prebuild_row_filter_candidates(
                         continue;
                     }
                 };
-            let mut candidate = PrebuiltRowFilterCandidate::try_new(candidate)?;
+            let mut candidate = PrebuiltRowFilterCandidate::try_new(candidate, position)?;
             // The gate watches the dynamic filters in the same (live)
             // expression that the predicate evaluates, thus it sees their
             // updates.
@@ -753,9 +814,11 @@ pub(crate) fn prebuild_row_filter_candidates(
 }
 
 impl PrebuiltRowFilterCandidate {
-    /// A candidate without a gate. Its expression is column-reassigned to
-    /// the projected schema of the candidate.
-    fn try_new(candidate: FilterCandidate) -> Result<Self> {
+    /// A candidate without a gate, for the conjunct at `position` in the
+    /// root `AND` chain of the file predicate. Its expression is
+    /// column-reassigned to the projected schema of the candidate.
+    fn try_new(candidate: FilterCandidate, position: usize) -> Result<Self> {
+        let source_expr = Arc::clone(&candidate.expr);
         let physical_expr =
             reassign_expr_columns(candidate.expr, &candidate.read_plan.projected_schema)?;
         Ok(Self {
@@ -764,6 +827,9 @@ impl PrebuiltRowFilterCandidate {
             required_bytes: candidate.required_bytes,
             gate: None,
             saving: None,
+            position,
+            source_expr,
+            placement_stats: None,
         })
     }
 }
@@ -771,13 +837,13 @@ impl PrebuiltRowFilterCandidate {
 /// Returns the candidates in evaluation order: the order of `prebuilt`, or
 /// ordered by `required_bytes` if `reorder_predicates` is true. Gated
 /// optional candidates always come last (see [`row_filter_from_prebuilt`]).
-fn order_candidates(
-    prebuilt: &[PrebuiltRowFilterCandidate],
+fn order_candidates<'a>(
+    prebuilt: impl IntoIterator<Item = &'a PrebuiltRowFilterCandidate>,
     reorder_predicates: bool,
-) -> Vec<&PrebuiltRowFilterCandidate> {
+) -> Vec<&'a PrebuiltRowFilterCandidate> {
     // Collect references into a working list we can sort without disturbing
     // the shared cache.
-    let mut ordered: Vec<&PrebuiltRowFilterCandidate> = prebuilt.iter().collect();
+    let mut ordered: Vec<&PrebuiltRowFilterCandidate> = prebuilt.into_iter().collect();
     if reorder_predicates {
         ordered.sort_unstable_by_key(|c| (c.is_optional(), c.required_bytes));
     } else {
@@ -799,8 +865,8 @@ fn order_candidates(
 /// the optional candidates see only the rows that the required candidates
 /// let pass: the gate then measures the selectivity that the optional filter
 /// adds, and a skipped optional filter costs nothing.
-pub(crate) fn row_filter_from_prebuilt(
-    prebuilt: &[PrebuiltRowFilterCandidate],
+pub(crate) fn row_filter_from_prebuilt<'a>(
+    prebuilt: impl IntoIterator<Item = &'a PrebuiltRowFilterCandidate>,
     reorder_predicates: bool,
     file_metrics: &ParquetFileMetrics,
 ) -> RowFilter {
@@ -828,6 +894,7 @@ pub(crate) fn row_filter_from_prebuilt(
                 rows_matched: predicate_rows_matched,
                 time: time.clone(),
                 gate: candidate.gate.clone(),
+                placement_stats: candidate.placement_stats.clone(),
             }) as Box<dyn ArrowPredicate>
         })
         .collect();
@@ -3007,7 +3074,8 @@ mod optional_filter_tests {
         let rewritten = simplifier
             .simplify(rewriter.rewrite(Arc::clone(&predicate)).unwrap())
             .unwrap();
-        let (required, optional_filters) = split_optional(&rewritten);
+        let (required, optional_filters) =
+            datafusion_physical_expr::utils::split_optional(&rewritten);
         assert_eq!(required.len(), 1);
         assert_eq!(optional_filters.len(), 1);
 
@@ -3059,4 +3127,85 @@ mod optional_filter_tests {
             vec![Some(false), Some(false), Some(true)]
         );
     }
+
+    /// The adaptive filter placement of a file with a required conjunct and
+    /// a gated optional conjunct. The gate decisions use the durations
+    /// passed to `record`, thus the test is deterministic.
+    #[test]
+    fn file_placement_follows_gate_and_measurements() {
+        use crate::filter_placement::{FilePlacement, Placement, PlacementOptions};
+        use arrow::array::BooleanArray;
+        use datafusion_physical_plan::metrics::Count;
+
+        let (_file, metadata, file_schema) = test_file();
+        // a > 5 AND Optional(b >= 0). The optional conjunct removes no row.
+        let predicate = conjunction([
+            col_op("a", Operator::Gt, 5, &file_schema),
+            optional(col_op("b", Operator::GtEq, 0, &file_schema)),
+        ]);
+        let options = options(OptionalFilterMode::Adaptive);
+        let metrics = ExecutionPlanMetricsSet::new();
+        let mut candidates =
+            prebuild(&predicate, &file_schema, &metadata, &options, &metrics);
+        let placement_options =
+            PlacementOptions::new(true, Arc::default(), Some(&predicate));
+        let output = ProjectionMask::all();
+        let batch_size = 100;
+        let changes = Count::new();
+        let mut placement = FilePlacement::try_new(
+            &mut candidates,
+            &predicate,
+            &placement_options,
+            &output,
+            &metadata,
+            batch_size,
+            1000,
+            changes.clone(),
+        )
+        .unwrap();
+        let required = candidates.iter().position(|c| !c.is_optional()).unwrap();
+        let gated = candidates.iter().position(|c| c.is_optional()).unwrap();
+        // Initial rule: the required conjunct does not read `b` and `c`,
+        // thus it starts as a row filter. The gate is not paused.
+        assert_eq!(placement.placements(), vec![Placement::RowFilter; 2]);
+
+        // The gate pauses the optional filter: it removed no row in its
+        // first window (`sample_batches` = 2).
+        let gate = Arc::clone(candidates[gated].gate().unwrap());
+        for _ in 0..2 {
+            let mut gate = gate.lock();
+            assert_eq!(gate.begin_batch(batch_size), GateDecision::Evaluate);
+            gate.record(batch_size, batch_size, Duration::from_micros(1));
+        }
+        assert!(gate.lock().is_paused());
+        assert!(placement.decide(1000));
+        assert!(!placement.in_row_filter(gated));
+        assert!(placement.in_row_filter(required));
+        assert_eq!(changes.value(), 1);
+
+        // The pause (`initial_pause_batches` = 4) ends while the scan skips
+        // the next row group (10 batches): the filter is a row filter again.
+        assert!(placement.decide(1000));
+        assert!(placement.in_row_filter(gated));
+        assert!(!gate.lock().is_paused());
+
+        // The required conjunct removes every second row: no run of removed
+        // rows that the decoder can skip, thus it moves to the post-scan
+        // filter.
+        let scattered: BooleanArray = (0..20_000).map(|i| Some(i % 2 == 0)).collect();
+        let stats = candidates[required].placement_stats.clone().unwrap();
+        stats.record_evaluation(&scattered);
+        assert!(placement.decide(1000));
+        assert!(!placement.in_row_filter(required));
+        let post_scan = placement.post_scan_conjuncts();
+        assert_eq!(post_scan.len(), 1);
+        assert_eq!(post_scan[0].expr.to_string(), "a@0 > 5");
+
+        // A frozen placement does not change.
+        let before = placement.placements();
+        placement.restore_and_freeze(&[Placement::RowFilter, Placement::RowFilter]);
+        assert!(!placement.decide(1000));
+        assert_ne!(placement.placements(), before);
+        assert_eq!(changes.value(), 3);
+    }
 }
diff --git a/datafusion/datasource-parquet/src/source.rs b/datafusion/datasource-parquet/src/source.rs
index ec853d525b8b3..5bb782debfff4 100644
--- a/datafusion/datasource-parquet/src/source.rs
+++ b/datafusion/datasource-parquet/src/source.rs
@@ -23,6 +23,7 @@ use std::sync::Arc;
 use crate::DefaultParquetFileReaderFactory;
 use crate::ParquetFileReaderFactory;
 use crate::ParquetFileSchemaProvider;
+use crate::filter_placement::{PlacementOptions, PlacementSites};
 use crate::opener::ParquetMorselizer;
 use crate::opener::build_pruning_predicates;
 use crate::opener::build_virtual_columns_state;
@@ -343,6 +344,13 @@ pub struct ParquetSource {
     /// partitions and files of this scan. The gates of the optional filters
     /// use it. Replaced each time the predicate changes.
     optional_filter_decode_cost: Arc<DecodeCost>,
+    /// `datafusion.execution.adaptive_filter_placement`, see
+    /// [`crate::filter_placement`].
+    adaptive_filter_placement: bool,
+    /// Pooled measurements of the adaptive filter placement, shared by all
+    /// partitions and files of this scan. Replaced each time the predicate
+    /// changes.
+    filter_placement_sites: Arc<PlacementSites>,
 }
 
 impl ParquetSource {
@@ -374,6 +382,8 @@ impl ParquetSource {
             optional_filter_mode: OptionalFilterMode::default(),
             optional_filter_gate_config: OptionalFilterGateConfig::default(),
             optional_filter_decode_cost: Arc::default(),
+            adaptive_filter_placement: false,
+            filter_placement_sites: Arc::default(),
         }
     }
 
@@ -407,6 +417,7 @@ impl ParquetSource {
         let mut conf = self.clone();
         conf.predicate = Some(Arc::clone(&predicate));
         conf.optional_filter_decode_cost = Arc::default();
+        conf.filter_placement_sites = Arc::default();
         conf
     }
 
@@ -710,6 +721,11 @@ impl FileSource for ParquetSource {
                 gate_config: self.optional_filter_gate_config,
                 decode_cost: Arc::clone(&self.optional_filter_decode_cost),
             },
+            filter_placement: PlacementOptions::new(
+                self.adaptive_filter_placement && self.pushdown_filters(),
+                Arc::clone(&self.filter_placement_sites),
+                self.predicate.as_ref(),
+            ),
         }))
     }
 
@@ -942,6 +958,8 @@ impl FileSource for ParquetSource {
         source.optional_filter_mode = config.execution.optional_filter_mode;
         source.optional_filter_gate_config = (&config.execution).into();
         source.optional_filter_decode_cost = Arc::default();
+        source.adaptive_filter_placement = config.execution.adaptive_filter_placement;
+        source.filter_placement_sites = Arc::default();
         let source = Arc::new(source);
         // The parquet scan always accepts pushable filters: report each
         // pushable filter as accepted (`Yes`) so the parent `FilterExec` is
@@ -1210,6 +1228,10 @@ impl FileSource for ParquetSource {
             optional_filter_gate_config: _,
             // Runtime state, recreated when the source is decoded.
             optional_filter_decode_cost: _,
+            // Not serialized: the decoded source does not use adaptive
+            // filter placement.
+            adaptive_filter_placement: _,
+            filter_placement_sites: _,
         } = self;
 
         if schema_provider.is_some() {
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt b/datafusion/sqllogictest/test_files/information_schema.slt
index 3f6ea66b6f1d0..2da3ee55afbf5 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -213,6 +213,7 @@ datafusion.catalog.has_header true
 datafusion.catalog.information_schema true
 datafusion.catalog.location NULL
 datafusion.catalog.newlines_in_values false
+datafusion.execution.adaptive_filter_placement false
 datafusion.execution.batch_size 8192
 datafusion.execution.coalesce_batches true
 datafusion.execution.collect_statistics true
@@ -376,6 +377,7 @@ datafusion.catalog.has_header true Default value for `format.has_header` for `CR
 datafusion.catalog.information_schema true Should DataFusion provide access to `information_schema` virtual tables for displaying schema information
 datafusion.catalog.location NULL Location scanned to load tables for `default` schema
 datafusion.catalog.newlines_in_values false Specifies whether newlines in (quoted) CSV values are supported. This is the default value for `format.newlines_in_values` for `CREATE EXTERNAL TABLE` if not specified explicitly in the statement. Parsing newlines in quoted values may be affected by execution behaviour such as parallel file scanning. Setting this to `true` ensures that newlines in values are parsed successfully, which may reduce performance.
+datafusion.execution.adaptive_filter_placement false When true and `datafusion.execution.parquet.pushdown_filters` is true, the Parquet scan decides for each filter conjunct if it is a row filter (late materialization) or a filter on the decoded batches. The scan measures the rows that each conjunct removes, the decode time and the fetch latency, and it changes the placement at row group boundaries. When `datafusion.execution.optional_filter_mode` is `adaptive`, an optional filter that is paused is also removed from the row filter, thus its columns are not decoded.
 datafusion.execution.batch_size 8192 Default batch size while creating new batches, it's especially useful for buffer-in-memory batches since creating tiny batches would result in too much metadata memory consumption
 datafusion.execution.coalesce_batches true When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting
 datafusion.execution.collect_statistics true Should DataFusion collect statistics when first creating a table. Has no effect after the table is created. Defaults to true.
diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md
index 3c87a2888e86f..f6d4fdf58dd0d 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -147,6 +147,7 @@ The following configuration settings are available:
 | datafusion.execution.hash_join_buffering_capacity                       | 0                         | How many bytes to buffer in the probe side of hash joins while the build side is concurrently being built. Without this, hash joins will wait until the full materialization of the build side before polling the probe side. This is useful in scenarios where the query is not completely CPU bounded, allowing to do some early work concurrently and reducing the latency of the query. Note that when hash join buffering is enabled, the probe side will start eagerly polling data, not giving time for the producer side of dynamic filters to produce any meaningful predicate. Queries with dynamic filters might see performance degradation. Disabled by default, set to a number greater than 0 for enabling it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | datafusion.execution.optional_filter_mode                               | always                    | Controls how DataFusion evaluates filters that are not needed for correctness, such as the dynamic filters that hash joins and TopK push down into scans. `always` evaluates these filters like any other pushed-down filter. `adaptive` pauses these filters when they cost more than they save (see `datafusion.execution.optional_filter_min_saving_ns_per_row`) or when they remove no rows, and tries them again at intervals. `pruning_only` uses these filters only to prune files, row groups and pages with statistics, and never evaluates them row by row. This option is most important when `datafusion.execution.parquet.pushdown_filters` is true, because then the Parquet reader evaluates pushed-down filters row by row.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | datafusion.execution.optional_filter_min_saving_ns_per_row              | 20                        | The assumed work, in nanoseconds, that each row removed by an optional filter saves downstream. Optional filters are filters that are not needed for correctness, such as the dynamic filters that hash joins and TopK push down into scans. When an operator evaluates optional filters adaptively, it pauses an optional filter whose evaluation costs more than the work that it saves. Consumers that can measure the saving (the Parquet scan) add their measured decode cost. The default is about the cost of a hash table probe for one row. The best value depends on the hardware.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+| datafusion.execution.adaptive_filter_placement                          | false                     | When true and `datafusion.execution.parquet.pushdown_filters` is true, the Parquet scan decides for each filter conjunct if it is a row filter (late materialization) or a filter on the decoded batches. The scan measures the rows that each conjunct removes, the decode time and the fetch latency, and it changes the placement at row group boundaries. When `datafusion.execution.optional_filter_mode` is `adaptive`, an optional filter that is paused is also removed from the row filter, thus its columns are not decoded.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
 | datafusion.optimizer.enable_distinct_aggregation_soft_limit             | true                      | When set to true, the optimizer will push a limit operation into grouped aggregations which have no aggregate expressions, as a soft limit, emitting groups once the limit is reached, before all rows in the group are read.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | datafusion.optimizer.enable_round_robin_repartition                     | true                      | When set to true, the physical plan optimizer will try to add round robin repartitioning to increase parallelism to leverage more CPU cores                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | datafusion.optimizer.enable_topk_aggregation                            | true                      | When set to true, the optimizer will attempt to perform limit operations during aggregations, if possible                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

From 1fbda42ee03370ab597f9cc54f79c041d17061e0 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 15:04:17 -0500
Subject: [PATCH 18/42] test: sqllogictests for adaptive filter placement

The same queries give the same results with and without `datafusion.execution.adaptive_filter_placement`: a scattered filter (moves to the post-scan filter), a clustered filter (stays a row filter), a mix of both with a string filter, a `LIMIT` with `ORDER BY` and a `LIMIT` without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../parquet_adaptive_filter_placement.slt     | 118 ++++++++++++++++++
 1 file changed, 118 insertions(+)
 create mode 100644 datafusion/sqllogictest/test_files/parquet_adaptive_filter_placement.slt

diff --git a/datafusion/sqllogictest/test_files/parquet_adaptive_filter_placement.slt b/datafusion/sqllogictest/test_files/parquet_adaptive_filter_placement.slt
new file mode 100644
index 0000000000000..41d4463f0d87a
--- /dev/null
+++ b/datafusion/sqllogictest/test_files/parquet_adaptive_filter_placement.slt
@@ -0,0 +1,118 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Tests for `datafusion.execution.adaptive_filter_placement`: the Parquet scan
+# places each filter conjunct in the row filter or in the post-scan filter,
+# and changes the placement at row group boundaries. The results must be the
+# same as without adaptive placement.
+
+statement ok
+set datafusion.execution.parquet.max_row_group_size = 1024;
+
+# `a` is scattered (every second row), `b` is clustered (increases every 100
+# rows), `s` is a wide string column that the filters do not read.
+query I
+COPY (
+  SELECT i % 2 AS a, i / 100 AS b, repeat('x', 50) || i AS s, i AS v
+  FROM generate_series(0, 9999) AS t(i)
+) TO 'test_files/scratch/parquet_adaptive_filter_placement/t.parquet'
+STORED AS PARQUET;
+----
+10000
+
+statement ok
+reset datafusion.execution.parquet.max_row_group_size;
+
+statement ok
+CREATE EXTERNAL TABLE t STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_adaptive_filter_placement/t.parquet';
+
+statement ok
+set datafusion.execution.parquet.pushdown_filters = true;
+
+statement ok
+set datafusion.execution.batch_size = 256;
+
+statement ok
+set datafusion.execution.adaptive_filter_placement = true;
+
+query IIII
+SELECT count(*), min(v), max(v), sum(v) FROM t WHERE a = 0;
+----
+5000 0 9998 24995000
+
+query IIII
+SELECT count(*), min(v), max(v), sum(v) FROM t WHERE b < 5 OR b BETWEEN 50 AND 54;
+----
+1000 0 5499 2749500
+
+query IIII
+SELECT count(*), min(v), max(v), sum(v) FROM t WHERE a = 0 AND b >= 90 AND s LIKE '%8';
+----
+100 9008 9998 950300
+
+query IT
+SELECT v, s FROM t WHERE a = 1 AND b = 42 ORDER BY v LIMIT 3;
+----
+4201 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx4201
+4203 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx4203
+4205 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx4205
+
+# A limit without an order: the placement can move `a = 0` to the post-scan
+# filter in the middle of the file.
+query I
+SELECT count(*) FROM (SELECT v FROM t WHERE a = 0 LIMIT 3000);
+----
+3000
+
+# The same queries without adaptive placement give the same results.
+statement ok
+set datafusion.execution.adaptive_filter_placement = false;
+
+query IIII
+SELECT count(*), min(v), max(v), sum(v) FROM t WHERE a = 0;
+----
+5000 0 9998 24995000
+
+query IIII
+SELECT count(*), min(v), max(v), sum(v) FROM t WHERE b < 5 OR b BETWEEN 50 AND 54;
+----
+1000 0 5499 2749500
+
+query IIII
+SELECT count(*), min(v), max(v), sum(v) FROM t WHERE a = 0 AND b >= 90 AND s LIKE '%8';
+----
+100 9008 9998 950300
+
+query IT
+SELECT v, s FROM t WHERE a = 1 AND b = 42 ORDER BY v LIMIT 3;
+----
+4201 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx4201
+4203 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx4203
+4205 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx4205
+
+statement ok
+reset datafusion.execution.batch_size;
+
+statement ok
+reset datafusion.execution.parquet.pushdown_filters;
+
+statement ok
+reset datafusion.execution.adaptive_filter_placement;
+
+statement ok
+DROP TABLE t;

From b0088dc1ba47bbf4a69a51f5824dea2f73226835 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 23:54:01 -0500
Subject: [PATCH 19/42] feat(parquet): carry row selections across placement
 rebuilds

The opener froze the adaptive filter placement of a file when a row
selection was live (#24355). Page index pruning installs a selection on
every row group, also a select-all one, thus the placement of a file
with a page index never changed (`filter_placement_changes=0` on TPC-H).

#24355 was about runtime row group pruning, which drops row groups. The
placement rebuild only changes the `RowFilter` and the projection mask:
`ParquetPushDecoder::into_builder` returns the remaining row group plan
with the selection of each row group that is not read yet, and the
rebuild does not replace it. Thus the placement can change with a live
selection. Remove the freeze.

The freeze for a change of the decoded batch schema (nested columns)
stays: the batch coalescer holds batches with the current schema.

The new opener test uses a page index selection that skips rows in the
first three row groups. It fails with the freeze (no placement change)
and when the rebuild drops the remaining selections (the scan fetches
the pruned pages).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/filter_placement/mod.rs               |  15 +-
 .../datasource-parquet/src/opener/mod.rs      | 146 ++++++++++++++++--
 .../datasource-parquet/src/push_decoder.rs    |  10 +-
 3 files changed, 147 insertions(+), 24 deletions(-)

diff --git a/datafusion/datasource-parquet/src/filter_placement/mod.rs b/datafusion/datasource-parquet/src/filter_placement/mod.rs
index 8cb023fa6b932..b85ec9fcd368b 100644
--- a/datafusion/datasource-parquet/src/filter_placement/mod.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/mod.rs
@@ -35,8 +35,14 @@
 //! [`FilePlacement`] decides at file open and again at each row group
 //! boundary. The stream then rebuilds the decoder
 //! (`ParquetPushDecoder::into_builder`) with the new `RowFilter` and
-//! projection mask, and rebuilds the post-scan filter. The measurements are
-//! pooled over all files and partitions of the scan ([`PlacementSites`]).
+//! projection mask, and rebuilds the post-scan filter. The rebuilt decoder
+//! keeps the row selections of the remaining row groups (for example from
+//! the page index). The measurements are pooled over all files and
+//! partitions of the scan ([`PlacementSites`]).
+//!
+//! The placement of a file stops changing only when a change would change
+//! the schema of the decoded batches (possible with nested columns): the
+//! batch coalescer holds batches with the current schema.
 //!
 //! Conjuncts that `build_row_filter` rejects for a file always run in the
 //! post-scan filter (required) or are not used (optional). Optional
@@ -288,11 +294,6 @@ impl FilePlacement {
         self.frozen = true;
     }
 
-    /// Stops all later changes.
-    pub(crate) fn freeze(&mut self) {
-        self.frozen = true;
-    }
-
     /// True if the candidate at `index` is a predicate of the `RowFilter`.
     pub(crate) fn in_row_filter(&self, index: usize) -> bool {
         self.conjuncts
diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index 3d8ffc5fb60db..afe31bfb8396f 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -1927,17 +1927,11 @@ impl RowGroupsPrunedParquetOpen {
             // installed filter is owned by the decoder and is not
             // recoverable once replaced.
             let first_rg_fully_matched = rg_plan.front().is_some_and(|e| e.fully_matched);
-            let mut row_filter_context = precomputed_context;
-            // A rebuild at a row group boundary cannot keep a live row
-            // selection (#24355), thus the adaptive filter placement keeps
-            // its first decision for this file.
-            if has_row_selection
-                && let Some(placement) = row_filter_context
-                    .as_mut()
-                    .and_then(|ctx| ctx.placement.as_mut())
-            {
-                placement.freeze();
-            }
+            // The adaptive filter placement can change at a row group
+            // boundary also when a row selection is live: the rebuild keeps
+            // the selections of the remaining row groups (see
+            // `PushDecoderStreamState::rebuild_decoder_at_boundary`).
+            let row_filter_context = precomputed_context;
 
             let mut builder = decoder_config.build(selections, reader_metadata.clone());
             let mut filter_installed = false;
@@ -5704,6 +5698,16 @@ mod test {
         async fn write_file(
             store: &Arc<dyn ObjectStore>,
             a: impl Fn(usize) -> i32,
+        ) -> (SchemaRef, PartitionedFile) {
+            write_file_with_pages(store, a, None).await
+        }
+
+        /// Like [`write_file`], with at most `page_rows` rows in each data
+        /// page if it is `Some`.
+        async fn write_file_with_pages(
+            store: &Arc<dyn ObjectStore>,
+            a: impl Fn(usize) -> i32,
+            page_rows: Option<usize>,
         ) -> (SchemaRef, PartitionedFile) {
             let schema = Arc::new(Schema::new(vec![
                 Field::new("a", DataType::Int32, false),
@@ -5717,9 +5721,14 @@ mod test {
                 ],
             )
             .unwrap();
-            let props = WriterProperties::builder()
-                .set_max_row_group_row_count(Some(ROWS_PER_ROW_GROUP))
-                .build();
+            let mut props = WriterProperties::builder()
+                .set_max_row_group_row_count(Some(ROWS_PER_ROW_GROUP));
+            if let Some(page_rows) = page_rows {
+                props = props
+                    .set_data_page_row_count_limit(page_rows)
+                    .set_write_batch_size(page_rows);
+            }
+            let props = props.build();
             let size = write_parquet_batches(
                 Arc::clone(store),
                 "placement.parquet",
@@ -5822,6 +5831,115 @@ mod test {
             assert_eq!(post_scan.post_scan_rows(), TOTAL_ROWS);
         }
 
+        /// The values of `v` that a scan with the page index returns, and
+        /// its metrics.
+        async fn scan_values_with_page_index(
+            store: &Arc<dyn ObjectStore>,
+            schema: &SchemaRef,
+            file: &PartitionedFile,
+            predicate: &Arc<dyn PhysicalExpr>,
+            pushdown_filters: bool,
+            filter_placement: bool,
+        ) -> (Vec<i64>, ExecutionPlanMetricsSet) {
+            let metrics = ExecutionPlanMetricsSet::new();
+            let morselizer = ParquetMorselizerBuilder::new()
+                .with_store(Arc::clone(store))
+                .with_schema(Arc::clone(schema))
+                .with_predicate(Arc::clone(predicate))
+                .with_pushdown_filters(pushdown_filters)
+                .with_filter_placement(filter_placement)
+                .with_enable_page_index(true)
+                .with_metrics(metrics.clone())
+                .build();
+            let stream = open_file(&morselizer, file.clone()).await.unwrap();
+            let batches = stream.try_collect::<Vec<_>>().await.unwrap();
+            let values = batches
+                .iter()
+                .flat_map(|batch| {
+                    batch
+                        .column(1)
+                        .as_any()
+                        .downcast_ref::<Int64Array>()
+                        .unwrap()
+                        .values()
+                        .to_vec()
+                })
+                .collect();
+            (values, metrics)
+        }
+
+        /// The page index skips the first pages of the first row group
+        /// (`v >= 1000`) and pages of the second and third row groups
+        /// (`v < 20000 OR v >= 40000`), thus the scan has a live row
+        /// selection. The placement still changes at the first row group
+        /// boundary, the rebuilt decoder keeps the selections of the
+        /// remaining row groups, and the result is the same as with a row
+        /// filter only and with a post-scan filter only.
+        #[tokio::test]
+        async fn placement_changes_with_page_index_selection() {
+            use datafusion_physical_plan::metrics::MetricValue;
+
+            let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+            let (schema, file) =
+                write_file_with_pages(&store, |i| (i % 2) as i32, Some(100)).await;
+            let predicate = logical2physical(
+                &col("a").eq(lit(0)).and(col("v").gt_eq(lit(1000_i64))).and(
+                    col("v")
+                        .lt(lit(20_000_i64))
+                        .or(col("v").gt_eq(lit(40_000_i64))),
+                ),
+                &schema,
+            );
+
+            let (adaptive, adaptive_metrics) = scan_values_with_page_index(
+                &store, &schema, &file, &predicate, true, true,
+            )
+            .await;
+            let (row_filter, row_filter_metrics) = scan_values_with_page_index(
+                &store, &schema, &file, &predicate, true, false,
+            )
+            .await;
+            let (post_scan, post_scan_metrics) = scan_values_with_page_index(
+                &store, &schema, &file, &predicate, false, false,
+            )
+            .await;
+
+            let expected: Vec<i64> = (1000..20_000)
+                .chain(40_000..TOTAL_ROWS as i64)
+                .step_by(2)
+                .collect();
+            assert_eq!(adaptive, expected);
+            assert_eq!(row_filter, expected);
+            assert_eq!(post_scan, expected);
+
+            // The page index skipped rows: the selection is not select-all.
+            let MetricValue::PruningMetrics {
+                pruning_metrics, ..
+            } = adaptive_metrics
+                .clone_inner()
+                .sum_by_name("page_index_rows_pruned")
+                .unwrap()
+            else {
+                panic!("expected page pruning metrics");
+            };
+            // Whole pages only: a page that has a row in the result is kept.
+            assert!(pruning_metrics.pruned() > 20_000);
+            assert!(
+                counter_metric_value(&adaptive_metrics, "filter_placement_changes") > 0
+            );
+            // The rebuilt decoder fetches only the pages that the selections
+            // keep, as the scans that do not change the placement. A rebuild
+            // that dropped the selections of the remaining row groups fetches
+            // the pruned pages of the second and third row groups.
+            let bytes = |metrics: &ExecutionPlanMetricsSet| {
+                counter_metric_value(metrics, "bytes_scanned")
+            };
+            assert!(
+                bytes(&adaptive_metrics)
+                    <= bytes(&row_filter_metrics).max(bytes(&post_scan_metrics))
+            );
+        }
+
         /// A filter that removes long runs of rows saves decode time in a row
         /// filter, thus it stays a row filter.
         #[tokio::test]
diff --git a/datafusion/datasource-parquet/src/push_decoder.rs b/datafusion/datasource-parquet/src/push_decoder.rs
index 7a520ed017f4d..c46d05d04c306 100644
--- a/datafusion/datasource-parquet/src/push_decoder.rs
+++ b/datafusion/datasource-parquet/src/push_decoder.rs
@@ -902,9 +902,13 @@ impl PushDecoderStreamState {
 
         let decoder = self.decoder.take().expect("decoder present");
         let mut builder = decoder.into_builder().map_err(DataFusionError::from)?;
-        // Filter-only rebuilds preserve the decoder's remaining selections.
-        // Runtime pruning is disabled for scans with selections, so pruned
-        // plans can be rebuilt from row-group indexes alone.
+        // `into_builder` returns the decoder's remaining row group plan with
+        // the selection of each row group that is not read yet. Thus a
+        // rebuild that changes only the filter or the projection (the
+        // fully-matched toggle, the adaptive filter placement) keeps a live
+        // row selection (for example from the page index). Runtime pruning
+        // is disabled for scans with selections, so pruned plans can be
+        // rebuilt from row-group indexes alone.
         if pruned_count > 0 {
             let selections = self
                 .rg_plan

From 98259e56263e05be6f6c1fdcd4a4f0d6a3519ab3 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Thu, 24 Sep 2026 23:56:22 -0500
Subject: [PATCH 20/42] feat(parquet): measure skippable rows on the stage
 input

The adaptive filter placement uses the skippable rows of a conjunct:
the rows in windows of 64 rows where no row passes. The post-scan filter
counted them on the working batch. After a compaction, the working
batch holds only the live rows, thus a window of 64 working rows is not
a window of the decoded rows, and a conjunct after the compaction was
measured with distorted windows.

A conjunct that moves to the row filter runs after the row filter
predicates and before all post-scan conjuncts. Thus its input is the
input batch of the post-scan filter, not the rows that the earlier
post-scan conjuncts let pass. `StageSelection` maps the working rows to
the rows of the input batch and measures each conjunct there:

* Before the first compaction, the conjunct was evaluated on all rows:
  the measurement is exact (also for a conjunct after other conjuncts).
* After a compaction, the rows that the compaction removed count as
  passing. This counts too few skippable rows, a conservative
  approximation: the conjunct moves to the row filter only if the
  measured saving pays for it.

The marginal rows (skippable after the conjunct minus skippable before
it, on the live rows) are not the right measure. Tested on TPC-H Q6 and
Q12, they made the last conjunct look very selective: it moved to the
row filter, where it runs first and saves nothing, and it moved back at
the next row group (10 to 12 placement changes for each query).

Only a window whose rows are all in the working batch can be skippable,
thus `StageSelection` keeps only these windows and their first working
row. Its cost is O(rows / 64) for each measured conjunct and each
compaction, and it is allocated only when the post-scan filter has a
measured conjunct. (Mapping every row back to the input batch cost
about 36 ms of CPU on TPC-H SF1 Q6, more than the filter itself.)

The row filter cannot do the same: arrow-rs evaluates a predicate only
on the rows that the earlier predicates let pass and does not give their
positions. The first predicate is exact. For the other predicates the
windows are windows of the rows that pass, which counts too few
skippable rows. This approximation is documented on
`ConjunctStats::record_evaluation`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/decoder_projection.rs                 |  69 ++++-
 .../src/filter_placement/mod.rs               |   2 +-
 .../src/filter_placement/stats.rs             | 258 +++++++++++++++++-
 3 files changed, 318 insertions(+), 11 deletions(-)

diff --git a/datafusion/datasource-parquet/src/decoder_projection.rs b/datafusion/datasource-parquet/src/decoder_projection.rs
index a9b9309088883..714866d97d329 100644
--- a/datafusion/datasource-parquet/src/decoder_projection.rs
+++ b/datafusion/datasource-parquet/src/decoder_projection.rs
@@ -51,7 +51,7 @@ use parquet::arrow::ProjectionMask;
 use parquet::schema::types::{SchemaDescPtr, SchemaDescriptor};
 
 use crate::ParquetFileMetrics;
-use crate::filter_placement::ConjunctStats;
+use crate::filter_placement::{ConjunctStats, StageSelection};
 use crate::opener::{VirtualColumnsState, append_fields};
 use crate::projection_read_plan::build_projection_read_plan;
 
@@ -166,6 +166,13 @@ impl PostScanFilter {
         // compaction, `None` meaning "all of them are still live".
         let mut working = batch;
         let mut acc: Option<BooleanArray> = None;
+        // Maps the working rows to the rows of the input batch, for the
+        // conjuncts that the adaptive filter placement measures.
+        let mut stages = self
+            .conjuncts
+            .iter()
+            .any(|conjunct| conjunct.stats.is_some())
+            .then(|| StageSelection::new(input_rows));
 
         let last = self.conjuncts.len() - 1;
         for (
@@ -189,8 +196,8 @@ impl PostScanFilter {
                 Some(_) => prep_null_mask_filter(mask),
                 None => mask.clone(),
             };
-            if let Some(stats) = stats {
-                stats.record_evaluation(&mask);
+            if let (Some(stages), Some(stats)) = (stages.as_ref(), stats) {
+                stages.record(stats, &mask);
             }
             // An all-true conjunct leaves the accumulated selection untouched.
             if mask.true_count() == rows_in {
@@ -214,6 +221,9 @@ impl PostScanFilter {
             if position < last
                 && (alive as f64) <= COMPACTION_SELECTIVITY_THRESHOLD * rows_in as f64
             {
+                if let Some(stages) = stages.as_mut() {
+                    stages.compact(&folded);
+                }
                 working = filter_record_batch(&working, &folded)?;
                 acc = None;
             } else {
@@ -625,6 +635,59 @@ mod tests {
         }
     }
 
+    /// A conjunct after a compaction is measured in the positions of the
+    /// input batch, not of the compacted batch. The rows that the compaction
+    /// removed count as passing.
+    #[test]
+    fn measures_skippable_rows_on_input_positions() {
+        // (a % 4) = 0: keeps every fourth row, no empty window. The loop then
+        // compacts the batch to 64 rows.
+        let every_fourth: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
+            Arc::new(BinaryExpr::new(
+                Arc::new(Column::new("a", 0)),
+                Operator::Modulo,
+                Arc::new(Literal::new(ScalarValue::Int32(Some(4)))),
+            )),
+            Operator::Eq,
+            Arc::new(Literal::new(ScalarValue::Int32(Some(0)))),
+        ));
+        // b > 127: removes the first 32 rows of the compacted batch, that is
+        // the live rows of the input rows 0..128. In the compacted batch this
+        // is half of one window of 64 rows.
+        let upper_half = gt("b", 1, 127);
+        let first = Arc::new(ConjunctStats::default());
+        let second = Arc::new(ConjunctStats::default());
+        let filter = PostScanFilter {
+            conjuncts: vec![
+                PostScanConjunct {
+                    expr: every_fourth,
+                    stats: Some(Arc::clone(&first)),
+                },
+                PostScanConjunct {
+                    expr: upper_half,
+                    stats: Some(Arc::clone(&second)),
+                },
+            ],
+            rows_pruned: Count::new(),
+            rows_matched: Count::new(),
+            eval_time: Time::new(),
+        };
+        let rows = survivors(&filter, batch((0..256).map(Some).collect()));
+        assert_eq!(rows, (128..256).step_by(4).map(Some).collect::<Vec<_>>());
+
+        let first = first.observation();
+        assert_eq!(
+            (first.rows_in, first.rows_out, first.skippable_rows),
+            (256, 64, 0)
+        );
+        // The rows that the first conjunct removed pass: no empty window.
+        let second = second.observation();
+        assert_eq!(
+            (second.rows_in, second.rows_out, second.skippable_rows),
+            (256, 256 - 32, 0)
+        );
+    }
+
     #[test]
     fn compacts_between_conjuncts_without_losing_rows() {
         // 100 rows; `a > 79` keeps 20 (20% — below the compaction threshold, so
diff --git a/datafusion/datasource-parquet/src/filter_placement/mod.rs b/datafusion/datasource-parquet/src/filter_placement/mod.rs
index b85ec9fcd368b..3ed4ab15c3f8b 100644
--- a/datafusion/datasource-parquet/src/filter_placement/mod.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/mod.rs
@@ -66,7 +66,7 @@ use crate::row_filter::{PrebuiltRowFilterCandidate, SharedOptionalFilterGate};
 
 pub(crate) use model::Placement;
 use model::{RequiredConjunctInputs, place_optional, place_required};
-pub(crate) use stats::{ConjunctStats, PlacementSites};
+pub(crate) use stats::{ConjunctStats, PlacementSites, StageSelection};
 
 /// The adaptive placement settings of one scan.
 #[derive(Debug, Clone, Default)]
diff --git a/datafusion/datasource-parquet/src/filter_placement/stats.rs b/datafusion/datasource-parquet/src/filter_placement/stats.rs
index 73466b8d52593..3525afa7b4924 100644
--- a/datafusion/datasource-parquet/src/filter_placement/stats.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/stats.rs
@@ -24,6 +24,7 @@ use std::sync::atomic::{AtomicU64, Ordering};
 use std::time::Duration;
 
 use arrow::array::{Array, BooleanArray};
+use arrow::buffer::BooleanBuffer;
 use datafusion_physical_expr::PhysicalExpr;
 use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
 use datafusion_physical_expr::filter_stats::duration_nanos;
@@ -47,10 +48,15 @@ pub(crate) const SKIP_WINDOW_ROWS: usize = 64;
 /// is counted if no row in it passes.
 pub(crate) fn skippable_rows(result: &BooleanArray) -> usize {
     let values = result.values();
-    let passed = match result.nulls() {
-        Some(nulls) => values & nulls.inner(),
-        None => values.clone(),
-    };
+    match result.nulls() {
+        Some(nulls) => skippable_in(&(values & nulls.inner())),
+        None => skippable_in(values),
+    }
+}
+
+/// Rows of `passed` in windows of [`SKIP_WINDOW_ROWS`] rows where no bit is
+/// set, see [`skippable_rows`].
+fn skippable_in(passed: &BooleanBuffer) -> usize {
     let chunks = passed.bit_chunks();
     let full = chunks.iter().filter(|chunk| *chunk == 0).count() * SKIP_WINDOW_ROWS;
     let remainder = if chunks.remainder_len() > 0 && chunks.remainder_bits() == 0 {
@@ -69,7 +75,8 @@ pub(crate) struct Observation {
     pub(crate) rows_in: u64,
     /// Rows that passed the conjunct.
     pub(crate) rows_out: u64,
-    /// Rows in windows where no row passed, see [`skippable_rows`].
+    /// Rows in windows where no row passed the conjunct, see
+    /// [`skippable_rows`] and [`StageSelection`].
     pub(crate) skippable_rows: u64,
     /// Row groups that the conjunct alone pruned with statistics.
     pub(crate) row_groups_pruned: u64,
@@ -103,8 +110,16 @@ pub(crate) struct ConjunctStats {
 }
 
 impl ConjunctStats {
-    /// Records one evaluation of the conjunct. `result` has one value for
-    /// each evaluated row.
+    /// Records one evaluation of the conjunct as a row filter predicate.
+    /// `result` has one value for each evaluated row.
+    ///
+    /// The decoder evaluates a row filter predicate only on the rows that
+    /// the earlier predicates let pass, thus the windows of a predicate
+    /// that is not the first one are windows of these rows, not of the
+    /// rows of the file. This is an approximation: an empty window then
+    /// spans 64 or more rows of the file, thus the skippable rows are
+    /// counted too low. The first predicate is measured exactly. See
+    /// [`StageSelection`] for the post-scan filter.
     pub(crate) fn record_evaluation(&self, result: &BooleanArray) {
         let rows_in = result.len() as u64;
         if rows_in == 0 {
@@ -124,6 +139,18 @@ impl ConjunctStats {
         self.skippable_rows.fetch_add(skippable, Ordering::Relaxed);
     }
 
+    /// Records one evaluation of the conjunct on `rows_in` rows, of which
+    /// `rows_out` passed and which made `skippable` rows skippable.
+    pub(crate) fn record(&self, rows_in: usize, rows_out: usize, skippable: usize) {
+        if rows_in == 0 {
+            return;
+        }
+        self.rows_in.fetch_add(rows_in as u64, Ordering::Relaxed);
+        self.rows_out.fetch_add(rows_out as u64, Ordering::Relaxed);
+        self.skippable_rows
+            .fetch_add(skippable as u64, Ordering::Relaxed);
+    }
+
     /// The pooled values. The values are not read at the same instant.
     pub(crate) fn observation(&self) -> Observation {
         Observation {
@@ -136,6 +163,103 @@ impl ConjunctStats {
     }
 }
 
+/// Measures the conjuncts of one batch of the post-scan filter in the
+/// positions of its input batch.
+///
+/// The post-scan filter evaluates its conjuncts one after the other and
+/// compacts the working batch to the rows that are still live (see
+/// `PostScanFilter`). Windows of the compacted rows are not windows of the
+/// decoded rows, thus a conjunct after a compaction must not count its
+/// skippable rows on its own result.
+///
+/// A conjunct that moves to the row filter runs after the row filter
+/// predicates and before all post-scan conjuncts: its input is the input
+/// batch of the post-scan filter, not the rows that the earlier post-scan
+/// conjuncts let pass. Thus each conjunct is measured on the input batch.
+/// Before the first compaction it was evaluated on all rows of the input
+/// batch and the measurement is exact. After a compaction, it was not
+/// evaluated on the rows that the compaction removed; these rows count as
+/// passing. This counts too few skippable rows (a conservative
+/// approximation: the conjunct is a row filter only if its measured saving
+/// pays for it). The first conjunct is always exact.
+///
+/// Only a window whose rows are all in the working batch can be skippable.
+/// Thus this keeps these windows and their first row in the working batch,
+/// and the cost is O(rows / 64) for each conjunct and compaction.
+#[derive(Debug)]
+pub(crate) struct StageSelection {
+    input_rows: usize,
+    /// The windows of the input batch whose rows are all in the working
+    /// batch, as (rows of the window, first row in the working batch).
+    /// `None` if the working batch is the input batch (not compacted).
+    complete_windows: Option<Vec<(usize, usize)>>,
+}
+
+impl StageSelection {
+    /// For an input batch of `input_rows` rows.
+    pub(crate) fn new(input_rows: usize) -> Self {
+        Self {
+            input_rows,
+            complete_windows: None,
+        }
+    }
+
+    /// Records in `stats` the evaluation of a conjunct with the result
+    /// `passed` (without nulls) for each row of the working batch.
+    pub(crate) fn record(&self, stats: &ConjunctStats, passed: &BooleanArray) {
+        let values = passed.values();
+        match &self.complete_windows {
+            None => stats.record(
+                self.input_rows,
+                values.count_set_bits(),
+                skippable_in(values),
+            ),
+            Some(windows) => {
+                // The rows that the conjunct did not see pass.
+                let unseen = self.input_rows - values.len();
+                let skippable = windows
+                    .iter()
+                    .filter(|(len, start)| {
+                        values.slice(*start, *len).count_set_bits() == 0
+                    })
+                    .map(|(len, _)| len)
+                    .sum();
+                stats.record(
+                    self.input_rows,
+                    unseen + values.count_set_bits(),
+                    skippable,
+                );
+            }
+        }
+    }
+
+    /// The working batch is compacted to the rows of `live` (a selection of
+    /// the rows of the working batch, without nulls).
+    pub(crate) fn compact(&mut self, live: &BooleanArray) {
+        let live = live.values();
+        let windows: Vec<(usize, usize)> = match self.complete_windows.take() {
+            Some(windows) => windows,
+            None => (0..self.input_rows)
+                .step_by(SKIP_WINDOW_ROWS)
+                .map(|start| (SKIP_WINDOW_ROWS.min(self.input_rows - start), start))
+                .collect(),
+        };
+        // `rank` is the number of live rows before `position`: the row of
+        // the new working batch at `position`.
+        let mut rank = 0;
+        let mut position = 0;
+        let mut complete = Vec::with_capacity(windows.len());
+        for (len, start) in windows {
+            rank += live.slice(position, start - position).count_set_bits();
+            position = start;
+            if live.slice(start, len).count_set_bits() == len {
+                complete.push((len, rank));
+            }
+        }
+        self.complete_windows = Some(complete);
+    }
+}
+
 /// The measured decode speed of the columns that the decoder produces, for
 /// one scan. Lock-free.
 #[derive(Debug, Default)]
@@ -312,6 +436,126 @@ mod tests {
         assert_eq!(Observation::default().pruned_fraction(), None);
     }
 
+    /// A conjunct after a compaction is measured in the positions of the
+    /// input batch, with the rows that it did not see as passing.
+    #[test]
+    fn stage_selection_measures_on_input_positions() {
+        let rows = 256;
+        let mut selection = StageSelection::new(rows);
+        // Conjunct 1 keeps every fourth row: no empty window. Exact.
+        let first: BooleanArray = (0..rows).map(|i| Some(i % 4 == 0)).collect();
+        let first_stats = ConjunctStats::default();
+        selection.record(&first_stats, &first);
+        assert_eq!(
+            first_stats.observation(),
+            Observation {
+                rows_in: 256,
+                rows_out: 64,
+                skippable_rows: 0,
+                ..Default::default()
+            }
+        );
+        // The working batch is compacted to the 64 live rows.
+        selection.compact(&first);
+        // Conjunct 2 fails the live rows of input rows 0..128 (the first 32
+        // working rows). The rows that conjunct 1 removed count as passing,
+        // thus no window is empty.
+        let second: BooleanArray = (0..64).map(|i| Some(i >= 32)).collect();
+        let second_stats = ConjunctStats::default();
+        selection.record(&second_stats, &second);
+        assert_eq!(
+            second_stats.observation(),
+            Observation {
+                rows_in: 256,
+                rows_out: 256 - 32,
+                skippable_rows: 0,
+                ..Default::default()
+            }
+        );
+    }
+
+    /// Before a compaction, a conjunct is measured exactly on all rows,
+    /// also after an earlier conjunct.
+    #[test]
+    fn stage_selection_is_exact_before_compaction() {
+        let selection = StageSelection::new(256);
+        // Fails input rows 0..128: two empty windows.
+        let passed: BooleanArray = (0..256).map(|i| Some(i >= 128)).collect();
+        let stats = ConjunctStats::default();
+        selection.record(&stats, &passed);
+        assert_eq!(
+            stats.observation(),
+            Observation {
+                rows_in: 256,
+                rows_out: 128,
+                skippable_rows: 128,
+                ..Default::default()
+            }
+        );
+    }
+
+    /// A window whose rows are all in the working batch after a compaction
+    /// is measured on the rows of the working batch.
+    #[test]
+    fn stage_selection_keeps_complete_windows() {
+        let mut selection = StageSelection::new(256);
+        // Removes input rows 0..64 and the odd rows of 64..128.
+        let first: BooleanArray = (0..256)
+            .map(|i| Some(i >= 128 || (i >= 64 && i % 2 == 0)))
+            .collect();
+        selection.compact(&first);
+        // 160 working rows: 32 rows of window 1, then windows 2 and 3
+        // (working rows 32..96 and 96..160). Fails working rows 32..96 and
+        // one row of window 3.
+        let second: BooleanArray = (0..160)
+            .map(|i| Some(!(32..96).contains(&i) && i != 100))
+            .collect();
+        let stats = ConjunctStats::default();
+        selection.record(&stats, &second);
+        assert_eq!(
+            stats.observation(),
+            Observation {
+                rows_in: 256,
+                rows_out: 256 - 65,
+                skippable_rows: 64,
+                ..Default::default()
+            }
+        );
+        // Compact again: window 2 has no live row and window 3 lost a row,
+        // thus no window is complete. Nothing is skippable any more.
+        selection.compact(&second);
+        let third: BooleanArray = (0..95).map(|i| Some(i >= 64)).collect();
+        let stats = ConjunctStats::default();
+        selection.record(&stats, &third);
+        assert_eq!(stats.observation().skippable_rows, 0);
+        assert_eq!(stats.observation().rows_out, 256 - 64);
+    }
+
+    /// Two compactions: the second maps through the first.
+    #[test]
+    fn stage_selection_composes_compactions() {
+        let mut selection = StageSelection::new(256);
+        // Keep the even rows, then (of those) the ones in the upper half.
+        let even: BooleanArray = (0..256).map(|i| Some(i % 2 == 0)).collect();
+        selection.compact(&even);
+        let upper: BooleanArray = (0..128).map(|i| Some(i >= 64)).collect();
+        selection.compact(&upper);
+        // 64 working rows: the even rows of 128..256. A conjunct that fails
+        // all of them leaves the odd rows (not seen) passing.
+        let none: BooleanArray = (0..64).map(|_| Some(false)).collect();
+        let stats = ConjunctStats::default();
+        selection.record(&stats, &none);
+        assert_eq!(
+            stats.observation(),
+            Observation {
+                rows_in: 256,
+                rows_out: 192,
+                skippable_rows: 0,
+                ..Default::default()
+            }
+        );
+    }
+
     #[test]
     fn decode_speed() {
         let decode = DecodeSpeed::default();

From ed19bdc1e4c8303179468b75b4c0c44934392090 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 00:05:57 -0500
Subject: [PATCH 21/42] feat(parquet): start filters post-scan and add a row
 filter stage cost

The adaptive filter placement started a required conjunct as a row
filter when it did not read all output columns, and it compared the
decode saving of a row filter only with the fetch latency. Profiles of
TPC-H SF1 Q6, Q12 and Q4 show that each row filter stage costs 4 to
16 ns for each row that it evaluates, independent of the predicate
(`RowSelection` building, `skip_records`, the predicate cache and the
extra fetch), about 20 times the time of the predicate. The first row
groups of each scan paid this cost before the measurements could move
the conjunct.

Changes:

* A required conjunct starts in the post-scan filter. It moves to the
  row filter only when the measurements (in either placement, at least
  8192 rows) show that the decode time it saves is more than its cost.
* The cost of a row filter stage for each row is now a fixed stage cost
  (`ROW_FILTER_STAGE_NS_PER_ROW` = 8 ns, from the profiles above), plus
  the fetch latency, plus the second decode of the output columns that
  the conjunct reads, for the rows that pass
  (`pass ratio * read output bytes * decode ns for each byte`).
* The 10% hysteresis stays.
* The E2 statistics prior now only keeps a conjunct in the post-scan
  filter: a conjunct that statistics pruned in at least half of the row
  groups is clustered, the page index already skips the pages that it
  removes, and the row groups that are left pass most rows.

Tests: the model tests use the new cost terms, the `FilePlacement` test
and the opener tests use fixed decode and fetch costs
(`PlacementSites::with_fixed_costs`) for deterministic decisions. The
opener tests now cover a scattered conjunct that stays post-scan, a
clustered conjunct that moves to the row filter at the first row group
boundary (also with a limit, and with a live page index selection), and
the statistics prior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/common/src/config.rs               |   8 +-
 .../src/filter_placement/mod.rs               |  20 +-
 .../src/filter_placement/model.rs             | 184 ++++++++++++------
 .../src/filter_placement/stats.rs             |  27 +++
 .../datasource-parquet/src/opener/mod.rs      |  88 ++++++---
 .../datasource-parquet/src/row_filter.rs      |  50 +++--
 .../test_files/information_schema.slt         |   2 +-
 .../parquet_adaptive_filter_placement.slt     |  16 +-
 docs/source/user-guide/configs.md             |   2 +-
 9 files changed, 280 insertions(+), 117 deletions(-)

diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index a861c520f1c64..4e4cec9f5f972 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -1269,9 +1269,11 @@ config_namespace! {
         /// When true and `datafusion.execution.parquet.pushdown_filters` is
         /// true, the Parquet scan decides for each filter conjunct if it is a
         /// row filter (late materialization) or a filter on the decoded
-        /// batches. The scan measures the rows that each conjunct removes, the
-        /// decode time and the fetch latency, and it changes the placement at
-        /// row group boundaries. When `datafusion.execution.optional_filter_mode`
+        /// batches. Each conjunct starts as a filter on the decoded batches.
+        /// The scan measures the rows that each conjunct removes, the decode
+        /// time and the fetch latency, and it makes a conjunct a row filter at
+        /// a row group boundary only when the decode time that the row filter
+        /// saves is more than its cost. When `datafusion.execution.optional_filter_mode`
         /// is `adaptive`, an optional filter that is paused is also removed
         /// from the row filter, thus its columns are not decoded.
         pub adaptive_filter_placement: bool, default = false
diff --git a/datafusion/datasource-parquet/src/filter_placement/mod.rs b/datafusion/datasource-parquet/src/filter_placement/mod.rs
index 3ed4ab15c3f8b..43deae1183ae3 100644
--- a/datafusion/datasource-parquet/src/filter_placement/mod.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/mod.rs
@@ -28,9 +28,12 @@
 //! | [`Placement::Skip`] | Not evaluated, and its columns are not decoded | Optional conjuncts in the `adaptive` optional filter mode, while the gate is paused |
 //!
 //! A row filter is not always better. It saves decode time only when it
-//! removes long runs of rows, and the decoder fetches the columns of each
-//! row filter predicate before the other columns (one more round trip for
-//! each row group). See [`model`] for the decision.
+//! removes long runs of rows, each row filter stage has a fixed cost for
+//! each row, and the decoder fetches the columns of each row filter
+//! predicate before the other columns (one more round trip for each row
+//! group). Thus a required conjunct starts in the post-scan filter and
+//! becomes a row filter only on measured evidence. See [`model`] for the
+//! decision.
 //!
 //! [`FilePlacement`] decides at file open and again at each row group
 //! boundary. The stream then rebuilds the decoder
@@ -116,6 +119,9 @@ enum ConjunctKind {
         /// Compressed bytes for each row of the output columns that the
         /// conjunct does not read.
         unread_output_bytes_per_row: f64,
+        /// Compressed bytes for each row of the output columns that the
+        /// conjunct reads.
+        read_output_bytes_per_row: f64,
     },
     /// An optional conjunct with a gate (the `adaptive` optional filter
     /// mode).
@@ -186,10 +192,16 @@ impl FilePlacement {
                         output_projection.leaf_included(leaf)
                             && !candidate.reads_leaf(leaf)
                     });
+                let read_output_bytes_per_row =
+                    compressed_bytes_per_row(metadata, |leaf| {
+                        output_projection.leaf_included(leaf)
+                            && candidate.reads_leaf(leaf)
+                    });
                 ConjunctKind::Required {
                     expr: Arc::clone(candidate.source_expr()),
                     stats,
                     unread_output_bytes_per_row,
+                    read_output_bytes_per_row,
                 }
             };
             conjuncts.push(ManagedConjunct {
@@ -254,11 +266,13 @@ impl FilePlacement {
                 ConjunctKind::Required {
                     stats,
                     unread_output_bytes_per_row,
+                    read_output_bytes_per_row,
                     ..
                 } => place_required(
                     &RequiredConjunctInputs {
                         observation: stats.observation(),
                         unread_output_bytes_per_row: *unread_output_bytes_per_row,
+                        read_output_bytes_per_row: *read_output_bytes_per_row,
                         decode_ns_per_byte,
                         fetch_ns_per_row,
                     },
diff --git a/datafusion/datasource-parquet/src/filter_placement/model.rs b/datafusion/datasource-parquet/src/filter_placement/model.rs
index 25c23d438f0cb..4b216b4364418 100644
--- a/datafusion/datasource-parquet/src/filter_placement/model.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/model.rs
@@ -31,14 +31,24 @@ pub(crate) enum Placement {
     Skip,
 }
 
-/// Minimum number of evaluated rows before the measurements replace the
-/// initial rule.
+/// Minimum number of evaluated rows before a conjunct can move to the row
+/// filter.
 pub(crate) const MIN_OBSERVED_ROWS: u64 = 8192;
 
 /// A conjunct that the statistics pruned in at least this fraction of the
-/// row groups starts in the post-scan filter. See [`place_required`].
+/// row groups stays in the post-scan filter. See [`place_required`].
 pub(crate) const CLUSTERED_PRUNED_FRACTION: f64 = 0.5;
 
+/// The fixed cost of a row filter stage for each row that it evaluates,
+/// in nanoseconds. It does not depend on the predicate: the decoder builds a
+/// `RowSelection` from the result, skips or reads the records of the other
+/// columns with it (`skip_records`), fills the predicate cache
+/// (`CachedArrayReader`) and fetches the columns of the stage before the
+/// other columns. Profiles of TPC-H SF1 Q6, Q12 and Q4 on an Apple M-series
+/// laptop showed 4 to 16 ns for each row and stage, about 20 times the time
+/// of the predicate. The post-scan filter has none of these costs.
+pub(crate) const ROW_FILTER_STAGE_NS_PER_ROW: f64 = 8.0;
+
 /// A row filter conjunct moves to the post-scan filter only if its cost is
 /// larger than this multiple of its benefit. The same margin as the
 /// optional filter gate.
@@ -57,6 +67,11 @@ pub(crate) struct RequiredConjunctInputs {
     /// conjunct does not read. A row filter saves their decode for the rows
     /// that it removes.
     pub(crate) unread_output_bytes_per_row: f64,
+    /// Compressed bytes for each row of the output columns that the
+    /// conjunct reads. A row filter decodes them for all rows to evaluate
+    /// the conjunct, and again for the output of the rows that pass (or
+    /// copies them from the predicate cache).
+    pub(crate) read_output_bytes_per_row: f64,
     /// Measured decode time for each compressed byte.
     pub(crate) decode_ns_per_byte: f64,
     /// The extra fetch latency of a row filter stage, for each row of the
@@ -66,8 +81,8 @@ pub(crate) struct RequiredConjunctInputs {
 }
 
 impl RequiredConjunctInputs {
-    /// Decode time, in nanoseconds for each row, that a row filter saves.
-    /// `None` without enough measurements.
+    /// Decode time, in nanoseconds for each evaluated row, that a row
+    /// filter saves. `None` without enough measurements.
     pub(crate) fn benefit_ns_per_row(&self) -> Option<f64> {
         if self.observation.rows_in < MIN_OBSERVED_ROWS {
             return None;
@@ -75,17 +90,32 @@ impl RequiredConjunctInputs {
         let skippable = self.observation.skippable_fraction()?;
         Some(skippable * self.unread_output_bytes_per_row * self.decode_ns_per_byte)
     }
+
+    /// The extra time, in nanoseconds for each evaluated row, of a row
+    /// filter stage compared to the post-scan filter: the fixed stage cost,
+    /// the fetch latency and the second decode of the output columns that
+    /// the conjunct reads, for the rows that pass.
+    pub(crate) fn cost_ns_per_row(&self) -> f64 {
+        let pass_ratio = self.observation.pass_ratio().unwrap_or(1.0);
+        ROW_FILTER_STAGE_NS_PER_ROW
+            + self.fetch_ns_per_row
+            + pass_ratio * self.read_output_bytes_per_row * self.decode_ns_per_byte
+    }
 }
 
 /// The placement of a required conjunct: [`Placement::RowFilter`] or
 /// [`Placement::PostScan`].
 ///
-/// With enough measurements, the conjunct is a row filter if the decode
-/// time that it saves is larger than the extra fetch latency:
+/// A conjunct starts in the post-scan filter. It moves to the row filter
+/// only on measured evidence: after [`MIN_OBSERVED_ROWS`] evaluated rows
+/// (in either placement), if the decode time that it saves is larger than
+/// the extra cost of a row filter stage:
 ///
 /// ```text
-/// benefit = skippable fraction * unread output bytes * decode ns for each byte
-/// cost    = fetch latency / rows of the next row group
+/// benefit = skippable rows / rows in * unread output bytes * decode ns for each byte
+/// cost    = fixed stage cost
+///         + fetch latency / rows of the next row group
+///         + rows out / rows in * read output bytes * decode ns for each byte
 /// ```
 ///
 /// The time to evaluate the conjunct is the same in both placements, thus
@@ -94,37 +124,31 @@ impl RequiredConjunctInputs {
 /// a conjunct does not move at each row group when the benefit and the cost
 /// are almost equal.
 ///
-/// Without enough measurements (the initial rule), the conjunct is a post-scan
-/// filter if the row group statistics pruned it in at least
-/// [`CLUSTERED_PRUNED_FRACTION`] of the row groups (the data is clustered on
-/// its columns, thus the row groups that are left pass most rows), or if
-/// the conjunct reads all output columns (a row filter cannot save decode
-/// time). Otherwise it is a row filter, as without adaptive placement.
+/// A conjunct that the row group statistics pruned in at least
+/// [`CLUSTERED_PRUNED_FRACTION`] of the row groups stays in the post-scan
+/// filter (the statistics prior). The data is then clustered on its
+/// columns: the row groups that are left pass most rows, and the page
+/// index already skips the pages that the conjunct removes.
 pub(crate) fn place_required(
     inputs: &RequiredConjunctInputs,
     current: Option<Placement>,
 ) -> Placement {
     let Some(benefit) = inputs.benefit_ns_per_row() else {
-        if let Some(current) = current {
-            return current;
-        }
-        let clustered = inputs
-            .observation
-            .pruned_fraction()
-            .is_some_and(|pruned| pruned >= CLUSTERED_PRUNED_FRACTION);
-        return if clustered || inputs.unread_output_bytes_per_row <= 0.0 {
-            Placement::PostScan
-        } else {
-            Placement::RowFilter
-        };
+        return current.unwrap_or(Placement::PostScan);
     };
-    let cost = inputs.fetch_ns_per_row;
+    let clustered = inputs
+        .observation
+        .pruned_fraction()
+        .is_some_and(|pruned| pruned >= CLUSTERED_PRUNED_FRACTION);
+    if clustered {
+        return Placement::PostScan;
+    }
+    let cost = inputs.cost_ns_per_row();
     let row_filter = match current {
         Some(Placement::RowFilter) => cost <= benefit * LEAVE_ROW_FILTER_MARGIN,
         _ => cost < benefit * ENTER_ROW_FILTER_MARGIN,
     };
-    // A conjunct that saves nothing is never a row filter.
-    if row_filter && benefit > 0.0 {
+    if row_filter {
         Placement::RowFilter
     } else {
         Placement::PostScan
@@ -146,6 +170,8 @@ pub(crate) fn place_optional(gate_paused: bool) -> Placement {
 mod tests {
     use super::*;
 
+    /// Inputs with a decode speed of 4 ns for each byte and no output
+    /// column that the conjunct reads.
     fn inputs(
         rows_in: u64,
         skippable_rows: u64,
@@ -163,59 +189,61 @@ mod tests {
                 row_groups_kept: kept,
             },
             unread_output_bytes_per_row,
+            read_output_bytes_per_row: 0.0,
             decode_ns_per_byte: 4.0,
             fetch_ns_per_row,
         }
     }
 
     #[test]
-    fn initial_rule() {
-        // Output columns that the conjunct does not read: row filter.
-        let wide = inputs(0, 0, 0, 0, 10.0, 0.0);
-        assert_eq!(place_required(&wide, None), Placement::RowFilter);
-        // The conjunct reads all output columns: post-scan.
-        let narrow = inputs(0, 0, 0, 0, 0.0, 0.0);
-        assert_eq!(place_required(&narrow, None), Placement::PostScan);
-        // Statistics pruned most row groups: post-scan.
-        let clustered = inputs(0, 0, 6, 4, 10.0, 0.0);
-        assert_eq!(place_required(&clustered, None), Placement::PostScan);
-        let not_clustered = inputs(0, 0, 4, 6, 10.0, 0.0);
-        assert_eq!(place_required(&not_clustered, None), Placement::RowFilter);
-        // Too few measured rows: the initial rule, and no change later.
-        let few = inputs(MIN_OBSERVED_ROWS - 1, 0, 0, 0, 10.0, 0.0);
-        assert_eq!(place_required(&few, None), Placement::RowFilter);
+    fn starts_post_scan() {
+        // Without measurements a conjunct is a post-scan filter, also with
+        // wide output columns that it does not read.
+        let wide = inputs(0, 0, 0, 0, 100.0, 0.0);
+        assert_eq!(place_required(&wide, None), Placement::PostScan);
+        let few = inputs(
+            MIN_OBSERVED_ROWS - 1,
+            MIN_OBSERVED_ROWS - 1,
+            0,
+            0,
+            100.0,
+            0.0,
+        );
+        assert_eq!(place_required(&few, None), Placement::PostScan);
+        // Too few measured rows: no change.
         assert_eq!(
-            place_required(&few, Some(Placement::PostScan)),
-            Placement::PostScan
+            place_required(&few, Some(Placement::RowFilter)),
+            Placement::RowFilter
         );
     }
 
     #[test]
     fn measured_benefit() {
         let rows = 100_000;
-        // Half of the rows are in empty windows: a row filter saves decode
-        // time, and the fetch latency is small.
+        // Half of the rows are skippable: benefit = 0.5 * 10 * 4 = 20 ns,
+        // cost = 8 + 1 = 9 ns.
         let clustered = inputs(rows, rows / 2, 0, 0, 10.0, 1.0);
         assert_eq!(place_required(&clustered, None), Placement::RowFilter);
         assert_eq!(
             place_required(&clustered, Some(Placement::PostScan)),
             Placement::RowFilter
         );
-        // No empty windows (for example a uniform 50% filter): nothing to
+        // No skippable rows (for example a uniform 50% filter): nothing to
         // save, thus post-scan. Also without any fetch latency.
         let scattered = inputs(rows, 0, 0, 0, 10.0, 0.0);
         assert_eq!(
             place_required(&scattered, Some(Placement::RowFilter)),
             Placement::PostScan
         );
-        // The measurements replace the statistics prior.
-        let clustered_by_stats = inputs(rows, rows / 2, 9, 1, 10.0, 0.0);
+        // The saving is smaller than the fixed stage cost: benefit =
+        // 0.1 * 10 * 4 = 4 ns.
+        let small = inputs(rows, rows / 10, 0, 0, 10.0, 0.0);
         assert_eq!(
-            place_required(&clustered_by_stats, None),
-            Placement::RowFilter
+            place_required(&small, Some(Placement::PostScan)),
+            Placement::PostScan
         );
         // A high fetch latency (a remote object store) is more than the
-        // saving: benefit = 0.5 * 10 * 4 = 20 ns, cost = 30 ns.
+        // saving: benefit = 20 ns, cost = 8 + 30 ns.
         let remote = inputs(rows, rows / 2, 0, 0, 10.0, 30.0);
         assert_eq!(
             place_required(&remote, Some(Placement::RowFilter)),
@@ -223,11 +251,47 @@ mod tests {
         );
     }
 
+    #[test]
+    fn double_decode_of_read_output_columns() {
+        let rows = 100_000;
+        // benefit = 0.5 * 10 * 4 = 20 ns. Without read output columns,
+        // cost = 8 ns: row filter.
+        let mut shared = inputs(rows, rows / 2, 0, 0, 10.0, 0.0);
+        assert_eq!(place_required(&shared, None), Placement::RowFilter);
+        // The conjunct also reads 4 bytes of output columns for each row:
+        // the rows that pass (50%) decode them again, cost = 8 + 0.5 * 4 *
+        // 4 = 16 ns, still less than 0.9 * 20.
+        shared.read_output_bytes_per_row = 4.0;
+        assert_eq!(place_required(&shared, None), Placement::RowFilter);
+        // 8 bytes: cost = 8 + 16 = 24 ns.
+        shared.read_output_bytes_per_row = 8.0;
+        assert_eq!(place_required(&shared, None), Placement::PostScan);
+    }
+
+    #[test]
+    fn statistics_prior_keeps_post_scan() {
+        let rows = 100_000;
+        // The statistics pruned most row groups: post-scan, also when the
+        // measurements favor the row filter.
+        let clustered_by_stats = inputs(rows, rows / 2, 6, 4, 10.0, 0.0);
+        assert_eq!(
+            place_required(&clustered_by_stats, None),
+            Placement::PostScan
+        );
+        assert_eq!(
+            place_required(&clustered_by_stats, Some(Placement::RowFilter)),
+            Placement::PostScan
+        );
+        let not_clustered = inputs(rows, rows / 2, 4, 6, 10.0, 0.0);
+        assert_eq!(place_required(&not_clustered, None), Placement::RowFilter);
+    }
+
     #[test]
     fn margin_prevents_flapping() {
         let rows = 100_000;
-        // benefit = 0.5 * 10 * 4 = 20 ns; cost = 21 ns is inside the margin.
-        let close = inputs(rows, rows / 2, 0, 0, 10.0, 21.0);
+        // benefit = 0.5 * 10 * 4 = 20 ns; cost = 8 + 13 = 21 ns is inside
+        // the margin.
+        let close = inputs(rows, rows / 2, 0, 0, 10.0, 13.0);
         assert_eq!(
             place_required(&close, Some(Placement::RowFilter)),
             Placement::RowFilter
@@ -236,13 +300,13 @@ mod tests {
             place_required(&close, Some(Placement::PostScan)),
             Placement::PostScan
         );
-        // Outside the margin in both directions.
-        let expensive = inputs(rows, rows / 2, 0, 0, 10.0, 23.0);
+        // Outside the margin in both directions: cost = 23 ns and 17 ns.
+        let expensive = inputs(rows, rows / 2, 0, 0, 10.0, 15.0);
         assert_eq!(
             place_required(&expensive, Some(Placement::RowFilter)),
             Placement::PostScan
         );
-        let cheap = inputs(rows, rows / 2, 0, 0, 10.0, 17.0);
+        let cheap = inputs(rows, rows / 2, 0, 0, 10.0, 9.0);
         assert_eq!(
             place_required(&cheap, Some(Placement::PostScan)),
             Placement::RowFilter
diff --git a/datafusion/datasource-parquet/src/filter_placement/stats.rs b/datafusion/datasource-parquet/src/filter_placement/stats.rs
index 3525afa7b4924..0d5e2be9c1170 100644
--- a/datafusion/datasource-parquet/src/filter_placement/stats.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/stats.rs
@@ -91,6 +91,12 @@ impl Observation {
         (self.rows_in > 0).then(|| self.skippable_rows as f64 / self.rows_in as f64)
     }
 
+    /// Fraction of the evaluated rows that passed, or `None` if the
+    /// conjunct was not evaluated on any row.
+    pub(crate) fn pass_ratio(&self) -> Option<f64> {
+        (self.rows_in > 0).then(|| self.rows_out as f64 / self.rows_in as f64)
+    }
+
     /// Fraction of the row groups that the conjunct pruned with statistics,
     /// or `None` if there is no statistics pruning result.
     pub(crate) fn pruned_fraction(&self) -> Option<f64> {
@@ -344,6 +350,27 @@ pub(crate) struct PlacementSites {
 }
 
 impl PlacementSites {
+    /// Sites with a decode speed of `decode_ns_per_byte` and a fetch
+    /// latency of `fetch_ns`, weighted so that the measurements of a test
+    /// do not change them, for deterministic placement decisions.
+    #[cfg(test)]
+    pub(crate) fn with_fixed_costs(decode_ns_per_byte: f64, fetch_ns: f64) -> Self {
+        const WEIGHT: u64 = 1 << 40;
+        let sites = Self::default();
+        let weighted = |ns: f64| (ns * WEIGHT as f64) as u64;
+        sites
+            .decode
+            .nanos
+            .store(weighted(decode_ns_per_byte), Ordering::Relaxed);
+        sites.decode.bytes.store(WEIGHT, Ordering::Relaxed);
+        sites
+            .fetch
+            .nanos
+            .store(weighted(fetch_ns), Ordering::Relaxed);
+        sites.fetch.fetches.store(WEIGHT, Ordering::Relaxed);
+        sites
+    }
+
     /// The fetch latency of the scan.
     pub(crate) fn fetch(&self) -> &FetchCost {
         &self.fetch
diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index afe31bfb8396f..dfb74c2b335cc 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -27,6 +27,8 @@ use crate::access_plan::PreparedAccessPlan;
 use crate::decoder_projection::{
     DecoderProjection, DecoderProjectionBuilder, PostScanConjunct,
 };
+#[cfg(test)]
+use crate::filter_placement::PlacementSites;
 use crate::filter_placement::{FilePlacement, PlacementOptions};
 use crate::metrics::{ByteProgress, RowFilterSkippedFullyMatchedMetric};
 use crate::optional_filter::OptionalFilterOptions;
@@ -2375,6 +2377,7 @@ mod test {
         preserve_order: bool,
         optional_filters: OptionalFilterOptions,
         filter_placement: bool,
+        placement_sites: Option<Arc<PlacementSites>>,
     }
 
     #[test]
@@ -2600,6 +2603,7 @@ mod test {
                 preserve_order: false,
                 optional_filters: OptionalFilterOptions::default(),
                 filter_placement: false,
+                placement_sites: None,
             }
         }
 
@@ -2698,12 +2702,19 @@ mod test {
             self
         }
 
-        /// Set whether the scan must preserve file order.
+        /// Set whether the scan uses the adaptive filter placement.
         fn with_filter_placement(mut self, enable: bool) -> Self {
             self.filter_placement = enable;
             self
         }
 
+        /// Sets the pooled measurements of the adaptive filter placement,
+        /// for example [`PlacementSites::with_fixed_costs`].
+        fn with_placement_sites(mut self, sites: Arc<PlacementSites>) -> Self {
+            self.placement_sites = Some(sites);
+            self
+        }
+
         fn with_optional_filters(mut self, options: OptionalFilterOptions) -> Self {
             self.optional_filters = options;
             self
@@ -2761,7 +2772,7 @@ mod test {
 
             let filter_placement = PlacementOptions::new(
                 self.filter_placement,
-                Arc::default(),
+                self.placement_sites.unwrap_or_default(),
                 self.predicate.as_ref(),
             );
             Ok(ParquetMorselizer {
@@ -5740,6 +5751,17 @@ mod test {
             (schema, file)
         }
 
+        /// Fixed costs for deterministic placement decisions: 10 ns to
+        /// decode each compressed byte and no fetch latency.
+        fn fixed_sites() -> Arc<PlacementSites> {
+            Arc::new(PlacementSites::with_fixed_costs(10.0, 0.0))
+        }
+
+        /// `a` makes runs of 1000 rows with the values 0 and 1.
+        fn runs_of_1000(i: usize) -> i32 {
+            ((i / 1000) % 2) as i32
+        }
+
         struct Scan {
             rows: usize,
             metrics: ExecutionPlanMetricsSet,
@@ -5777,6 +5799,7 @@ mod test {
                 .with_predicate(Arc::clone(predicate))
                 .with_pushdown_filters(pushdown_filters)
                 .with_filter_placement(filter_placement)
+                .with_placement_sites(fixed_sites())
                 .with_optional_filters(optional_filters)
                 .with_metrics(metrics.clone())
                 .build();
@@ -5787,12 +5810,10 @@ mod test {
 
         /// A filter that removes every second row saves no decode time in a
         /// row filter: the removed rows do not make runs that the decoder can
-        /// skip. The scan starts it as a row filter (the initial rule: the
-        /// filter does not read `v`), measures it in the first row group and
-        /// moves it to the post-scan filter at the first row group boundary.
-        /// The result is the same in all placements.
+        /// skip. The scan starts it in the post-scan filter and it stays
+        /// there. The result is the same in all placements.
         #[tokio::test]
-        async fn scattered_filter_moves_post_scan_at_row_group_boundary() {
+        async fn scattered_filter_stays_post_scan() {
             let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
             let (schema, file) = write_file(&store, |i| (i % 2) as i32).await;
             let predicate = logical2physical(&col("a").eq(lit(0)), &schema);
@@ -5824,9 +5845,9 @@ mod test {
             for scan in [&adaptive, &row_filter, &post_scan] {
                 assert_eq!(scan.rows, TOTAL_ROWS / 2);
             }
-            assert_eq!(adaptive.count("filter_placement_changes"), 1);
-            assert_eq!(adaptive.row_filter_rows(), ROWS_PER_ROW_GROUP);
-            assert_eq!(adaptive.post_scan_rows(), TOTAL_ROWS - ROWS_PER_ROW_GROUP);
+            assert_eq!(adaptive.count("filter_placement_changes"), 0);
+            assert_eq!(adaptive.row_filter_rows(), 0);
+            assert_eq!(adaptive.post_scan_rows(), TOTAL_ROWS);
             assert_eq!(row_filter.row_filter_rows(), TOTAL_ROWS);
             assert_eq!(post_scan.post_scan_rows(), TOTAL_ROWS);
         }
@@ -5848,6 +5869,7 @@ mod test {
                 .with_predicate(Arc::clone(predicate))
                 .with_pushdown_filters(pushdown_filters)
                 .with_filter_placement(filter_placement)
+                .with_placement_sites(fixed_sites())
                 .with_enable_page_index(true)
                 .with_metrics(metrics.clone())
                 .build();
@@ -5871,17 +5893,20 @@ mod test {
         /// The page index skips the first pages of the first row group
         /// (`v >= 1000`) and pages of the second and third row groups
         /// (`v < 20000 OR v >= 40000`), thus the scan has a live row
-        /// selection. The placement still changes at the first row group
-        /// boundary, the rebuilt decoder keeps the selections of the
-        /// remaining row groups, and the result is the same as with a row
-        /// filter only and with a post-scan filter only.
+        /// selection. `a = 0` passes the first row of each page of 100
+        /// rows: the page index cannot prune its pages, but it leaves empty
+        /// windows of 64 rows, thus it moves to the row filter. The placement
+        /// changes at a row group boundary, the rebuilt decoder keeps the
+        /// selections of the remaining row groups, and the result is the
+        /// same as with a row filter only and with a post-scan filter only.
         #[tokio::test]
         async fn placement_changes_with_page_index_selection() {
             use datafusion_physical_plan::metrics::MetricValue;
 
             let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
             let (schema, file) =
-                write_file_with_pages(&store, |i| (i % 2) as i32, Some(100)).await;
+                write_file_with_pages(&store, |i| i32::from(i % 100 != 0), Some(100))
+                    .await;
             let predicate = logical2physical(
                 &col("a").eq(lit(0)).and(col("v").gt_eq(lit(1000_i64))).and(
                     col("v")
@@ -5906,7 +5931,7 @@ mod test {
 
             let expected: Vec<i64> = (1000..20_000)
                 .chain(40_000..TOTAL_ROWS as i64)
-                .step_by(2)
+                .filter(|v| v % 100 == 0)
                 .collect();
             assert_eq!(adaptive, expected);
             assert_eq!(row_filter, expected);
@@ -5941,9 +5966,11 @@ mod test {
         }
 
         /// A filter that removes long runs of rows saves decode time in a row
-        /// filter, thus it stays a row filter.
+        /// filter. The scan starts it in the post-scan filter, measures it in
+        /// the first row group and moves it to the row filter at the first
+        /// row group boundary.
         #[tokio::test]
-        async fn clustered_filter_stays_row_filter() {
+        async fn clustered_filter_moves_to_row_filter() {
             let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
             // `a` increases by 1 every 1000 rows.
             let (schema, file) = write_file(&store, |i| (i / 1000) as i32).await;
@@ -5963,15 +5990,16 @@ mod test {
             )
             .await;
             assert_eq!(adaptive.rows, 20_000);
-            assert_eq!(adaptive.count("filter_placement_changes"), 0);
-            assert_eq!(adaptive.row_filter_rows(), TOTAL_ROWS);
-            assert_eq!(adaptive.post_scan_rows(), 0);
+            assert_eq!(adaptive.count("filter_placement_changes"), 1);
+            assert_eq!(adaptive.post_scan_rows(), ROWS_PER_ROW_GROUP);
+            assert_eq!(adaptive.row_filter_rows(), TOTAL_ROWS - ROWS_PER_ROW_GROUP);
         }
 
         /// A filter that reads all output columns cannot save decode time in
-        /// a row filter, thus it starts in the post-scan filter.
+        /// a row filter, thus it stays in the post-scan filter, also where it
+        /// removes long runs of rows.
         #[tokio::test]
-        async fn filter_on_all_output_columns_starts_post_scan() {
+        async fn filter_on_all_output_columns_stays_post_scan() {
             let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
             let (schema, file) = write_file(&store, |i| (i / 1000) as i32).await;
             let predicate = logical2physical(&col("a").lt(lit(10)), &schema);
@@ -5983,6 +6011,7 @@ mod test {
                 .with_predicate(predicate)
                 .with_pushdown_filters(true)
                 .with_filter_placement(true)
+                .with_placement_sites(fixed_sites())
                 .with_metrics(metrics.clone())
                 .build();
             let stream = open_file(&morselizer, file).await.unwrap();
@@ -6036,13 +6065,14 @@ mod test {
         }
 
         /// With a limit, the results are the same when the placement moves a
-        /// filter to the post-scan filter in the middle of the file.
+        /// filter to the row filter in the middle of the file.
         #[tokio::test]
         async fn limit_is_applied_after_placement_change() {
             let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
-            let (schema, file) = write_file(&store, |i| (i % 2) as i32).await;
+            let (schema, file) = write_file(&store, runs_of_1000).await;
             let predicate = logical2physical(&col("a").eq(lit(0)), &schema);
-            let limit = ROWS_PER_ROW_GROUP / 2 + 100;
+            // The first row group has 8384 rows with `a = 0`.
+            let limit = 9000;
             let metrics = ExecutionPlanMetricsSet::new();
             let morselizer = ParquetMorselizerBuilder::new()
                 .with_store(Arc::clone(&store))
@@ -6050,6 +6080,7 @@ mod test {
                 .with_predicate(predicate)
                 .with_pushdown_filters(true)
                 .with_filter_placement(true)
+                .with_placement_sites(fixed_sites())
                 .with_limit(limit)
                 .with_metrics(metrics.clone())
                 .build();
@@ -6067,7 +6098,10 @@ mod test {
                         .to_vec()
                 })
                 .collect();
-            let expected: Vec<i64> = (0..limit as i64).map(|i| i * 2).collect();
+            let expected: Vec<i64> = (0..TOTAL_ROWS as i64)
+                .filter(|v| (v / 1000) % 2 == 0)
+                .take(limit)
+                .collect();
             assert_eq!(values, expected);
             assert_eq!(
                 counter_metric_value(&metrics, "filter_placement_changes"),
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 599b1364d43b5..9cb6349c067a6 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -3133,7 +3133,9 @@ mod optional_filter_tests {
     /// passed to `record`, thus the test is deterministic.
     #[test]
     fn file_placement_follows_gate_and_measurements() {
-        use crate::filter_placement::{FilePlacement, Placement, PlacementOptions};
+        use crate::filter_placement::{
+            FilePlacement, Placement, PlacementOptions, PlacementSites,
+        };
         use arrow::array::BooleanArray;
         use datafusion_physical_plan::metrics::Count;
 
@@ -3147,9 +3149,13 @@ mod optional_filter_tests {
         let metrics = ExecutionPlanMetricsSet::new();
         let mut candidates =
             prebuild(&predicate, &file_schema, &metadata, &options, &metrics);
-        let placement_options =
-            PlacementOptions::new(true, Arc::default(), Some(&predicate));
-        let output = ProjectionMask::all();
+        // A slow decode: a row filter that skips rows saves a lot.
+        let sites = Arc::new(PlacementSites::with_fixed_costs(1000.0, 0.0));
+        let placement_options = PlacementOptions::new(true, sites, Some(&predicate));
+        // The output columns are `b`, `c` and `s`: the required conjunct
+        // reads no output column (no second decode in a row filter).
+        let output =
+            ProjectionMask::leaves(metadata.file_metadata().schema_descr(), [1, 2, 3]);
         let batch_size = 100;
         let changes = Count::new();
         let mut placement = FilePlacement::try_new(
@@ -3165,9 +3171,10 @@ mod optional_filter_tests {
         .unwrap();
         let required = candidates.iter().position(|c| !c.is_optional()).unwrap();
         let gated = candidates.iter().position(|c| c.is_optional()).unwrap();
-        // Initial rule: the required conjunct does not read `b` and `c`,
-        // thus it starts as a row filter. The gate is not paused.
-        assert_eq!(placement.placements(), vec![Placement::RowFilter; 2]);
+        // Without measurements, the required conjunct starts in the
+        // post-scan filter. The gate is not paused.
+        assert!(!placement.in_row_filter(required));
+        assert!(placement.in_row_filter(gated));
 
         // The gate pauses the optional filter: it removed no row in its
         // first window (`sample_batches` = 2).
@@ -3180,7 +3187,7 @@ mod optional_filter_tests {
         assert!(gate.lock().is_paused());
         assert!(placement.decide(1000));
         assert!(!placement.in_row_filter(gated));
-        assert!(placement.in_row_filter(required));
+        assert!(!placement.in_row_filter(required));
         assert_eq!(changes.value(), 1);
 
         // The pause (`initial_pause_batches` = 4) ends while the scan skips
@@ -3189,21 +3196,30 @@ mod optional_filter_tests {
         assert!(placement.in_row_filter(gated));
         assert!(!gate.lock().is_paused());
 
-        // The required conjunct removes every second row: no run of removed
-        // rows that the decoder can skip, thus it moves to the post-scan
-        // filter.
-        let scattered: BooleanArray = (0..20_000).map(|i| Some(i % 2 == 0)).collect();
-        let stats = candidates[required].placement_stats.clone().unwrap();
-        stats.record_evaluation(&scattered);
-        assert!(placement.decide(1000));
-        assert!(!placement.in_row_filter(required));
+        // The post-scan filter measures the required conjunct.
         let post_scan = placement.post_scan_conjuncts();
         assert_eq!(post_scan.len(), 1);
         assert_eq!(post_scan[0].expr.to_string(), "a@0 > 5");
+        let stats = post_scan[0].stats.clone().unwrap();
+
+        // It removes every second row: no run of removed rows that the
+        // decoder can skip, thus it stays in the post-scan filter.
+        let scattered: BooleanArray = (0..20_000).map(|i| Some(i % 2 == 0)).collect();
+        stats.record_evaluation(&scattered);
+        assert!(!placement.decide(1000));
+        assert!(!placement.in_row_filter(required));
+
+        // More rows where it removes a long run of rows: it moves to the
+        // row filter.
+        let clustered: BooleanArray = (0..40_000).map(|i| Some(i >= 30_000)).collect();
+        stats.record_evaluation(&clustered);
+        assert!(placement.decide(1000));
+        assert!(placement.in_row_filter(required));
+        assert!(placement.post_scan_conjuncts().is_empty());
 
         // A frozen placement does not change.
         let before = placement.placements();
-        placement.restore_and_freeze(&[Placement::RowFilter, Placement::RowFilter]);
+        placement.restore_and_freeze(&[Placement::PostScan, Placement::PostScan]);
         assert!(!placement.decide(1000));
         assert_ne!(placement.placements(), before);
         assert_eq!(changes.value(), 3);
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt b/datafusion/sqllogictest/test_files/information_schema.slt
index 2da3ee55afbf5..64160c90b575d 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -377,7 +377,7 @@ datafusion.catalog.has_header true Default value for `format.has_header` for `CR
 datafusion.catalog.information_schema true Should DataFusion provide access to `information_schema` virtual tables for displaying schema information
 datafusion.catalog.location NULL Location scanned to load tables for `default` schema
 datafusion.catalog.newlines_in_values false Specifies whether newlines in (quoted) CSV values are supported. This is the default value for `format.newlines_in_values` for `CREATE EXTERNAL TABLE` if not specified explicitly in the statement. Parsing newlines in quoted values may be affected by execution behaviour such as parallel file scanning. Setting this to `true` ensures that newlines in values are parsed successfully, which may reduce performance.
-datafusion.execution.adaptive_filter_placement false When true and `datafusion.execution.parquet.pushdown_filters` is true, the Parquet scan decides for each filter conjunct if it is a row filter (late materialization) or a filter on the decoded batches. The scan measures the rows that each conjunct removes, the decode time and the fetch latency, and it changes the placement at row group boundaries. When `datafusion.execution.optional_filter_mode` is `adaptive`, an optional filter that is paused is also removed from the row filter, thus its columns are not decoded.
+datafusion.execution.adaptive_filter_placement false When true and `datafusion.execution.parquet.pushdown_filters` is true, the Parquet scan decides for each filter conjunct if it is a row filter (late materialization) or a filter on the decoded batches. Each conjunct starts as a filter on the decoded batches. The scan measures the rows that each conjunct removes, the decode time and the fetch latency, and it makes a conjunct a row filter at a row group boundary only when the decode time that the row filter saves is more than its cost. When `datafusion.execution.optional_filter_mode` is `adaptive`, an optional filter that is paused is also removed from the row filter, thus its columns are not decoded.
 datafusion.execution.batch_size 8192 Default batch size while creating new batches, it's especially useful for buffer-in-memory batches since creating tiny batches would result in too much metadata memory consumption
 datafusion.execution.coalesce_batches true When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting
 datafusion.execution.collect_statistics true Should DataFusion collect statistics when first creating a table. Has no effect after the table is created. Defaults to true.
diff --git a/datafusion/sqllogictest/test_files/parquet_adaptive_filter_placement.slt b/datafusion/sqllogictest/test_files/parquet_adaptive_filter_placement.slt
index 41d4463f0d87a..cad5b393d8069 100644
--- a/datafusion/sqllogictest/test_files/parquet_adaptive_filter_placement.slt
+++ b/datafusion/sqllogictest/test_files/parquet_adaptive_filter_placement.slt
@@ -16,9 +16,10 @@
 # under the License.
 
 # Tests for `datafusion.execution.adaptive_filter_placement`: the Parquet scan
-# places each filter conjunct in the row filter or in the post-scan filter,
-# and changes the placement at row group boundaries. The results must be the
-# same as without adaptive placement.
+# starts each filter conjunct in the post-scan filter, and moves it to the row
+# filter at a row group boundary when the measurements show that the row
+# filter saves more than it costs. The results must be the same as without
+# adaptive placement.
 
 statement ok
 set datafusion.execution.parquet.max_row_group_size = 1024;
@@ -72,13 +73,18 @@ SELECT v, s FROM t WHERE a = 1 AND b = 42 ORDER BY v LIMIT 3;
 4203 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx4203
 4205 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx4205
 
-# A limit without an order: the placement can move `a = 0` to the post-scan
-# filter in the middle of the file.
+# A limit without an order: the placement can move the clustered filter on
+# `b` to the row filter in the middle of the file.
 query I
 SELECT count(*) FROM (SELECT v FROM t WHERE a = 0 LIMIT 3000);
 ----
 3000
 
+query I
+SELECT count(*) FROM (SELECT v FROM t WHERE b < 5 OR b BETWEEN 50 AND 54 LIMIT 700);
+----
+700
+
 # The same queries without adaptive placement give the same results.
 statement ok
 set datafusion.execution.adaptive_filter_placement = false;
diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md
index f6d4fdf58dd0d..d8b16445044d3 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -147,7 +147,7 @@ The following configuration settings are available:
 | datafusion.execution.hash_join_buffering_capacity                       | 0                         | How many bytes to buffer in the probe side of hash joins while the build side is concurrently being built. Without this, hash joins will wait until the full materialization of the build side before polling the probe side. This is useful in scenarios where the query is not completely CPU bounded, allowing to do some early work concurrently and reducing the latency of the query. Note that when hash join buffering is enabled, the probe side will start eagerly polling data, not giving time for the producer side of dynamic filters to produce any meaningful predicate. Queries with dynamic filters might see performance degradation. Disabled by default, set to a number greater than 0 for enabling it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | datafusion.execution.optional_filter_mode                               | always                    | Controls how DataFusion evaluates filters that are not needed for correctness, such as the dynamic filters that hash joins and TopK push down into scans. `always` evaluates these filters like any other pushed-down filter. `adaptive` pauses these filters when they cost more than they save (see `datafusion.execution.optional_filter_min_saving_ns_per_row`) or when they remove no rows, and tries them again at intervals. `pruning_only` uses these filters only to prune files, row groups and pages with statistics, and never evaluates them row by row. This option is most important when `datafusion.execution.parquet.pushdown_filters` is true, because then the Parquet reader evaluates pushed-down filters row by row.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | datafusion.execution.optional_filter_min_saving_ns_per_row              | 20                        | The assumed work, in nanoseconds, that each row removed by an optional filter saves downstream. Optional filters are filters that are not needed for correctness, such as the dynamic filters that hash joins and TopK push down into scans. When an operator evaluates optional filters adaptively, it pauses an optional filter whose evaluation costs more than the work that it saves. Consumers that can measure the saving (the Parquet scan) add their measured decode cost. The default is about the cost of a hash table probe for one row. The best value depends on the hardware.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
-| datafusion.execution.adaptive_filter_placement                          | false                     | When true and `datafusion.execution.parquet.pushdown_filters` is true, the Parquet scan decides for each filter conjunct if it is a row filter (late materialization) or a filter on the decoded batches. The scan measures the rows that each conjunct removes, the decode time and the fetch latency, and it changes the placement at row group boundaries. When `datafusion.execution.optional_filter_mode` is `adaptive`, an optional filter that is paused is also removed from the row filter, thus its columns are not decoded.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| datafusion.execution.adaptive_filter_placement                          | false                     | When true and `datafusion.execution.parquet.pushdown_filters` is true, the Parquet scan decides for each filter conjunct if it is a row filter (late materialization) or a filter on the decoded batches. Each conjunct starts as a filter on the decoded batches. The scan measures the rows that each conjunct removes, the decode time and the fetch latency, and it makes a conjunct a row filter at a row group boundary only when the decode time that the row filter saves is more than its cost. When `datafusion.execution.optional_filter_mode` is `adaptive`, an optional filter that is paused is also removed from the row filter, thus its columns are not decoded.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
 | datafusion.optimizer.enable_distinct_aggregation_soft_limit             | true                      | When set to true, the optimizer will push a limit operation into grouped aggregations which have no aggregate expressions, as a soft limit, emitting groups once the limit is reached, before all rows in the group are read.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | datafusion.optimizer.enable_round_robin_repartition                     | true                      | When set to true, the physical plan optimizer will try to add round robin repartitioning to increase parallelism to leverage more CPU cores                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | datafusion.optimizer.enable_topk_aggregation                            | true                      | When set to true, the optimizer will attempt to perform limit operations during aggregations, if possible                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

From 9b24e9b6f3f7f290aac6953080b56b1265cf0365 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 10:41:52 -0500
Subject: [PATCH 22/42] fix(parquet): hand coalesced rows downstream at row
 group boundaries

With adaptive filter placement, the Parquet scan always has a batch
coalescer, and each conjunct starts after the decode. The coalescer held
the small batches of small row groups until it had `batch_size` rows.
Thus a TopK above the scan saw no rows until the end of the file, its
dynamic filter did not get tighter, and the runtime row group pruning
pruned no row group (`row_groups_pruned_dynamic_filter` went from 4 to 0
in `dynamic_row_group_pruning.slt`).

When the predicate has a dynamic filter that can still change, the scan
now flushes the partial batch of the coalescer at each row group
boundary, before it prunes the next row groups. Batches do not span row
groups in the decoder, thus the scan can still coalesce the slivers that
a selective filter leaves in one row group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../datasource-parquet/src/opener/mod.rs      | 22 +++++++++++++-----
 .../datasource-parquet/src/push_decoder.rs    | 23 +++++++++++++++++++
 .../test_files/dynamic_row_group_pruning.slt  | 19 +++++++++++++++
 3 files changed, 58 insertions(+), 6 deletions(-)

diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index dfb74c2b335cc..ab3356670756c 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -2002,6 +2002,17 @@ impl RowGroupsPrunedParquetOpen {
         let files_ranges_pruned_statistics =
             prepared.file_metrics.files_ranges_pruned_statistics.clone();
 
+        // True if the predicate has a dynamic filter that can still change.
+        // Such a filter gets tighter only when its producer (for example a
+        // TopK) sees the rows of the scan.
+        let watches_dynamic_filter =
+            prepared.predicate.as_ref().is_some_and(|predicate| {
+                matches!(
+                    DynamicFilterTracking::classify(predicate),
+                    DynamicFilterTracking::Watching(_)
+                )
+            });
+
         // Build a dynamic row-group pruner only when all three conditions hold:
         //   1) the scan has a predicate (so there is something to evaluate),
         //   2) the predicate has at least one not-yet-complete dynamic filter
@@ -2018,12 +2029,7 @@ impl RowGroupsPrunedParquetOpen {
         // and is tracked in https://github.com/apache/datafusion/issues/24358.
         let row_group_pruner =
             match (&prepared.predicate, rg_plan.len() > 1, has_row_selection) {
-                (Some(predicate), true, false)
-                    if matches!(
-                        DynamicFilterTracking::classify(predicate),
-                        DynamicFilterTracking::Watching(_)
-                    ) =>
-                {
+                (Some(predicate), true, false) if watches_dynamic_filter => {
                     Some(RowGroupPruner::new(
                         Arc::clone(predicate),
                         Arc::clone(&prepared.physical_file_schema),
@@ -2071,6 +2077,10 @@ impl RowGroupsPrunedParquetOpen {
                     .with_biggest_coalesce_batch_size(biggest)
             }),
             flushed: false,
+            // Hand the buffered rows downstream at each row group boundary
+            // when a dynamic filter can still change, so that its producer
+            // can tighten it before the scan prunes the next row groups.
+            flush_at_row_group_boundary: watches_dynamic_filter,
             projection_builder,
         }
         .into_stream();
diff --git a/datafusion/datasource-parquet/src/push_decoder.rs b/datafusion/datasource-parquet/src/push_decoder.rs
index c46d05d04c306..ab279b1678920 100644
--- a/datafusion/datasource-parquet/src/push_decoder.rs
+++ b/datafusion/datasource-parquet/src/push_decoder.rs
@@ -349,6 +349,16 @@ pub(crate) struct PushDecoderStreamState {
     /// end-of-input flushing happens exactly once no matter which terminal
     /// path reached it.
     pub(crate) flushed: bool,
+    /// True if the partial batch in [`Self::batch_coalescer`] is handed
+    /// downstream at each row group boundary, before the runtime row group
+    /// pruning for the next row groups.
+    ///
+    /// Set when the predicate has a dynamic filter that can still change.
+    /// Its producer (for example a TopK) can tighten the filter only after it
+    /// sees rows. If the coalescer held the rows of small row groups until it
+    /// has `batch_size` rows, the producer would see no rows until the end of
+    /// the file, and the scan could not prune any row group with the filter.
+    pub(crate) flush_at_row_group_boundary: bool,
     /// Builds a new [`DecoderProjection`] when the adaptive filter placement
     /// changes the post-scan conjuncts at a row group boundary. `Some`
     /// exactly when [`RowFilterContext::placement`] is `Some`.
@@ -710,6 +720,19 @@ impl PushDecoderStreamState {
             {
                 return Some((Err(e), self));
             }
+            // Before the pruning below, hand the rows of the previous row
+            // groups downstream, so that a dynamic filter can use them. The
+            // next call comes back to this boundary with an empty coalescer.
+            if at_boundary
+                && self.flush_at_row_group_boundary
+                && let Some(coalescer) = self.batch_coalescer.as_mut()
+                && coalescer.get_buffered_rows() > 0
+            {
+                if let Err(e) = coalescer.finish_buffered_batch() {
+                    return Some((Err(DataFusionError::from(e)), self));
+                }
+                return self.emit_completed();
+            }
             if at_boundary && !self.rg_plan.is_empty() {
                 let pruned_count = self.prune_boundary_row_groups();
                 match self.rebuild_decoder_at_boundary(pruned_count) {
diff --git a/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt b/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt
index fec9add854b3a..c5b86457ad409 100644
--- a/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt
+++ b/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt
@@ -100,6 +100,25 @@ Plan with Metrics
 01)SortExec: TopK(fetch=3), expr=[v@0 DESC], preserve_partitioning=[false], filter=[v@0 IS NULL OR v@0 > 12], metrics=[output_rows=3, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>]
 02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_row_group_pruning/data.parquet]]}, projection=[v], file_type=parquet, predicate=DynamicFilter [ v@0 IS NULL OR v@0 > 12 ], sort_order_for_reorder=[v@0 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=v_null_count@0 > 0 OR v_null_count@0 != row_count@2 AND v_max@1 > 12, required_guarantees=[], metrics=[output_rows=3, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=5 total → 5 matched, row_groups_pruned_bloom_filter=5 total → 5 matched, page_index_pages_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, bytes_processed=<slt:ignore>, bytes_scanned=<slt:ignore>, row_groups_pruned_dynamic_filter=4, metadata_load_time=<slt:ignore>, scan_efficiency_ratio=<slt:ignore>]
 
+# The same query with adaptive filter placement and with the TopK dynamic
+# filter evaluated after the decode: the scan coalesces the small batches of
+# these 3-row row groups, but it must hand the buffered rows to the TopK at
+# each row group boundary. Otherwise the TopK sees no row until the end of
+# the file and the scan prunes no row group. The scan must prune the same
+# four row groups as above.
+statement ok
+set datafusion.execution.adaptive_filter_placement = true;
+
+query TT
+explain analyze select v from t order by v desc limit 3;
+----
+Plan with Metrics
+01)SortExec: TopK(fetch=3), expr=[v@0 DESC], preserve_partitioning=[false], filter=[v@0 IS NULL OR v@0 > 12], metrics=[output_rows=3, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_row_group_pruning/data.parquet]]}, projection=[v], file_type=parquet, predicate=DynamicFilter [ v@0 IS NULL OR v@0 > 12 ], sort_order_for_reorder=[v@0 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=v_null_count@0 > 0 OR v_null_count@0 != row_count@2 AND v_max@1 > 12, required_guarantees=[], metrics=[output_rows=3, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=5 total → 5 matched, row_groups_pruned_bloom_filter=5 total → 5 matched, page_index_pages_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, bytes_processed=<slt:ignore>, bytes_scanned=<slt:ignore>, filter_placement_changes=0, row_groups_pruned_dynamic_filter=4, metadata_load_time=<slt:ignore>, scan_efficiency_ratio=<slt:ignore>]
+
+statement ok
+RESET datafusion.execution.adaptive_filter_placement;
+
 # `EXPLAIN ANALYZE` with a *static* predicate must surface the
 # `row_filter_skipped_fully_matched` metric with a non-zero value — this
 # is the only way to see the fully-matched RowFilter-skip optimization

From 824dc2771541cc91aca972ca24d77e49a6f93fa9 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 10:41:57 -0500
Subject: [PATCH 23/42] fix(parquet): count the actual rows in
 optional_filter_rows_skipped

When adaptive filter placement removes a paused optional filter from the
row filter for a row group, it counts the batches of that row group down
on the gate at the next boundary. Each batch added `batch_size` rows to
`optional_filter_rows_skipped`, also when the row group had fewer rows.
Thus a 15-row file reported 16.38K skipped rows.

Count the rows of the row group instead: all batches have `batch_size`
rows, except the last, which has the remaining rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../core/tests/parquet/optional_filters.rs    | 71 +++++++++++++++++--
 .../src/filter_placement/mod.rs               |  3 +-
 .../datasource-parquet/src/row_filter.rs      | 26 +++----
 3 files changed, 80 insertions(+), 20 deletions(-)

diff --git a/datafusion/core/tests/parquet/optional_filters.rs b/datafusion/core/tests/parquet/optional_filters.rs
index 8d5e10ae55225..73fdd45a9d4b0 100644
--- a/datafusion/core/tests/parquet/optional_filters.rs
+++ b/datafusion/core/tests/parquet/optional_filters.rs
@@ -72,6 +72,15 @@ const BATCH_SIZE: usize = 100;
 /// group has all values `0..100`, thus statistics cannot prune it), `rg` is
 /// the row group index and `v` is the row number.
 fn write_file() -> (NamedTempFile, SchemaRef) {
+    write_file_with(ROW_GROUPS, ROWS_PER_ROW_GROUP)
+}
+
+/// Like [`write_file`], with `row_groups` row groups of
+/// `rows_per_row_group` rows.
+fn write_file_with(
+    row_groups: usize,
+    rows_per_row_group: usize,
+) -> (NamedTempFile, SchemaRef) {
     let schema = Arc::new(Schema::new(vec![
         Field::new("a", DataType::Int32, false),
         Field::new("rg", DataType::Int32, false),
@@ -79,16 +88,16 @@ fn write_file() -> (NamedTempFile, SchemaRef) {
     ]));
     let file = NamedTempFile::new().unwrap();
     let props = WriterProperties::builder()
-        .set_max_row_group_row_count(Some(ROWS_PER_ROW_GROUP))
+        .set_max_row_group_row_count(Some(rows_per_row_group))
         .build();
     let mut writer =
         ArrowWriter::try_new(file.reopen().unwrap(), Arc::clone(&schema), Some(props))
             .unwrap();
-    for rg in 0..ROW_GROUPS {
-        let start = rg * ROWS_PER_ROW_GROUP;
-        let rows = start..start + ROWS_PER_ROW_GROUP;
+    for rg in 0..row_groups {
+        let start = rg * rows_per_row_group;
+        let rows = start..start + rows_per_row_group;
         let a: Int32Array = rows.clone().map(|i| (i % 100) as i32).collect();
-        let rg_col = Int32Array::from(vec![rg as i32; ROWS_PER_ROW_GROUP]);
+        let rg_col = Int32Array::from(vec![rg as i32; rows_per_row_group]);
         let v: Int64Array = rows.map(|i| i as i64).collect();
         let batch = RecordBatch::try_new(
             Arc::clone(&schema),
@@ -98,7 +107,7 @@ fn write_file() -> (NamedTempFile, SchemaRef) {
         writer.write(&batch).unwrap();
     }
     let metadata = writer.close().unwrap();
-    assert_eq!(metadata.num_row_groups(), ROW_GROUPS);
+    assert_eq!(metadata.num_row_groups(), row_groups);
     (file, schema)
 }
 
@@ -477,6 +486,56 @@ async fn optional_filter_is_not_evaluated_post_scan() {
     assert_eq!(metric(scan.as_ref(), "post_scan_rows_pruned"), 0);
 }
 
+/// With adaptive filter placement, the scan removes a paused optional
+/// filter from the row filter for complete row groups, and counts the rows of
+/// these row groups as skipped. It must count the rows that the row groups
+/// have, not the batch size: here each row group has 15 rows and the batch
+/// size is 8192.
+#[tokio::test]
+async fn skipped_rows_of_placement_are_the_rows_of_the_row_groups() {
+    let row_groups = 20;
+    let rows_per_row_group = 15;
+    let total_rows = row_groups * rows_per_row_group;
+    let (file, schema) = write_file_with(row_groups, rows_per_row_group);
+    let filter = removes_no_rows(&schema);
+
+    let mut options = ConfigOptions::default();
+    options.execution.parquet.pushdown_filters = true;
+    options.execution.optional_filter_mode = OptionalFilterMode::Adaptive;
+    options.execution.optional_filter_min_saving_ns_per_row = 1e9;
+    options.execution.adaptive_filter_placement = true;
+    let source = ParquetSource::new(Arc::clone(&schema))
+        .try_pushdown_filters(vec![optional(Arc::clone(&filter))], &options)
+        .unwrap()
+        .updated_node
+        .expect("the scan accepts the predicate");
+    let path = file.path().to_str().unwrap().to_string();
+    let size = std::fs::metadata(&path).unwrap().len();
+    let config = FileScanConfigBuilder::new(ObjectStoreUrl::local_filesystem(), source)
+        .with_file(PartitionedFile::new(path, size))
+        .build();
+    let scan: Arc<dyn ExecutionPlan> = DataSourceExec::from_data_source(config);
+    let plan: Arc<dyn ExecutionPlan> =
+        Arc::new(FilterExec::try_new(filter, Arc::clone(&scan)).unwrap());
+    let ctx = SessionContext::new_with_config(SessionConfig::new().with_batch_size(8192));
+    let batches = collect(plan, ctx.task_ctx()).await.unwrap();
+    assert_eq!(values(&batches).len(), total_rows);
+
+    // The gate evaluates the filter on the first row groups, then pauses
+    // it. The placement then removes it from the row filter.
+    assert!(metric(scan.as_ref(), "optional_filter_pauses") > 0);
+    let skipped = metric(scan.as_ref(), "optional_filter_rows_skipped");
+    assert!(skipped > 0, "expected skipped rows");
+    assert_eq!(skipped % rows_per_row_group, 0, "skipped {skipped} rows");
+    // Each row is skipped or evaluated at most once.
+    let evaluated = metric(scan.as_ref(), "pushdown_rows_matched")
+        + metric(scan.as_ref(), "pushdown_rows_pruned");
+    assert!(
+        skipped + evaluated <= total_rows,
+        "skipped {skipped} and evaluated {evaluated} of {total_rows} rows"
+    );
+}
+
 /// A file with the columns `a` (`i % 100`) and `s` (a struct that is null
 /// for each even row). The row filter cannot evaluate `s IS NOT NULL`,
 /// because the predicate reads the complete struct.
diff --git a/datafusion/datasource-parquet/src/filter_placement/mod.rs b/datafusion/datasource-parquet/src/filter_placement/mod.rs
index 43deae1183ae3..b9f3158316063 100644
--- a/datafusion/datasource-parquet/src/filter_placement/mod.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/mod.rs
@@ -238,12 +238,11 @@ impl FilePlacement {
         // A skipped optional conjunct did not see the batches of the last
         // row group. Count them down on its gate, so that the pause can
         // end.
-        let batches = self.row_group_rows.div_ceil(self.batch_size);
         for conjunct in &self.conjuncts {
             if let (Placement::Skip, ConjunctKind::Optional { gate }) =
                 (conjunct.placement, &conjunct.kind)
             {
-                gate.lock().skip_batches(batches, self.batch_size);
+                gate.lock().skip_rows(self.row_group_rows, self.batch_size);
             }
         }
         self.row_group_rows = next_row_group_rows;
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 9cb6349c067a6..59466d4d7f37c 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -176,18 +176,20 @@ impl OptionalFilterGateState {
         self.gate.is_paused()
     }
 
-    /// Counts down up to `batches` batches of `rows_per_batch` rows that the
-    /// scan did not evaluate the filter on, while the gate is paused. The
-    /// adaptive filter placement calls this when it removed the paused
-    /// filter from the `RowFilter` (see [`crate::filter_placement`]). Stops
-    /// when the gate stops the pause, for example because the filter
-    /// changed.
-    pub(crate) fn skip_batches(&mut self, batches: usize, rows_per_batch: usize) {
-        for _ in 0..batches {
-            if !self.gate.is_paused() {
-                break;
-            }
-            self.begin_batch(rows_per_batch);
+    /// Counts down the batches of `rows` rows that the scan did not evaluate
+    /// the filter on, while the gate is paused. The scan reads the rows in
+    /// batches of `batch_size` rows at most: the last batch has the
+    /// remaining rows. The adaptive filter placement calls this when it
+    /// removed the paused filter from the `RowFilter` (see
+    /// [`crate::filter_placement`]). Stops when the gate stops the pause,
+    /// for example because the filter changed.
+    pub(crate) fn skip_rows(&mut self, rows: usize, batch_size: usize) {
+        let batch_size = batch_size.max(1);
+        let mut remaining = rows;
+        while remaining > 0 && self.gate.is_paused() {
+            let batch_rows = remaining.min(batch_size);
+            self.begin_batch(batch_rows);
+            remaining -= batch_rows;
         }
     }
 

From 49b7bb1b19f3886ba1df5661965fa93992480cd4 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 11:51:52 -0500
Subject: [PATCH 24/42] fix(parquet): count only skippable rows in the saving
 of an optional row filter

The saving of a row that an optional row filter removed was the decode
time of the output columns that the filter does not read. Two errors made
it far too large, thus costly filters never paused:

* The decoder skips the other columns only for removed rows in long runs.
  A filter that keeps 5% of the rows spread over the file removes 95% of
  the rows, but the decoder still decodes all pages. The saving now counts
  only the skippable rows (removed rows in 64-row windows where no row
  passes), as the adaptive filter placement does: each gate counts the
  removed and skippable rows of its filter.
* The decode speed was measured on all output batches, also those of row
  groups where the row filter kept few rows spread over the row group.
  Each output row then costs much more to decode (TPC-H Q9: 21 to 28 ns
  for each byte against 1 to 4 ns). The scan now measures only the output
  batches of row groups where the row filter removed no rows.

The gate also adds the fixed cost of a row filter stage
(`ROW_FILTER_STAGE_NS_PER_ROW`, 8 ns) to the evaluation time. The constant
and `skippable_rows` move to the new `row_filter_cost` module, which the
adaptive filter placement uses too.

PR: #25682 (the new `row_filter_cost` module is also used by #25727)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/filter_placement/model.rs             |  11 +-
 .../src/filter_placement/stats.rs             |  61 +------
 datafusion/datasource-parquet/src/mod.rs      |   1 +
 .../datasource-parquet/src/opener/mod.rs      |  82 +++++++++
 .../datasource-parquet/src/optional_filter.rs | 170 +++++++++++++-----
 .../datasource-parquet/src/push_decoder.rs    |  41 ++++-
 .../datasource-parquet/src/row_filter.rs      |  53 ++++--
 .../datasource-parquet/src/row_filter_cost.rs | 102 +++++++++++
 8 files changed, 390 insertions(+), 131 deletions(-)
 create mode 100644 datafusion/datasource-parquet/src/row_filter_cost.rs

diff --git a/datafusion/datasource-parquet/src/filter_placement/model.rs b/datafusion/datasource-parquet/src/filter_placement/model.rs
index 4b216b4364418..b1edf1b73e19a 100644
--- a/datafusion/datasource-parquet/src/filter_placement/model.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/model.rs
@@ -19,6 +19,7 @@
 //! [module documentation](super) for the model.
 
 use super::stats::Observation;
+use crate::row_filter_cost::ROW_FILTER_STAGE_NS_PER_ROW;
 
 /// Where the scan evaluates one conjunct of its predicate.
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -39,16 +40,6 @@ pub(crate) const MIN_OBSERVED_ROWS: u64 = 8192;
 /// row groups stays in the post-scan filter. See [`place_required`].
 pub(crate) const CLUSTERED_PRUNED_FRACTION: f64 = 0.5;
 
-/// The fixed cost of a row filter stage for each row that it evaluates,
-/// in nanoseconds. It does not depend on the predicate: the decoder builds a
-/// `RowSelection` from the result, skips or reads the records of the other
-/// columns with it (`skip_records`), fills the predicate cache
-/// (`CachedArrayReader`) and fetches the columns of the stage before the
-/// other columns. Profiles of TPC-H SF1 Q6, Q12 and Q4 on an Apple M-series
-/// laptop showed 4 to 16 ns for each row and stage, about 20 times the time
-/// of the predicate. The post-scan filter has none of these costs.
-pub(crate) const ROW_FILTER_STAGE_NS_PER_ROW: f64 = 8.0;
-
 /// A row filter conjunct moves to the post-scan filter only if its cost is
 /// larger than this multiple of its benefit. The same margin as the
 /// optional filter gate.
diff --git a/datafusion/datasource-parquet/src/filter_placement/stats.rs b/datafusion/datasource-parquet/src/filter_placement/stats.rs
index 0d5e2be9c1170..94775eb21fcc0 100644
--- a/datafusion/datasource-parquet/src/filter_placement/stats.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/stats.rs
@@ -23,8 +23,7 @@ use std::sync::Arc;
 use std::sync::atomic::{AtomicU64, Ordering};
 use std::time::Duration;
 
-use arrow::array::{Array, BooleanArray};
-use arrow::buffer::BooleanBuffer;
+use arrow::array::BooleanArray;
 use datafusion_physical_expr::PhysicalExpr;
 use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
 use datafusion_physical_expr::filter_stats::duration_nanos;
@@ -32,40 +31,7 @@ use datafusion_physical_expr::utils::is_optional_filter;
 use parking_lot::Mutex;
 
 use crate::optional_filter::DEFAULT_DECODE_NS_PER_BYTE;
-
-/// Number of rows in one window of [`skippable_rows`].
-///
-/// The Parquet decoder skips the rows that a row filter removes only when
-/// the removed rows make long runs. Its default selection policy
-/// (`RowSelectionPolicy::Auto { threshold: 32 }`) decodes all rows and then
-/// filters them when the runs are shorter than 32 rows on average. A window
-/// of 64 rows is one `u64` of the filter result.
-pub(crate) const SKIP_WINDOW_ROWS: usize = 64;
-
-/// Rows of `result` in windows of [`SKIP_WINDOW_ROWS`] rows where no row
-/// passes (a `null` does not pass). These are the rows that a row filter
-/// lets the decoder skip for the other columns. A partial window at the end
-/// is counted if no row in it passes.
-pub(crate) fn skippable_rows(result: &BooleanArray) -> usize {
-    let values = result.values();
-    match result.nulls() {
-        Some(nulls) => skippable_in(&(values & nulls.inner())),
-        None => skippable_in(values),
-    }
-}
-
-/// Rows of `passed` in windows of [`SKIP_WINDOW_ROWS`] rows where no bit is
-/// set, see [`skippable_rows`].
-fn skippable_in(passed: &BooleanBuffer) -> usize {
-    let chunks = passed.bit_chunks();
-    let full = chunks.iter().filter(|chunk| *chunk == 0).count() * SKIP_WINDOW_ROWS;
-    let remainder = if chunks.remainder_len() > 0 && chunks.remainder_bits() == 0 {
-        chunks.remainder_len()
-    } else {
-        0
-    };
-    full + remainder
-}
+use crate::row_filter_cost::{SKIP_WINDOW_ROWS, skippable_in, skippable_rows};
 
 /// What the scan measured for one conjunct: in the row filter or in the
 /// post-scan filter, and in the row group statistics pruning.
@@ -76,7 +42,7 @@ pub(crate) struct Observation {
     /// Rows that passed the conjunct.
     pub(crate) rows_out: u64,
     /// Rows in windows where no row passed the conjunct, see
-    /// [`skippable_rows`] and [`StageSelection`].
+    /// [`skippable_rows`](crate::row_filter_cost::skippable_rows) and [`StageSelection`].
     pub(crate) skippable_rows: u64,
     /// Row groups that the conjunct alone pruned with statistics.
     pub(crate) row_groups_pruned: u64,
@@ -418,27 +384,6 @@ mod tests {
         values.into_iter().collect()
     }
 
-    #[test]
-    fn skippable_rows_counts_empty_windows() {
-        // Window 0 (rows 0..64) has one passing row, window 1 has none, the
-        // partial window (rows 128..160) has none.
-        let mut values = vec![Some(false); 160];
-        values[10] = Some(true);
-        assert_eq!(skippable_rows(&bools(values.clone())), 64 + 32);
-
-        // A null does not pass.
-        values[70] = None;
-        assert_eq!(skippable_rows(&bools(values.clone())), 64 + 32);
-
-        // One passing row in the partial window.
-        values[150] = Some(true);
-        assert_eq!(skippable_rows(&bools(values)), 64);
-
-        // Scattered passing rows: no window is empty.
-        let scattered = (0..640).map(|i| Some(i % 50 == 0));
-        assert_eq!(skippable_rows(&bools(scattered)), 0);
-    }
-
     #[test]
     fn conjunct_stats_accumulate() {
         let stats = ConjunctStats::default();
diff --git a/datafusion/datasource-parquet/src/mod.rs b/datafusion/datasource-parquet/src/mod.rs
index c5adb1eb349c6..94b307f65525e 100644
--- a/datafusion/datasource-parquet/src/mod.rs
+++ b/datafusion/datasource-parquet/src/mod.rs
@@ -40,6 +40,7 @@ mod pruning;
 mod push_decoder;
 mod reader;
 mod row_filter;
+mod row_filter_cost;
 mod row_group_filter;
 mod schema_coercion;
 mod sink;
diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index ab3356670756c..a1646a5b4e2e4 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -5652,6 +5652,88 @@ mod test {
         }
     }
 
+    /// The decode speed for the saving of optional filters is measured only
+    /// on the row groups where the row filter removed no rows: a row filter
+    /// that removes rows spread over the row group makes each output row
+    /// more expensive to decode.
+    #[tokio::test]
+    async fn decode_speed_is_measured_without_row_filter_selection() {
+        use arrow::array::{Int32Array, Int64Array};
+        use datafusion_common::config::OptionalFilterMode;
+        use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
+        use datafusion_physical_expr::optional_filter_gate::OptionalFilterGateConfig;
+
+        const ROWS_PER_ROW_GROUP: usize = 8192;
+        let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("a", DataType::Int32, false),
+            Field::new("v", DataType::Int64, false),
+        ]));
+        // Four row groups. `a = 0` removes every second row of the first
+        // three row groups, and no row of the last one.
+        let rows = 4 * ROWS_PER_ROW_GROUP;
+        let a = |i: usize| {
+            if i < 3 * ROWS_PER_ROW_GROUP {
+                (i % 2) as i32
+            } else {
+                0
+            }
+        };
+        let batch = RecordBatch::try_new(
+            Arc::clone(&schema),
+            vec![
+                Arc::new(Int32Array::from_iter_values((0..rows).map(a))),
+                Arc::new(Int64Array::from_iter_values(0..rows as i64)),
+            ],
+        )
+        .unwrap();
+        let props = WriterProperties::builder()
+            .set_max_row_group_row_count(Some(ROWS_PER_ROW_GROUP))
+            .build();
+        let size = write_parquet_batches(
+            Arc::clone(&store),
+            "decode.parquet",
+            vec![batch],
+            Some(props),
+        )
+        .await;
+        let file = PartitionedFile::new("decode.parquet".to_string(), size as u64);
+
+        let predicate: Arc<dyn PhysicalExpr> = Arc::new(OptionalFilterPhysicalExpr::new(
+            logical2physical(&col("a").eq(lit(0)), &schema),
+        ));
+        let options = OptionalFilterOptions {
+            mode: OptionalFilterMode::Adaptive,
+            // The gate never pauses the filter.
+            gate_config: OptionalFilterGateConfig {
+                min_saving_ns_per_row: 1e9,
+                ..Default::default()
+            },
+            ..Default::default()
+        };
+        let morselizer = ParquetMorselizerBuilder::new()
+            .with_store(Arc::clone(&store))
+            .with_schema(Arc::clone(&schema))
+            .with_predicate(predicate)
+            .with_pushdown_filters(true)
+            .with_optional_filters(options.clone())
+            .build();
+        let stream = open_file(&morselizer, file).await.unwrap();
+        let (_, output_rows) = count_batches_and_rows(stream).await;
+        assert_eq!(output_rows, 3 * ROWS_PER_ROW_GROUP / 2 + ROWS_PER_ROW_GROUP);
+
+        // Only the last row group is measured: its rows times the average
+        // compressed bytes for each row of the output columns.
+        let measured = options.decode_cost.measured_bytes() as f64;
+        let file_bytes_per_row = size as f64 / rows as f64;
+        let row_group_bytes = ROWS_PER_ROW_GROUP as f64 * file_bytes_per_row;
+        assert!(measured > 0.0);
+        assert!(
+            measured < 2.0 * row_group_bytes,
+            "measured {measured} bytes, one row group has about {row_group_bytes}"
+        );
+    }
+
     /// Writes a file with the columns `id` (Int32: 1, 2, 3) and `s`
     /// (Struct{value: Int32, label: Utf8}; row 1 is null).
     async fn write_struct_file(
diff --git a/datafusion/datasource-parquet/src/optional_filter.rs b/datafusion/datasource-parquet/src/optional_filter.rs
index 4318ea68bb75f..8e10a1e18951f 100644
--- a/datafusion/datasource-parquet/src/optional_filter.rs
+++ b/datafusion/datasource-parquet/src/optional_filter.rs
@@ -26,45 +26,48 @@
 //! complete predicate, thus optional filters. Only the row-level
 //! `RowFilter` (used when `pushdown_filters` is true) changes with the mode.
 //!
-//! # The saving of a removed row
+//! # The cost and the saving of a row filter predicate
 //!
 //! In [`OptionalFilterMode::Adaptive`], the [`OptionalFilterGate`] of an
-//! optional filter pauses the filter when its evaluation costs more than the
-//! work that the rows it removes save. The gate adds the configured minimum
-//! saving (`datafusion.execution.optional_filter_min_saving_ns_per_row`, for
-//! the work after the scan) to the saving that the scan measures: the time
-//! to decode the output columns that the filter does not read, for each
-//! row. The scan does not decode these columns for a removed row.
-//!
-//! The scan estimates this decode time from the compressed size of the
-//! column chunks and a measured decode speed:
+//! optional filter pauses the filter when it costs more than the work that
+//! the rows it removes save. The gate adds the configured minimum saving
+//! (`datafusion.execution.optional_filter_min_saving_ns_per_row`, for the
+//! work after the scan) to the saving that the scan measures, and the cost
+//! of the row filter stage to the evaluation time:
 //!
 //! ```text
-//! measured saving = ns_per_byte * (compressed bytes for each row of the
-//!                                  output columns that the filter does not read)
-//! ns_per_byte     = time to decode output batches
-//!                   / (output rows * compressed bytes for each row of the
-//!                      output columns)
+//! measured saving   = skippable rows / removed rows * ns_per_byte
+//!                     * (compressed bytes for each row of the output
+//!                        columns that the filter does not read)
+//! measured overhead = ROW_FILTER_STAGE_NS_PER_ROW
 //! ```
 //!
+//! The decoder does not decode the other output columns of a removed row
+//! only when the removed rows make long runs: *skippable rows* are the
+//! removed rows in windows of 64 rows where no row passes (see
+//! [`skippable_rows`]). A filter that keeps 5% of the rows, spread over the
+//! file, removes 95% of the rows, but the decoder still decodes all pages.
+//! The gate of each filter counts its removed and skippable rows.
+//!
 //! The bytes for each row come from the metadata of the file (the average
-//! over its row groups). `ns_per_byte` is measured over all files and
-//! partitions of the scan ([`DecodeCost`]): the scan times each call that
-//! decodes an output batch. This time does not include the row filter,
-//! because the Parquet decoder evaluates the row filter of a row group
-//! before it decodes the output columns. Before the first measurement,
-//! `ns_per_byte` is [`DEFAULT_DECODE_NS_PER_BYTE`].
+//! over its row groups). `ns_per_byte` is the decode speed of the output
+//! columns. It is measured over all files and partitions of the scan
+//! ([`DecodeCost`]): the scan times each call that decodes an output batch
+//! of a row group where the row filter removed no rows. Before the first
+//! measurement, `ns_per_byte` is [`DEFAULT_DECODE_NS_PER_BYTE`].
 //!
-//! This is an estimate: the decode time of a row depends on the encoding and
-//! on the rows that are selected (a sparse selection costs more for each
-//! row), not only on the compressed size. But it is cheap (a clock read and
-//! a few atomic operations for each output batch), it adapts to the data and
-//! the hardware, and it separates the two important cases: a filter that
-//! reads all the output columns (for example a hash join filter on the join
-//! keys) saves only the work after the scan, and a filter on one column of
-//! a wide table (for example a TopK filter with `SELECT *`) saves the decode
-//! of all the other columns.
+//! This is an estimate: the decode time of a row depends on the encoding,
+//! not only on the compressed size. But it is cheap (a clock read and a few
+//! atomic operations for each output batch, and a count of the empty 64-row
+//! windows of each filter result), it adapts to the data and the hardware,
+//! and it separates the important cases: a filter that reads all the output
+//! columns (for example a hash join filter on the join keys) or removes
+//! rows that are spread over the file saves only the work after the scan,
+//! and a filter on one column of a wide table that removes long runs of
+//! rows (for example a TopK filter with `SELECT *` on sorted data) saves the
+//! decode of all the other columns.
 //!
+//! [`skippable_rows`]: crate::row_filter_cost::skippable_rows
 //! [`OptionalFilterPhysicalExpr`]: datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr
 //! [`OptionalFilterGate`]: datafusion_physical_expr::optional_filter_gate::OptionalFilterGate
 
@@ -72,6 +75,8 @@ use std::sync::Arc;
 use std::sync::atomic::{AtomicU64, Ordering};
 use std::time::Duration;
 
+use arrow::array::BooleanArray;
+
 use datafusion_common::config::OptionalFilterMode;
 use datafusion_physical_expr::filter_stats::duration_nanos;
 use datafusion_physical_expr::optional_filter_gate::{
@@ -80,6 +85,8 @@ use datafusion_physical_expr::optional_filter_gate::{
 use parquet::arrow::ProjectionMask;
 use parquet::file::metadata::ParquetMetaData;
 
+use crate::row_filter_cost::{ROW_FILTER_STAGE_NS_PER_ROW, skippable_rows};
+
 /// The decode time for each compressed byte that the scan assumes before it
 /// measures it (see the [module documentation](self)). The TPC-DS SF1 scans
 /// on one core of a laptop take 4 to 8 ns for each compressed byte (scan
@@ -94,6 +101,13 @@ pub(crate) const DEFAULT_DECODE_NS_PER_BYTE: f64 = 4.0;
 /// The measured decode speed of the output columns of one scan, shared by
 /// all partitions and files of the scan. See the [module
 /// documentation](self).
+///
+/// Only the output batches of row groups where the row filter removed no
+/// rows are measured. A row filter that removes rows spread over the row
+/// group makes the decode of each output row much more expensive (the
+/// decoder decodes all pages and then drops most rows): TPC-H Q9 measured
+/// 21 to 28 ns for each byte with a filter that keeps 5% of the rows,
+/// against 1 to 4 ns without a row filter.
 #[derive(Debug, Default)]
 pub(crate) struct DecodeCost {
     /// Time to decode output batches, in nanoseconds.
@@ -111,6 +125,12 @@ impl DecodeCost {
         self.bytes.fetch_add(bytes, Ordering::Relaxed);
     }
 
+    /// The estimated compressed bytes of the measured output batches.
+    #[cfg(test)]
+    pub(crate) fn measured_bytes(&self) -> u64 {
+        self.bytes.load(Ordering::Relaxed)
+    }
+
     /// The decode time for each compressed byte, or
     /// [`DEFAULT_DECODE_NS_PER_BYTE`] before the first measurement.
     pub(crate) fn ns_per_byte(&self) -> f64 {
@@ -148,33 +168,87 @@ pub(crate) fn compressed_bytes_per_row(
     }
 }
 
-/// The measured saving of one gated optional filter of one file, see the
-/// [module documentation](self).
-#[derive(Debug, Clone)]
+/// The measured saving and overhead of the gate of one optional filter of
+/// one file, see the [module documentation](self).
+#[derive(Debug)]
 pub(crate) struct OptionalFilterSaving {
-    /// The value that the gate of the filter reads.
-    pub(crate) saving: Arc<MeasuredRowSaving>,
+    /// The values that the gate of the filter reads.
+    measured: Arc<MeasuredRowSaving>,
+    /// The decode speed of the scan.
+    decode_cost: Arc<DecodeCost>,
     /// Compressed bytes for each row of the output columns that the filter
     /// does not read.
     pub(crate) unread_bytes_per_row: f64,
+    /// Rows that the filter removed as a row filter predicate.
+    removed_rows: AtomicU64,
+    /// The removed rows that the decoder can skip, see
+    /// [`skippable_rows`].
+    skippable_rows: AtomicU64,
 }
 
 impl OptionalFilterSaving {
     /// A saving for a filter that does not read `unread_bytes_per_row`
-    /// compressed bytes of the output columns, with the current decode
-    /// speed of `decode_cost`.
-    pub(crate) fn new(unread_bytes_per_row: f64, decode_cost: &DecodeCost) -> Self {
+    /// compressed bytes of the output columns. The decode speed comes from
+    /// `decode_cost`.
+    pub(crate) fn new(unread_bytes_per_row: f64, decode_cost: Arc<DecodeCost>) -> Self {
         let saving = Self {
-            saving: Arc::new(MeasuredRowSaving::new()),
+            measured: Arc::new(MeasuredRowSaving::new()),
+            decode_cost,
             unread_bytes_per_row,
+            removed_rows: AtomicU64::new(0),
+            skippable_rows: AtomicU64::new(0),
         };
-        saving.update(decode_cost.ns_per_byte());
         saving
+            .measured
+            .set_overhead_ns_per_row(ROW_FILTER_STAGE_NS_PER_ROW);
+        saving
+    }
+
+    /// The values that the gate reads, see
+    /// [`OptionalFilterGate::with_measured_saving`](datafusion_physical_expr::optional_filter_gate::OptionalFilterGate::with_measured_saving).
+    pub(crate) fn measured(&self) -> &Arc<MeasuredRowSaving> {
+        &self.measured
+    }
+
+    /// Records one evaluation of the filter as a row filter predicate, with
+    /// one value in `result` for each evaluated row.
+    pub(crate) fn record_evaluation(&self, result: &BooleanArray) {
+        let rows_in = result.len();
+        // `true_count` does not count nulls.
+        let rows_out = result.true_count();
+        if rows_out == rows_in {
+            return;
+        }
+        let skippable = if rows_out == 0 {
+            rows_in
+        } else {
+            skippable_rows(result)
+        };
+        self.removed_rows
+            .fetch_add((rows_in - rows_out) as u64, Ordering::Relaxed);
+        self.skippable_rows
+            .fetch_add(skippable as u64, Ordering::Relaxed);
+        self.update();
+    }
+
+    /// The fraction of the removed rows that the decoder can skip, 0 before
+    /// the filter removed a row.
+    fn skippable_fraction(&self) -> f64 {
+        let removed = self.removed_rows.load(Ordering::Relaxed);
+        if removed == 0 {
+            return 0.0;
+        }
+        self.skippable_rows.load(Ordering::Relaxed) as f64 / removed as f64
     }
 
-    fn update(&self, ns_per_byte: f64) {
-        self.saving
-            .set_ns_per_row(ns_per_byte * self.unread_bytes_per_row);
+    /// Updates the measured saving with the current decode speed and
+    /// skippable fraction.
+    fn update(&self) {
+        self.measured.set_ns_per_row(
+            self.skippable_fraction()
+                * self.unread_bytes_per_row
+                * self.decode_cost.ns_per_byte(),
+        );
     }
 }
 
@@ -188,7 +262,7 @@ pub(crate) struct OptionalFilterSavings {
     /// Compressed bytes for each row of the output columns of the file.
     output_bytes_per_row: f64,
     /// The saving of each gated optional filter of the file.
-    filters: Vec<OptionalFilterSaving>,
+    filters: Vec<Arc<OptionalFilterSaving>>,
 }
 
 impl OptionalFilterSavings {
@@ -198,7 +272,7 @@ impl OptionalFilterSavings {
         decode_cost: Arc<DecodeCost>,
         metadata: &ParquetMetaData,
         output_projection: &ProjectionMask,
-        filters: Vec<OptionalFilterSaving>,
+        filters: Vec<Arc<OptionalFilterSaving>>,
     ) -> Option<Self> {
         if filters.is_empty() {
             return None;
@@ -214,7 +288,8 @@ impl OptionalFilterSavings {
     }
 
     /// Records that the decoder produced an output batch of `rows` rows in
-    /// `elapsed`, and updates the savings.
+    /// `elapsed`, and updates the savings. Call only for the batches of row
+    /// groups where the row filter removed no rows, see [`DecodeCost`].
     pub(crate) fn record_output_batch(&self, rows: usize, elapsed: Duration) {
         let bytes = rows as f64 * self.output_bytes_per_row;
         if bytes < 1.0 {
@@ -222,9 +297,8 @@ impl OptionalFilterSavings {
         }
         self.decode_cost
             .record(duration_nanos(elapsed), bytes as u64);
-        let ns_per_byte = self.decode_cost.ns_per_byte();
         for filter in &self.filters {
-            filter.update(ns_per_byte);
+            filter.update();
         }
     }
 }
diff --git a/datafusion/datasource-parquet/src/push_decoder.rs b/datafusion/datasource-parquet/src/push_decoder.rs
index ab279b1678920..42252c38bd024 100644
--- a/datafusion/datasource-parquet/src/push_decoder.rs
+++ b/datafusion/datasource-parquet/src/push_decoder.rs
@@ -405,6 +405,19 @@ pub(crate) struct RowFilterContext {
     /// [`crate::filter_placement`]). `None` when it is disabled or when no
     /// conjunct needs a decision.
     pub(crate) placement: Option<FilePlacement>,
+    /// See [`Self::start_reader`].
+    decode_measurement: DecodeMeasurement,
+}
+
+/// Which output batches measure the decode speed, see
+/// [`RowFilterContext::start_reader`].
+struct DecodeMeasurement {
+    /// Value of `pushdown_rows_pruned` when the decoder handed out the last
+    /// reader.
+    rows_pruned: usize,
+    /// True if the row filter removed no rows of the row group of the
+    /// current reader.
+    unfiltered: bool,
 }
 
 impl RowFilterContext {
@@ -445,6 +458,10 @@ impl RowFilterContext {
                                 .collect(),
                         )
                     });
+                    let decode_measurement = DecodeMeasurement {
+                        rows_pruned: file_metrics.pushdown_rows_pruned.value(),
+                        unfiltered: true,
+                    };
                     Self {
                         prebuilt: PrebuiltRowFilterCandidateList::new(prebuilt),
                         reorder_predicates,
@@ -452,6 +469,7 @@ impl RowFilterContext {
                         max_predicate_cache_size,
                         optional_savings,
                         placement: None,
+                        decode_measurement,
                     }
                 });
                 (context, rejected)
@@ -487,10 +505,26 @@ impl RowFilterContext {
         self.placement.as_ref()
     }
 
+    /// Call when the decoder hands out the reader of the next row group. The
+    /// decoder evaluated the row filter of the row group before: if the row
+    /// filter removed rows, the decode time of the output batches of the
+    /// reader is not measured. A row filter that removes rows spread over
+    /// the row group makes each decoded output row much more expensive
+    /// (the decoder decodes all pages and drops most rows), thus the
+    /// measured decode speed would be too slow, and the saving of a removed
+    /// row too large (see [`crate::optional_filter`]).
+    pub(crate) fn start_reader(&mut self) {
+        let rows_pruned = self.file_metrics.pushdown_rows_pruned.value();
+        let measurement = &mut self.decode_measurement;
+        measurement.unfiltered = rows_pruned == measurement.rows_pruned;
+        measurement.rows_pruned = rows_pruned;
+    }
+
     /// True if [`Self::record_output_batch`] needs the decode time of the
-    /// output batches.
+    /// output batches of the current reader.
     pub(crate) fn measures_decode(&self) -> bool {
-        self.optional_savings.is_some() || self.placement.is_some()
+        (self.optional_savings.is_some() || self.placement.is_some())
+            && self.decode_measurement.unfiltered
     }
 
     /// Records that the decoder produced an output batch of `rows` rows in
@@ -795,6 +829,9 @@ impl PushDecoderStreamState {
                     if let Some(entry) = self.rg_plan.pop_front() {
                         self.byte_progress.credit(entry.bytes);
                     }
+                    if let Some(ctx) = self.row_filter_context.as_mut() {
+                        ctx.start_reader();
+                    }
                     self.active_reader = Some(reader);
                 }
                 Ok(DecodeResult::Finished) => return self.finish(),
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 59466d4d7f37c..0653f1a72b8ed 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -136,6 +136,10 @@ pub(crate) struct DatafusionArrowPredicate {
     /// (see [`crate::filter_placement`]). `None` if the placement does not
     /// manage the conjunct.
     placement_stats: Option<Arc<ConjunctStats>>,
+    /// The measured saving of the gate: counts the rows that the filter
+    /// removes and that the decoder can skip. `None` for all filters
+    /// without a gate, and when the output projection is not known.
+    saving: Option<Arc<OptionalFilterSaving>>,
 }
 
 /// The [`OptionalFilterGate`] of one optional filter in one file, with its
@@ -225,6 +229,7 @@ impl DatafusionArrowPredicate {
             time,
             gate: None,
             placement_stats: None,
+            saving: None,
         })
     }
 }
@@ -263,6 +268,9 @@ impl ArrowPredicate for DatafusionArrowPredicate {
                 if let Some(stats) = &self.placement_stats {
                     stats.record_evaluation(&bool_arr);
                 }
+                if let Some(saving) = &self.saving {
+                    saving.record_evaluation(&bool_arr);
+                }
                 if let (Some(gate), Some(start)) = (gate.as_mut(), start) {
                     let elapsed = gate.gate.clock().now_nanos().saturating_sub(start);
                     gate.record(num_rows, num_matched, Duration::from_nanos(elapsed));
@@ -586,7 +594,7 @@ pub(crate) struct PrebuiltRowFilterCandidate {
     gate: Option<SharedOptionalFilterGate>,
     /// The measured saving that the gate reads, if the output projection of
     /// the file is known. See [`crate::optional_filter`].
-    saving: Option<OptionalFilterSaving>,
+    saving: Option<Arc<OptionalFilterSaving>>,
     /// Position of the conjunct in the root `AND` chain of the file
     /// predicate.
     position: usize,
@@ -603,7 +611,7 @@ impl PrebuiltRowFilterCandidate {
     }
 
     /// The measured saving of a gated optional filter.
-    pub(crate) fn optional_saving(&self) -> Option<&OptionalFilterSaving> {
+    pub(crate) fn optional_saving(&self) -> Option<&Arc<OptionalFilterSaving>> {
         self.saving.as_ref()
     }
 
@@ -789,11 +797,11 @@ pub(crate) fn prebuild_row_filter_candidates(
                     output.leaf_included(leaf)
                         && !candidate.projection_mask.leaf_included(leaf)
                 });
-                let saving = OptionalFilterSaving::new(
+                let saving = Arc::new(OptionalFilterSaving::new(
                     unread_bytes_per_row,
-                    &optional.options.decode_cost,
-                );
-                gate = gate.with_measured_saving(Arc::clone(&saving.saving));
+                    Arc::clone(&optional.options.decode_cost),
+                ));
+                gate = gate.with_measured_saving(Arc::clone(saving.measured()));
                 candidate.saving = Some(saving);
             }
             candidate.gate = Some(Arc::new(Mutex::new(OptionalFilterGateState {
@@ -897,6 +905,7 @@ pub(crate) fn row_filter_from_prebuilt<'a>(
                 time: time.clone(),
                 gate: candidate.gate.clone(),
                 placement_stats: candidate.placement_stats.clone(),
+                saving: candidate.saving.clone(),
             }) as Box<dyn ArrowPredicate>
         })
         .collect();
@@ -2970,7 +2979,9 @@ mod optional_filter_tests {
 
     /// With the output projection of the file, the gate of an optional
     /// filter adds the decode time of the output columns that the filter
-    /// does not read to the configured minimum saving.
+    /// does not read, for the removed rows that the decoder can skip, to the
+    /// configured minimum saving. It adds the row filter stage cost to the
+    /// evaluation time.
     #[test]
     fn saving_includes_decode_of_unread_output_columns() {
         use crate::optional_filter::{
@@ -3006,9 +3017,25 @@ mod optional_filter_tests {
             compressed_bytes_per_row(&metadata, |leaf| leaf == 0 || leaf == 2),
             unread
         );
-        // Before any measurement: the default decode speed.
+        // Before the filter removed any row: only the configured minimum,
+        // and the stage cost of a row filter.
         let gate = candidates[0].gate.as_ref().unwrap();
-        let expected = 5.0 + DEFAULT_DECODE_NS_PER_BYTE * unread;
+        assert_eq!(gate.lock().gate.saving_ns_per_row(), 5.0);
+        assert_eq!(
+            gate.lock().gate.overhead_ns_per_row(),
+            crate::row_filter_cost::ROW_FILTER_STAGE_NS_PER_ROW
+        );
+
+        // The filter removes one full window of 64 rows (skippable) and 32
+        // rows spread over a second window (not skippable): 2/3 of the
+        // removed rows are skippable.
+        let result: BooleanArray =
+            (0..128).map(|i| Some(i >= 64 && i % 2 == 0)).collect();
+        saving.record_evaluation(&result);
+        let expected = 5.0 + 2.0 / 3.0 * DEFAULT_DECODE_NS_PER_BYTE * unread;
+        assert!((gate.lock().gate.saving_ns_per_row() - expected).abs() < 1e-9);
+        // A filter result where all rows pass changes nothing.
+        saving.record_evaluation(&BooleanArray::from(vec![true; 64]));
         assert!((gate.lock().gate.saving_ns_per_row() - expected).abs() < 1e-9);
 
         // Output batches of 1000 rows that take 4 ns for each compressed
@@ -3018,14 +3045,14 @@ mod optional_filter_tests {
             Arc::clone(&options.decode_cost),
             &metadata,
             &output,
-            vec![saving.clone()],
+            vec![Arc::clone(saving)],
         )
         .unwrap();
-        let nanos = (4.0 * 1000.0 * output_bytes) as u64;
+        let nanos = (2.0 * 1000.0 * output_bytes) as u64;
         savings.record_output_batch(1000, Duration::from_nanos(nanos));
         let ns_per_byte = options.decode_cost.ns_per_byte();
-        assert!((ns_per_byte - 4.0).abs() < 0.01, "{ns_per_byte}");
-        let expected = 5.0 + ns_per_byte * unread;
+        assert!((ns_per_byte - 2.0).abs() < 0.01, "{ns_per_byte}");
+        let expected = 5.0 + 2.0 / 3.0 * ns_per_byte * unread;
         assert!((gate.lock().gate.saving_ns_per_row() - expected).abs() < 1e-9);
 
         // No gated optional filter: nothing to measure.
diff --git a/datafusion/datasource-parquet/src/row_filter_cost.rs b/datafusion/datasource-parquet/src/row_filter_cost.rs
new file mode 100644
index 0000000000000..4b9c7fcc34247
--- /dev/null
+++ b/datafusion/datasource-parquet/src/row_filter_cost.rs
@@ -0,0 +1,102 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! What a predicate of the Parquet `RowFilter` costs and what it saves,
+//! in addition to its evaluation time.
+//!
+//! * The *cost*: each row filter stage has a fixed cost for each row that it
+//!   evaluates ([`ROW_FILTER_STAGE_NS_PER_ROW`]).
+//! * The *saving*: the decoder does not decode the other columns of the
+//!   rows that the predicate removes only when these rows make long runs.
+//!   [`skippable_rows`] counts the rows in such runs.
+
+use arrow::array::{Array, BooleanArray};
+use arrow::buffer::BooleanBuffer;
+
+/// The fixed cost of a row filter stage for each row that it evaluates,
+/// in nanoseconds. It does not depend on the predicate: the decoder builds a
+/// `RowSelection` from the result, skips or reads the records of the other
+/// columns with it (`skip_records`), fills the predicate cache
+/// (`CachedArrayReader`) and fetches the columns of the stage before the
+/// other columns. Profiles of TPC-H SF1 Q6, Q12 and Q4 on an Apple M-series
+/// laptop showed 4 to 16 ns for each row and stage, about 20 times the time
+/// of the predicate. The post-scan filter has none of these costs.
+pub(crate) const ROW_FILTER_STAGE_NS_PER_ROW: f64 = 8.0;
+
+/// Number of rows in one window of [`skippable_rows`].
+///
+/// The Parquet decoder skips the rows that a row filter removes only when
+/// the removed rows make long runs. Its default selection policy
+/// (`RowSelectionPolicy::Auto { threshold: 32 }`) decodes all rows and then
+/// filters them when the runs are shorter than 32 rows on average. A window
+/// of 64 rows is one `u64` of the filter result.
+pub(crate) const SKIP_WINDOW_ROWS: usize = 64;
+
+/// Rows of `result` in windows of [`SKIP_WINDOW_ROWS`] rows where no row
+/// passes (a `null` does not pass). These are the rows that a row filter
+/// lets the decoder skip for the other columns. A partial window at the end
+/// is counted if no row in it passes.
+pub(crate) fn skippable_rows(result: &BooleanArray) -> usize {
+    let values = result.values();
+    match result.nulls() {
+        Some(nulls) => skippable_in(&(values & nulls.inner())),
+        None => skippable_in(values),
+    }
+}
+
+/// Rows of `passed` in windows of [`SKIP_WINDOW_ROWS`] rows where no bit is
+/// set, see [`skippable_rows`].
+pub(crate) fn skippable_in(passed: &BooleanBuffer) -> usize {
+    let chunks = passed.bit_chunks();
+    let full = chunks.iter().filter(|chunk| *chunk == 0).count() * SKIP_WINDOW_ROWS;
+    let remainder = if chunks.remainder_len() > 0 && chunks.remainder_bits() == 0 {
+        chunks.remainder_len()
+    } else {
+        0
+    };
+    full + remainder
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn bools(values: impl IntoIterator<Item = Option<bool>>) -> BooleanArray {
+        values.into_iter().collect()
+    }
+
+    #[test]
+    fn skippable_rows_counts_empty_windows() {
+        // Window 0 (rows 0..64) has one passing row, window 1 has none, the
+        // partial window (rows 128..160) has none.
+        let mut values = vec![Some(false); 160];
+        values[10] = Some(true);
+        assert_eq!(skippable_rows(&bools(values.clone())), 64 + 32);
+
+        // A null does not pass.
+        values[70] = None;
+        assert_eq!(skippable_rows(&bools(values.clone())), 64 + 32);
+
+        // One passing row in the partial window.
+        values[150] = Some(true);
+        assert_eq!(skippable_rows(&bools(values)), 64);
+
+        // Scattered passing rows: no window is empty.
+        let scattered = (0..640).map(|i| Some(i % 50 == 0));
+        assert_eq!(skippable_rows(&bools(scattered)), 0);
+    }
+}

From 9af896f17cc8453c0b825aaaa57aff9774aa8635 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 11:53:09 -0500
Subject: [PATCH 25/42] feat(parquet): share the pauses of optional filter
 gates in a scan

Each file of a scan made its own gates, thus each file paid for a window
of each optional filter before the gate could pause it, and each gate
probed on its own (TPC-H Q9: five join filters that remove no rows and a
CASE routing filter at 83 ns for each row, in each of 12 files).

`ParquetSource` now keeps an `OptionalFilterSites` map (reset when the
predicate changes, like the decode speed): one `SharedGateVerdict` for
each optional filter, keyed by the expression id of the filter. All gates
of the filter in all files and partitions of the scan use it. A filter
without an expression id does not share its pauses.

PR: #25682

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../datasource-parquet/src/opener/mod.rs      |  2 +-
 .../datasource-parquet/src/optional_filter.rs | 60 +++++++++++++++++--
 .../datasource-parquet/src/row_filter.rs      | 46 ++++++++++++--
 datafusion/datasource-parquet/src/source.rs   | 16 ++++-
 4 files changed, 112 insertions(+), 12 deletions(-)

diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index a1646a5b4e2e4..bf71d25077fea 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -6130,7 +6130,7 @@ mod test {
             let options = || OptionalFilterOptions {
                 mode: OptionalFilterMode::Adaptive,
                 gate_config: OptionalFilterGateConfig::default(),
-                decode_cost: Arc::default(),
+                ..Default::default()
             };
 
             let adaptive =
diff --git a/datafusion/datasource-parquet/src/optional_filter.rs b/datafusion/datasource-parquet/src/optional_filter.rs
index 8e10a1e18951f..9c4b6ccffa2e9 100644
--- a/datafusion/datasource-parquet/src/optional_filter.rs
+++ b/datafusion/datasource-parquet/src/optional_filter.rs
@@ -71,6 +71,7 @@
 //! [`OptionalFilterPhysicalExpr`]: datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr
 //! [`OptionalFilterGate`]: datafusion_physical_expr::optional_filter_gate::OptionalFilterGate
 
+use std::collections::HashMap;
 use std::sync::Arc;
 use std::sync::atomic::{AtomicU64, Ordering};
 use std::time::Duration;
@@ -78,10 +79,12 @@ use std::time::Duration;
 use arrow::array::BooleanArray;
 
 use datafusion_common::config::OptionalFilterMode;
+use datafusion_physical_expr::PhysicalExpr;
 use datafusion_physical_expr::filter_stats::duration_nanos;
 use datafusion_physical_expr::optional_filter_gate::{
-    MeasuredRowSaving, OptionalFilterGateConfig,
+    MeasuredRowSaving, OptionalFilterGateConfig, SharedGateVerdict,
 };
+use parking_lot::Mutex;
 use parquet::arrow::ProjectionMask;
 use parquet::file::metadata::ParquetMetaData;
 
@@ -303,12 +306,39 @@ impl OptionalFilterSavings {
     }
 }
 
+/// The shared pauses of the gates of the optional filters of one scan, one
+/// [`SharedGateVerdict`] for each optional filter, shared by all partitions
+/// and files of the scan.
+///
+/// The scan rewrites its predicate for each file, thus a filter is
+/// identified by its [`PhysicalExpr::expression_id`] (a
+/// `DynamicFilterPhysicalExpr` has one, and the rewrites keep it). The
+/// gates of a filter without an id do not share their pauses.
+#[derive(Debug, Default)]
+pub(crate) struct OptionalFilterSites {
+    verdicts: Mutex<HashMap<u64, Arc<SharedGateVerdict>>>,
+}
+
+impl OptionalFilterSites {
+    /// The shared verdict of the optional filter `filter` (the expression
+    /// inside the `OptionalFilterPhysicalExpr`), or `None` if it has no
+    /// expression id.
+    pub(crate) fn verdict_for(
+        &self,
+        filter: &Arc<dyn PhysicalExpr>,
+    ) -> Option<Arc<SharedGateVerdict>> {
+        let id = filter.expression_id()?;
+        Some(Arc::clone(self.verdicts.lock().entry(id).or_default()))
+    }
+}
+
 /// The optional filter settings of one scan, see the [module
 /// documentation](self).
 ///
-/// Each file makes its own gates: gates do not share state between files
-/// or partitions. Only the decode speed ([`DecodeCost`]) is shared by all
-/// partitions and files of the scan.
+/// Each file makes its own gates. The gates of one optional filter share
+/// their pauses in all partitions and files of the scan
+/// ([`OptionalFilterSites`]), and the decode speed ([`DecodeCost`]) is
+/// shared by all partitions and files of the scan.
 #[derive(Debug, Clone, Default)]
 pub(crate) struct OptionalFilterOptions {
     /// How the row filter handles optional conjuncts.
@@ -317,4 +347,26 @@ pub(crate) struct OptionalFilterOptions {
     pub(crate) gate_config: OptionalFilterGateConfig,
     /// The decode speed of the output columns of the scan.
     pub(crate) decode_cost: Arc<DecodeCost>,
+    /// The shared pauses of the gates of each optional filter.
+    pub(crate) sites: Arc<OptionalFilterSites>,
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use datafusion_physical_expr::expressions::{DynamicFilterPhysicalExpr, lit};
+
+    #[test]
+    fn sites_share_verdicts_by_expression_id() {
+        let sites = OptionalFilterSites::default();
+        let dynamic: Arc<dyn PhysicalExpr> =
+            Arc::new(DynamicFilterPhysicalExpr::new(vec![], lit(true)));
+        let other: Arc<dyn PhysicalExpr> =
+            Arc::new(DynamicFilterPhysicalExpr::new(vec![], lit(true)));
+        let first = sites.verdict_for(&dynamic).unwrap();
+        assert!(Arc::ptr_eq(&first, &sites.verdict_for(&dynamic).unwrap()));
+        assert!(!Arc::ptr_eq(&first, &sites.verdict_for(&other).unwrap()));
+        // A filter without an expression id has no shared verdict.
+        assert!(sites.verdict_for(&lit(true)).is_none());
+    }
 }
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 0653f1a72b8ed..8d76f57ede953 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -694,9 +694,12 @@ pub(crate) struct OptionalFilterRowFilterContext<'a> {
 ///   predicate.
 /// * [`OptionalFilterMode::Adaptive`]: each optional conjunct is a separate
 ///   candidate with an [`OptionalFilterGate`], which skips the conjunct while
-///   it costs more than it saves. Each file has its own gates. Optional
-///   candidates come after all required
-///   candidates, see [`row_filter_from_prebuilt`].
+///   it costs more than it saves. Each file has its own gates, and the
+///   gates of one filter share their pauses in all files and partitions of
+///   the scan (see
+///   [`OptionalFilterSites`](crate::optional_filter::OptionalFilterSites)).
+///   Optional candidates come after all required candidates, see
+///   [`row_filter_from_prebuilt`].
 ///
 /// An optional conjunct that cannot be evaluated as an `ArrowPredicate` for
 /// this file (for example because of schema evolution) is not used, in all
@@ -790,6 +793,13 @@ pub(crate) fn prebuild_row_filter_candidates(
                 Arc::clone(&candidate.physical_expr),
                 optional.options.gate_config,
             );
+            // The gates of the filter in the other files and partitions of
+            // the scan share their pauses.
+            if let Some(verdict) =
+                optional.options.sites.verdict_for(&candidate.source_expr)
+            {
+                gate = gate.with_shared_verdict(verdict);
+            }
             // A removed row saves the decode of the output columns that the
             // filter does not read.
             if let Some(output) = optional.output_projection {
@@ -2676,7 +2686,7 @@ mod optional_filter_tests {
                 min_saving_ns_per_row,
                 ..Default::default()
             },
-            decode_cost: Arc::default(),
+            ..Default::default()
         }
     }
 
@@ -2925,8 +2935,8 @@ mod optional_filter_tests {
         assert_eq!(rows_adaptive, rows_always);
         assert_eq!(file_metrics.pushdown_rows_matched.value(), rows_adaptive);
 
-        // Gates do not share state: the gate for the next file starts to
-        // evaluate the filter.
+        // A filter without an expression id does not share its pauses: the
+        // gate for the next file starts to evaluate the filter.
         let candidates = prebuild(
             &predicate,
             &file_schema,
@@ -2938,6 +2948,30 @@ mod optional_filter_tests {
         assert!(!gate.lock().gate.is_paused());
     }
 
+    /// The gates of a dynamic filter share their pauses: the gate for the
+    /// next file starts paused, without a window of its own.
+    #[test]
+    fn adaptive_gates_of_dynamic_filter_share_pauses() {
+        let (file, metadata, file_schema) = test_file();
+        let dynamic: Arc<dyn PhysicalExpr> = Arc::new(DynamicFilterPhysicalExpr::new(
+            vec![col("b", &file_schema).unwrap()],
+            col_op("b", Operator::GtEq, 0, &file_schema),
+        ));
+        let predicate = optional(dynamic);
+        let metrics = ExecutionPlanMetricsSet::new();
+        let file_metrics = ParquetFileMetrics::new(0, FILENAME, &metrics);
+        let options = options(OptionalFilterMode::Adaptive);
+        let candidates =
+            prebuild(&predicate, &file_schema, &metadata, &options, &metrics);
+        assert!(!candidates[0].gate.as_ref().unwrap().lock().gate.is_paused());
+        read_with_row_filter(&file, &candidates, &file_metrics);
+        assert!(metric(&metrics, "optional_filter_pauses") > 0);
+
+        let candidates =
+            prebuild(&predicate, &file_schema, &metadata, &options, &metrics);
+        assert!(candidates[0].gate.as_ref().unwrap().lock().gate.is_paused());
+    }
+
     #[test]
     fn adaptive_keeps_selective_optional_filter() {
         let (file, metadata, file_schema) = test_file();
diff --git a/datafusion/datasource-parquet/src/source.rs b/datafusion/datasource-parquet/src/source.rs
index 5bb782debfff4..bd333fe10d091 100644
--- a/datafusion/datasource-parquet/src/source.rs
+++ b/datafusion/datasource-parquet/src/source.rs
@@ -27,7 +27,7 @@ use crate::filter_placement::{PlacementOptions, PlacementSites};
 use crate::opener::ParquetMorselizer;
 use crate::opener::build_pruning_predicates;
 use crate::opener::build_virtual_columns_state;
-use crate::optional_filter::{DecodeCost, OptionalFilterOptions};
+use crate::optional_filter::{DecodeCost, OptionalFilterOptions, OptionalFilterSites};
 use crate::row_filter::can_expr_be_pushed_down_with_schemas;
 use arrow_schema::Fields;
 use arrow_schema::extension::ExtensionType;
@@ -344,6 +344,10 @@ pub struct ParquetSource {
     /// partitions and files of this scan. The gates of the optional filters
     /// use it. Replaced each time the predicate changes.
     optional_filter_decode_cost: Arc<DecodeCost>,
+    /// The shared pauses of the gates of each optional filter, shared by all
+    /// partitions and files of this scan. Replaced each time the predicate
+    /// changes.
+    optional_filter_sites: Arc<OptionalFilterSites>,
     /// `datafusion.execution.adaptive_filter_placement`, see
     /// [`crate::filter_placement`].
     adaptive_filter_placement: bool,
@@ -382,6 +386,7 @@ impl ParquetSource {
             optional_filter_mode: OptionalFilterMode::default(),
             optional_filter_gate_config: OptionalFilterGateConfig::default(),
             optional_filter_decode_cost: Arc::default(),
+            optional_filter_sites: Arc::default(),
             adaptive_filter_placement: false,
             filter_placement_sites: Arc::default(),
         }
@@ -417,6 +422,7 @@ impl ParquetSource {
         let mut conf = self.clone();
         conf.predicate = Some(Arc::clone(&predicate));
         conf.optional_filter_decode_cost = Arc::default();
+        conf.optional_filter_sites = Arc::default();
         conf.filter_placement_sites = Arc::default();
         conf
     }
@@ -720,6 +726,7 @@ impl FileSource for ParquetSource {
                 mode: self.optional_filter_mode,
                 gate_config: self.optional_filter_gate_config,
                 decode_cost: Arc::clone(&self.optional_filter_decode_cost),
+                sites: Arc::clone(&self.optional_filter_sites),
             },
             filter_placement: PlacementOptions::new(
                 self.adaptive_filter_placement && self.pushdown_filters(),
@@ -958,6 +965,7 @@ impl FileSource for ParquetSource {
         source.optional_filter_mode = config.execution.optional_filter_mode;
         source.optional_filter_gate_config = (&config.execution).into();
         source.optional_filter_decode_cost = Arc::default();
+        source.optional_filter_sites = Arc::default();
         source.adaptive_filter_placement = config.execution.adaptive_filter_placement;
         source.filter_placement_sites = Arc::default();
         let source = Arc::new(source);
@@ -1228,6 +1236,7 @@ impl FileSource for ParquetSource {
             optional_filter_gate_config: _,
             // Runtime state, recreated when the source is decoded.
             optional_filter_decode_cost: _,
+            optional_filter_sites: _,
             // Not serialized: the decoded source does not use adaptive
             // filter placement.
             adaptive_filter_placement: _,
@@ -2404,5 +2413,10 @@ mod tests {
             &updated.optional_filter_decode_cost,
             &source.optional_filter_decode_cost
         ));
+        // And the shared pauses of the gates are new.
+        assert!(!Arc::ptr_eq(
+            &updated.optional_filter_sites,
+            &source.optional_filter_sites
+        ));
     }
 }

From e3fb0f00ebfc832b1b78b54e1597924c77b927e0 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 12:16:49 -0500
Subject: [PATCH 26/42] feat(parquet): place and order required and optional
 conjuncts with one model

The adaptive filter placement started each required conjunct in the
post-scan filter, but it put each optional filter in the row filter. The
row filter runs before the post-scan filter, thus the optional filters saw
all rows first (TPC-DS Q13: a CASE routing filter on 2.69M rows, where the
required filters leave 89K rows; ClickBench Q25: the TopK filter on the
87% of rows that `SearchPhrase <> ''` removes).

Now required and optional conjuncts use the same model and the same
pooled measurements. The only difference is the action set: an optional
conjunct is skipped while its gate is paused.

* Placement: each conjunct starts in the post-scan filter and moves to the
  row filter on measured evidence (skippable rows times unread output
  bytes against the stage cost, the fetch latency and the second decode).
  A conjunct that reads all output columns has no unread bytes, thus no
  decode saving.
* Order: in each stage, the conjuncts run by measured rows removed for
  each nanosecond (`FilterCost::rows_removed_per_nano`, the key of the
  adaptive conjunct order of `FilterExec`). Unmeasured conjuncts run first
  in the written order, so that they are measured on all rows of their
  stage. The pooled statistics now include the evaluation time. The
  statistics prior is recorded for optional conjuncts too.
* The post-scan filter asks the gate of an optional conjunct for each
  batch. The measured saving and stage cost of the gate apply only while
  the conjunct is a row filter predicate.
* A placement change rebuilds the decoder only when its projection mask
  or its `RowFilter` changes. When only the post-scan filter changes (for
  example an optional conjunct on output columns that its gate pauses and
  probes), the stream swaps the post-scan filter.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/decoder_projection.rs                 |  95 +++++-
 .../src/filter_placement/mod.rs               | 304 ++++++++++--------
 .../src/filter_placement/model.rs             | 121 ++++++-
 .../src/filter_placement/stats.rs             |  61 +++-
 .../datasource-parquet/src/opener/mod.rs      |  60 ++--
 .../datasource-parquet/src/optional_filter.rs |  53 ++-
 .../datasource-parquet/src/push_decoder.rs    |  68 ++--
 .../datasource-parquet/src/row_filter.rs      | 206 ++++++++----
 8 files changed, 694 insertions(+), 274 deletions(-)

diff --git a/datafusion/datasource-parquet/src/decoder_projection.rs b/datafusion/datasource-parquet/src/decoder_projection.rs
index 714866d97d329..ffc58de794a71 100644
--- a/datafusion/datasource-parquet/src/decoder_projection.rs
+++ b/datafusion/datasource-parquet/src/decoder_projection.rs
@@ -32,6 +32,7 @@
 //! which calls [`map`](DecoderProjection::map) on every decoded batch.
 
 use std::sync::Arc;
+use std::time::Duration;
 
 use arrow::array::{Array, BooleanArray, RecordBatch, RecordBatchOptions};
 use arrow::compute::kernels::boolean::and;
@@ -39,7 +40,10 @@ use arrow::compute::kernels::filter::{filter_record_batch, prep_null_mask_filter
 use arrow::datatypes::SchemaRef;
 
 use datafusion_common::cast::as_boolean_array;
+use datafusion_common::instant::Instant;
 use datafusion_common::{Result, internal_err};
+use datafusion_physical_expr::filter_stats::duration_nanos;
+use datafusion_physical_expr::optional_filter_gate::GateDecision;
 use datafusion_physical_expr::projection::{ProjectionExprs, Projector};
 use datafusion_physical_expr::split_conjunction;
 use datafusion_physical_expr::utils::{collect_columns, reassign_expr_columns};
@@ -54,6 +58,7 @@ use crate::ParquetFileMetrics;
 use crate::filter_placement::{ConjunctStats, StageSelection};
 use crate::opener::{VirtualColumnsState, append_fields};
 use crate::projection_read_plan::build_projection_read_plan;
+use crate::row_filter::SharedOptionalFilterGate;
 
 /// Stream-schema column indices referenced by `projection`, in ascending
 /// order, or `None` when the projection already reads every column of
@@ -180,10 +185,20 @@ impl PostScanFilter {
             PostScanConjunct {
                 expr: conjunct,
                 stats,
+                gate,
             },
         ) in self.conjuncts.iter().enumerate()
         {
             let rows_in = working.num_rows();
+            // An optional conjunct that its gate skips lets all rows pass.
+            let mut gate = gate.as_ref().map(|gate| gate.lock());
+            if let Some(gate) = gate.as_mut()
+                && gate.begin_batch(rows_in) == GateDecision::Skip
+            {
+                continue;
+            }
+            let start = gate.as_ref().map(|gate| gate.now_nanos());
+            let measure_start = stats.as_ref().map(|_| Instant::now());
             let array = conjunct.evaluate(&working)?.into_array(rows_in)?;
             let Ok(mask) = as_boolean_array(array.as_ref()) else {
                 return internal_err!(
@@ -196,9 +211,16 @@ impl PostScanFilter {
                 Some(_) => prep_null_mask_filter(mask),
                 None => mask.clone(),
             };
-            if let (Some(stages), Some(stats)) = (stages.as_ref(), stats) {
-                stages.record(stats, &mask);
+            if let (Some(stages), Some(stats), Some(measure_start)) =
+                (stages.as_ref(), stats, measure_start)
+            {
+                stages.record(stats, &mask, duration_nanos(measure_start.elapsed()));
+            }
+            if let (Some(gate), Some(start)) = (gate.as_mut(), start) {
+                let elapsed = gate.now_nanos().saturating_sub(start);
+                gate.record(rows_in, mask.true_count(), Duration::from_nanos(elapsed));
             }
+            drop(gate);
             // An all-true conjunct leaves the accumulated selection untouched.
             if mask.true_count() == rows_in {
                 continue;
@@ -259,11 +281,19 @@ pub(crate) struct PostScanConjunct {
     /// The measurements of the adaptive filter placement, if it manages the
     /// conjunct. The post-scan filter records each evaluation.
     pub(crate) stats: Option<Arc<ConjunctStats>>,
+    /// The gate of an optional conjunct: the post-scan filter evaluates the
+    /// conjunct only on the batches that the gate does not skip. `None` for
+    /// a required conjunct.
+    pub(crate) gate: Option<SharedOptionalFilterGate>,
 }
 
 impl From<Arc<dyn PhysicalExpr>> for PostScanConjunct {
     fn from(expr: Arc<dyn PhysicalExpr>) -> Self {
-        Self { expr, stats: None }
+        Self {
+            expr,
+            stats: None,
+            gate: None,
+        }
     }
 }
 
@@ -478,6 +508,15 @@ impl DecoderProjection {
             for conjunct in post_scan_conjuncts {
                 let expr =
                     reassign_expr_columns(Arc::clone(&conjunct.expr), &stream_schema)?;
+                // The gate of an optional conjunct decides for the whole
+                // conjunct, thus it is not split.
+                if conjunct.gate.is_some() {
+                    rebased.push(PostScanConjunct {
+                        expr,
+                        ..conjunct.clone()
+                    });
+                    continue;
+                }
                 let pieces = split_conjunction(&expr);
                 // The measurements belong to the whole conjunct. The adaptive
                 // filter placement only manages conjuncts of the root `AND`
@@ -490,6 +529,7 @@ impl DecoderProjection {
                 rebased.extend(pieces.into_iter().map(|piece| PostScanConjunct {
                     expr: Arc::clone(piece),
                     stats: stats.clone(),
+                    gate: None,
                 }));
             }
             Some(PostScanFilter {
@@ -662,10 +702,12 @@ mod tests {
                 PostScanConjunct {
                     expr: every_fourth,
                     stats: Some(Arc::clone(&first)),
+                    gate: None,
                 },
                 PostScanConjunct {
                     expr: upper_half,
                     stats: Some(Arc::clone(&second)),
+                    gate: None,
                 },
             ],
             rows_pruned: Count::new(),
@@ -688,6 +730,53 @@ mod tests {
         );
     }
 
+    /// A gated optional conjunct sees only the rows that the conjuncts
+    /// before it let pass, and a paused gate skips it.
+    #[test]
+    fn gated_conjunct_sees_rows_after_earlier_conjuncts() {
+        use crate::metrics::OptionalFilterMetrics;
+        use crate::row_filter::OptionalFilterGateState;
+        use datafusion_physical_expr::filter_stats::ManualClock;
+        use datafusion_physical_expr::optional_filter_gate::{
+            OptionalFilterGate, OptionalFilterGateConfig,
+        };
+        use datafusion_physical_plan::metrics::ExecutionPlanMetricsSet;
+
+        // The first conjunct keeps the rows with `a > 99`. The optional
+        // conjunct `b > 99` removes the same rows: after the first conjunct
+        // it removes nothing, thus its gate pauses it after its first window
+        // of 2 batches.
+        let optional = gt("b", 1, 99);
+        let gate = OptionalFilterGateState::shared(
+            OptionalFilterGate::new(
+                Arc::clone(&optional),
+                OptionalFilterGateConfig::default(),
+            )
+            .with_clock(Arc::new(ManualClock::new())),
+            OptionalFilterMetrics::new(&ExecutionPlanMetricsSet::new(), 0, "file"),
+        );
+        let filter = PostScanFilter {
+            conjuncts: vec![
+                PostScanConjunct::from(gt("a", 0, 99)),
+                PostScanConjunct {
+                    expr: optional,
+                    stats: None,
+                    gate: Some(Arc::clone(&gate)),
+                },
+            ],
+            rows_pruned: Count::new(),
+            rows_matched: Count::new(),
+            eval_time: Time::new(),
+        };
+        let input = || batch((0..200).map(Some).collect());
+        for _ in 0..2 {
+            assert_eq!(survivors(&filter, input()).len(), 100);
+        }
+        assert!(gate.lock().is_paused());
+        // The paused gate skips the optional conjunct: same result.
+        assert_eq!(survivors(&filter, input()).len(), 100);
+    }
+
     #[test]
     fn compacts_between_conjuncts_without_losing_rows() {
         // 100 rows; `a > 79` keeps 20 (20% — below the compaction threshold, so
diff --git a/datafusion/datasource-parquet/src/filter_placement/mod.rs b/datafusion/datasource-parquet/src/filter_placement/mod.rs
index b9f3158316063..51b9c9e6901e7 100644
--- a/datafusion/datasource-parquet/src/filter_placement/mod.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/mod.rs
@@ -24,24 +24,32 @@
 //! | Placement | Where | For |
 //! |---|---|---|
 //! | [`Placement::RowFilter`] | A predicate of the Parquet `RowFilter`: the decoder skips the other columns of the rows that it removes | Required and optional conjuncts |
-//! | [`Placement::PostScan`] | The post-scan filter, on the decoded batches | Required conjuncts |
+//! | [`Placement::PostScan`] | The post-scan filter, on the decoded batches | Required and optional conjuncts |
 //! | [`Placement::Skip`] | Not evaluated, and its columns are not decoded | Optional conjuncts in the `adaptive` optional filter mode, while the gate is paused |
 //!
 //! A row filter is not always better. It saves decode time only when it
 //! removes long runs of rows, each row filter stage has a fixed cost for
 //! each row, and the decoder fetches the columns of each row filter
 //! predicate before the other columns (one more round trip for each row
-//! group). Thus a required conjunct starts in the post-scan filter and
-//! becomes a row filter only on measured evidence. See [`model`] for the
-//! decision.
+//! group). Thus a conjunct starts in the post-scan filter and becomes a row
+//! filter only on measured evidence. See [`model`] for the decision.
+//!
+//! Required and optional conjuncts use the same model and the same
+//! measurements. The only difference: an optional conjunct (in the
+//! `adaptive` optional filter mode) is skipped while its gate is paused.
+//! In each stage (the row filter and the post-scan filter), the conjuncts
+//! are evaluated in the order of their measured rows removed for each
+//! nanosecond ([`model::evaluation_order`]), thus a cheap conjunct that
+//! removes many rows runs before an expensive one, required or optional.
 //!
 //! [`FilePlacement`] decides at file open and again at each row group
-//! boundary. The stream then rebuilds the decoder
-//! (`ParquetPushDecoder::into_builder`) with the new `RowFilter` and
-//! projection mask, and rebuilds the post-scan filter. The rebuilt decoder
-//! keeps the row selections of the remaining row groups (for example from
-//! the page index). The measurements are pooled over all files and
-//! partitions of the scan ([`PlacementSites`]).
+//! boundary. When the decoder must change (its projection mask, or the
+//! predicates or order of the `RowFilter`), the stream rebuilds it
+//! (`ParquetPushDecoder::into_builder`). The rebuilt decoder keeps the row
+//! selections of the remaining row groups (for example from the page
+//! index). When only the post-scan filter changes, the stream rebuilds
+//! only the post-scan filter. The measurements are pooled over all files
+//! and partitions of the scan ([`PlacementSites`]).
 //!
 //! The placement of a file stops changing only when a change would change
 //! the schema of the decoded batches (possible with nested columns): the
@@ -64,11 +72,11 @@ use parquet::arrow::ProjectionMask;
 use parquet::file::metadata::ParquetMetaData;
 
 use crate::decoder_projection::PostScanConjunct;
-use crate::optional_filter::compressed_bytes_per_row;
+use crate::optional_filter::{OptionalFilterSaving, compressed_bytes_per_row};
 use crate::row_filter::{PrebuiltRowFilterCandidate, SharedOptionalFilterGate};
 
 pub(crate) use model::Placement;
-use model::{RequiredConjunctInputs, place_optional, place_required};
+use model::{ConjunctInputs, evaluation_order, place_optional, place_required};
 pub(crate) use stats::{ConjunctStats, PlacementSites, StageSelection};
 
 /// The adaptive placement settings of one scan.
@@ -105,27 +113,45 @@ impl PlacementOptions {
 struct ManagedConjunct {
     /// Index of the row filter candidate of the conjunct.
     candidate: usize,
-    kind: ConjunctKind,
+    /// The conjunct in terms of the file schema, for the post-scan filter
+    /// (the inner expression of an optional conjunct).
+    expr: Arc<dyn PhysicalExpr>,
+    /// The pooled measurements of the conjunct.
+    stats: Arc<ConjunctStats>,
+    /// Compressed bytes for each row of the output columns that the
+    /// conjunct does not read.
+    unread_output_bytes_per_row: f64,
+    /// Compressed bytes for each row of the output columns that the
+    /// conjunct reads.
+    read_output_bytes_per_row: f64,
+    /// `Some` for an optional conjunct with a gate (the `adaptive` optional
+    /// filter mode).
+    optional: Option<OptionalConjunct>,
     placement: Placement,
 }
 
+/// The parts of a managed optional conjunct.
 #[derive(Debug)]
-enum ConjunctKind {
-    Required {
-        /// The conjunct in terms of the file schema, for the post-scan
-        /// filter.
-        expr: Arc<dyn PhysicalExpr>,
-        stats: Arc<ConjunctStats>,
-        /// Compressed bytes for each row of the output columns that the
-        /// conjunct does not read.
-        unread_output_bytes_per_row: f64,
-        /// Compressed bytes for each row of the output columns that the
-        /// conjunct reads.
-        read_output_bytes_per_row: f64,
-    },
-    /// An optional conjunct with a gate (the `adaptive` optional filter
-    /// mode).
-    Optional { gate: SharedOptionalFilterGate },
+struct OptionalConjunct {
+    gate: SharedOptionalFilterGate,
+    /// The measured saving of the gate, which depends on the placement.
+    saving: Option<Arc<OptionalFilterSaving>>,
+}
+
+/// The placement and the evaluation order of all managed conjuncts of a
+/// file, see [`FilePlacement::snapshot`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct PlacementSnapshot {
+    placements: Vec<Placement>,
+    order: Vec<usize>,
+}
+
+/// A change of the placement at a row group boundary, see
+/// [`FilePlacement::decide`].
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) struct PlacementChange {
+    /// True if the predicates of the `RowFilter` or their order changed.
+    pub(crate) row_filter: bool,
 }
 
 /// The placement of the conjuncts of one file. See the [module
@@ -133,6 +159,9 @@ enum ConjunctKind {
 #[derive(Debug)]
 pub(crate) struct FilePlacement {
     conjuncts: Vec<ManagedConjunct>,
+    /// The evaluation order of `conjuncts` (indexes), see
+    /// [`model::evaluation_order`].
+    order: Vec<usize>,
     sites: Arc<PlacementSites>,
     metadata: Arc<ParquetMetaData>,
     batch_size: usize,
@@ -150,7 +179,7 @@ pub(crate) struct FilePlacement {
 impl FilePlacement {
     /// Returns `None` if no conjunct needs a placement decision.
     ///
-    /// Adds the pooled measurements of each required conjunct to its
+    /// Adds the pooled measurements of each managed conjunct to its
     /// candidate, so that the row filter records its evaluations. Makes the
     /// placement for the first row group, which has `first_row_group_rows`
     /// rows.
@@ -171,42 +200,37 @@ impl FilePlacement {
         let file_conjuncts = split_conjunction(predicate);
         let mut conjuncts = vec![];
         for (index, candidate) in candidates.iter_mut().enumerate() {
-            let kind = if let Some(gate) = candidate.gate() {
-                ConjunctKind::Optional {
-                    gate: Arc::clone(gate),
-                }
-            } else if is_optional_filter(candidate.source_expr())
-                || candidate.position() >= file_conjuncts.len()
+            if candidate.position() >= file_conjuncts.len()
+                || (candidate.gate().is_none()
+                    && is_optional_filter(candidate.source_expr()))
             {
                 // An optional conjunct in the `always` mode.
                 continue;
-            } else {
-                let stats = options.sites.stats_for(
-                    &file_conjuncts,
-                    candidate.position(),
-                    options.scan_conjunct_count,
-                );
-                candidate.set_placement_stats(Arc::clone(&stats));
-                let unread_output_bytes_per_row =
-                    compressed_bytes_per_row(metadata, |leaf| {
-                        output_projection.leaf_included(leaf)
-                            && !candidate.reads_leaf(leaf)
-                    });
-                let read_output_bytes_per_row =
-                    compressed_bytes_per_row(metadata, |leaf| {
-                        output_projection.leaf_included(leaf)
-                            && candidate.reads_leaf(leaf)
-                    });
-                ConjunctKind::Required {
-                    expr: Arc::clone(candidate.source_expr()),
-                    stats,
-                    unread_output_bytes_per_row,
-                    read_output_bytes_per_row,
-                }
-            };
+            }
+            let stats = options.sites.stats_for(
+                &file_conjuncts,
+                candidate.position(),
+                options.scan_conjunct_count,
+            );
+            candidate.set_placement_stats(Arc::clone(&stats));
+            let unread_output_bytes_per_row =
+                compressed_bytes_per_row(metadata, |leaf| {
+                    output_projection.leaf_included(leaf) && !candidate.reads_leaf(leaf)
+                });
+            let read_output_bytes_per_row = compressed_bytes_per_row(metadata, |leaf| {
+                output_projection.leaf_included(leaf) && candidate.reads_leaf(leaf)
+            });
+            let optional = candidate.gate().map(|gate| OptionalConjunct {
+                gate: Arc::clone(gate),
+                saving: candidate.optional_saving().cloned(),
+            });
             conjuncts.push(ManagedConjunct {
                 candidate: index,
-                kind,
+                expr: Arc::clone(candidate.source_expr()),
+                stats,
+                unread_output_bytes_per_row,
+                read_output_bytes_per_row,
+                optional,
                 placement: Placement::RowFilter,
             });
         }
@@ -214,6 +238,7 @@ impl FilePlacement {
             return None;
         }
         let mut placement = Self {
+            order: (0..conjuncts.len()).collect(),
             conjuncts,
             sites: Arc::clone(&options.sites),
             metadata: Arc::clone(metadata),
@@ -229,107 +254,134 @@ impl FilePlacement {
 
     /// Makes the placement for the next row group, which has
     /// `next_row_group_rows` rows. Call at each row group boundary. Returns
-    /// true if the placement changed: then rebuild the `RowFilter` and the
-    /// decoder projection.
-    pub(crate) fn decide(&mut self, next_row_group_rows: usize) -> bool {
+    /// `Some` if the placement or the evaluation order changed: then rebuild
+    /// the post-scan filter, and the decoder if its projection mask or its
+    /// `RowFilter` changed.
+    pub(crate) fn decide(
+        &mut self,
+        next_row_group_rows: usize,
+    ) -> Option<PlacementChange> {
         if self.frozen {
-            return false;
+            return None;
         }
         // A skipped optional conjunct did not see the batches of the last
         // row group. Count them down on its gate, so that the pause can
         // end.
         for conjunct in &self.conjuncts {
-            if let (Placement::Skip, ConjunctKind::Optional { gate }) =
-                (conjunct.placement, &conjunct.kind)
+            if let (Placement::Skip, Some(optional)) =
+                (conjunct.placement, &conjunct.optional)
             {
-                gate.lock().skip_rows(self.row_group_rows, self.batch_size);
+                optional
+                    .gate
+                    .lock()
+                    .skip_rows(self.row_group_rows, self.batch_size);
             }
         }
         self.row_group_rows = next_row_group_rows;
-        let changed = self.place_all(next_row_group_rows, false);
-        if changed {
-            self.changes.add(1);
+        let row_filter = self.row_filter_candidates();
+        let before = self.snapshot();
+        self.place_all(next_row_group_rows, false);
+        if self.snapshot() == before {
+            return None;
         }
-        changed
+        self.changes.add(1);
+        Some(PlacementChange {
+            row_filter: self.row_filter_candidates() != row_filter,
+        })
     }
 
-    /// Decides the placement of each conjunct. Returns true if a placement
-    /// changed.
-    fn place_all(&mut self, row_group_rows: usize, first: bool) -> bool {
+    /// Decides the placement of each conjunct and the evaluation order.
+    fn place_all(&mut self, row_group_rows: usize, first: bool) {
         let decode_ns_per_byte = self.sites.decode().ns_per_byte();
         let fetch_ns_per_row =
             self.sites.fetch().mean_nanos() / row_group_rows.max(1) as f64;
-        let mut changed = false;
+        let mut observations = Vec::with_capacity(self.conjuncts.len());
         for conjunct in &mut self.conjuncts {
-            let placement = match &conjunct.kind {
-                ConjunctKind::Required {
-                    stats,
-                    unread_output_bytes_per_row,
-                    read_output_bytes_per_row,
-                    ..
-                } => place_required(
-                    &RequiredConjunctInputs {
-                        observation: stats.observation(),
-                        unread_output_bytes_per_row: *unread_output_bytes_per_row,
-                        read_output_bytes_per_row: *read_output_bytes_per_row,
-                        decode_ns_per_byte,
-                        fetch_ns_per_row,
-                    },
-                    (!first).then_some(conjunct.placement),
-                ),
-                ConjunctKind::Optional { gate } => {
-                    place_optional(gate.lock().is_paused())
+            let observation = conjunct.stats.observation();
+            observations.push(observation);
+            let inputs = ConjunctInputs {
+                observation,
+                unread_output_bytes_per_row: conjunct.unread_output_bytes_per_row,
+                read_output_bytes_per_row: conjunct.read_output_bytes_per_row,
+                decode_ns_per_byte,
+                fetch_ns_per_row,
+            };
+            let current = (!first).then_some(conjunct.placement);
+            conjunct.placement = match &conjunct.optional {
+                None => place_required(&inputs, current),
+                Some(optional) => {
+                    let paused = optional.gate.lock().is_paused();
+                    let placement = place_optional(&inputs, paused, current);
+                    if let Some(saving) = &optional.saving {
+                        saving.set_row_filter(placement == Placement::RowFilter);
+                    }
+                    placement
                 }
             };
-            changed |= placement != conjunct.placement;
-            conjunct.placement = placement;
         }
-        changed
+        self.order = evaluation_order(&observations);
     }
 
-    /// Number of rows of the row group at `index` in the file.
-    pub(crate) fn row_group_rows(&self, index: usize) -> usize {
-        usize::try_from(self.metadata.row_group(index).num_rows()).unwrap_or(0)
-    }
-
-    /// The current placement of each managed conjunct.
-    pub(crate) fn placements(&self) -> Vec<Placement> {
-        self.conjuncts.iter().map(|c| c.placement).collect()
+    /// The placement and the evaluation order of the managed conjuncts.
+    pub(crate) fn snapshot(&self) -> PlacementSnapshot {
+        PlacementSnapshot {
+            placements: self.conjuncts.iter().map(|c| c.placement).collect(),
+            order: self.order.clone(),
+        }
     }
 
-    /// Sets the placement of each managed conjunct to `placements` (from
-    /// [`Self::placements`]) and stops all later changes. For a file where
-    /// the scan cannot apply a change.
-    pub(crate) fn restore_and_freeze(&mut self, placements: &[Placement]) {
-        for (conjunct, placement) in self.conjuncts.iter_mut().zip(placements) {
+    /// Sets the placement and the order to `snapshot` (from
+    /// [`Self::snapshot`]) and stops all later changes. For a file where the
+    /// scan cannot apply a change.
+    pub(crate) fn restore_and_freeze(&mut self, snapshot: &PlacementSnapshot) {
+        for (conjunct, placement) in self.conjuncts.iter_mut().zip(&snapshot.placements) {
             conjunct.placement = *placement;
         }
+        self.order.clone_from(&snapshot.order);
         self.frozen = true;
     }
 
-    /// True if the candidate at `index` is a predicate of the `RowFilter`.
-    pub(crate) fn in_row_filter(&self, index: usize) -> bool {
-        self.conjuncts
-            .iter()
-            .find(|c| c.candidate == index)
-            .is_none_or(|c| c.placement == Placement::RowFilter)
+    /// True if the placement manages the candidate at `index`.
+    pub(crate) fn manages(&self, index: usize) -> bool {
+        self.conjuncts.iter().any(|c| c.candidate == index)
     }
 
-    /// The managed conjuncts that the post-scan filter evaluates.
+    /// The candidates of the managed conjuncts that are predicates of the
+    /// `RowFilter`, in evaluation order.
+    pub(crate) fn row_filter_candidates(&self) -> Vec<usize> {
+        self.in_order(Placement::RowFilter)
+            .map(|conjunct| conjunct.candidate)
+            .collect()
+    }
+
+    /// The managed conjuncts that the post-scan filter evaluates, in
+    /// evaluation order. The optional conjuncts have their gate.
     pub(crate) fn post_scan_conjuncts(&self) -> Vec<PostScanConjunct> {
-        self.conjuncts
-            .iter()
-            .filter(|c| c.placement == Placement::PostScan)
-            .filter_map(|c| match &c.kind {
-                ConjunctKind::Required { expr, stats, .. } => Some(PostScanConjunct {
-                    expr: Arc::clone(expr),
-                    stats: Some(Arc::clone(stats)),
-                }),
-                ConjunctKind::Optional { .. } => None,
+        self.in_order(Placement::PostScan)
+            .map(|c| PostScanConjunct {
+                expr: Arc::clone(&c.expr),
+                stats: Some(Arc::clone(&c.stats)),
+                gate: c
+                    .optional
+                    .as_ref()
+                    .map(|optional| Arc::clone(&optional.gate)),
             })
             .collect()
     }
 
+    /// The managed conjuncts with `placement`, in evaluation order.
+    fn in_order(&self, placement: Placement) -> impl Iterator<Item = &ManagedConjunct> {
+        self.order
+            .iter()
+            .map(|&i| &self.conjuncts[i])
+            .filter(move |c| c.placement == placement)
+    }
+
+    /// Number of rows of the row group at `index` in the file.
+    pub(crate) fn row_group_rows(&self, index: usize) -> usize {
+        usize::try_from(self.metadata.row_group(index).num_rows()).unwrap_or(0)
+    }
+
     /// Sets the projection mask of the decoder, for the decode time
     /// measurement.
     pub(crate) fn set_decoder_mask(&mut self, mask: &ProjectionMask) {
diff --git a/datafusion/datasource-parquet/src/filter_placement/model.rs b/datafusion/datasource-parquet/src/filter_placement/model.rs
index b1edf1b73e19a..35f48b96b8e53 100644
--- a/datafusion/datasource-parquet/src/filter_placement/model.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/model.rs
@@ -49,9 +49,9 @@ const LEAVE_ROW_FILTER_MARGIN: f64 = 1.1;
 /// than this multiple of its benefit.
 const ENTER_ROW_FILTER_MARGIN: f64 = 0.9;
 
-/// What the decision for a required conjunct uses.
+/// The measured evidence for the placement of a conjunct.
 #[derive(Debug, Clone, Copy, PartialEq)]
-pub(crate) struct RequiredConjunctInputs {
+pub(crate) struct ConjunctInputs {
     /// The pooled measurements of the conjunct.
     pub(crate) observation: Observation,
     /// Compressed bytes for each row of the output columns that the
@@ -71,7 +71,7 @@ pub(crate) struct RequiredConjunctInputs {
     pub(crate) fetch_ns_per_row: f64,
 }
 
-impl RequiredConjunctInputs {
+impl ConjunctInputs {
     /// Decode time, in nanoseconds for each evaluated row, that a row
     /// filter saves. `None` without enough measurements.
     pub(crate) fn benefit_ns_per_row(&self) -> Option<f64> {
@@ -121,7 +121,7 @@ impl RequiredConjunctInputs {
 /// columns: the row groups that are left pass most rows, and the page
 /// index already skips the pages that the conjunct removes.
 pub(crate) fn place_required(
-    inputs: &RequiredConjunctInputs,
+    inputs: &ConjunctInputs,
     current: Option<Placement>,
 ) -> Placement {
     let Some(benefit) = inputs.benefit_ns_per_row() else {
@@ -146,15 +146,49 @@ pub(crate) fn place_required(
     }
 }
 
-/// The placement of an optional conjunct in the `adaptive` optional filter
-/// mode: [`Placement::Skip`] while its gate is paused, otherwise
-/// [`Placement::RowFilter`].
-pub(crate) fn place_optional(gate_paused: bool) -> Placement {
+/// The placement of an optional conjunct (in the `adaptive` optional
+/// filter mode): [`Placement::Skip`] while its gate is paused, else the same
+/// decision as for a required conjunct ([`place_required`]). A skipped
+/// conjunct is not a row filter: it needs the evidence again to become one.
+pub(crate) fn place_optional(
+    inputs: &ConjunctInputs,
+    gate_paused: bool,
+    current: Option<Placement>,
+) -> Placement {
     if gate_paused {
-        Placement::Skip
-    } else {
-        Placement::RowFilter
+        return Placement::Skip;
     }
+    let current = current.map(|current| match current {
+        Placement::Skip => Placement::PostScan,
+        other => other,
+    });
+    place_required(inputs, current)
+}
+
+/// The rank of a conjunct in the evaluation order of its stage: the rows
+/// that it removed for each nanosecond of evaluation time
+/// ([`FilterCost::rows_removed_per_nano`], the ranking key of the adaptive
+/// conjunct order of `FilterExec`). `None` before [`MIN_OBSERVED_ROWS`]
+/// evaluated rows.
+pub(crate) fn rank(observation: &Observation) -> Option<f64> {
+    if observation.rows_in < MIN_OBSERVED_ROWS {
+        return None;
+    }
+    observation.cost().rows_removed_per_nano()
+}
+
+/// The evaluation order of conjuncts, as indexes into `observations`. The
+/// same rule for required and optional conjuncts: a larger [`rank`] first.
+/// A conjunct without a rank comes first, in the written order, so that it
+/// is measured on all rows of its stage.
+pub(crate) fn evaluation_order(observations: &[Observation]) -> Vec<usize> {
+    let mut order: Vec<usize> = (0..observations.len()).collect();
+    // A stable sort keeps the written order for equal keys.
+    order.sort_by(|&a, &b| {
+        let key = |i: usize| rank(&observations[i]).unwrap_or(f64::INFINITY);
+        key(b).total_cmp(&key(a))
+    });
+    order
 }
 
 #[cfg(test)]
@@ -170,14 +204,15 @@ mod tests {
         kept: u64,
         unread_output_bytes_per_row: f64,
         fetch_ns_per_row: f64,
-    ) -> RequiredConjunctInputs {
-        RequiredConjunctInputs {
+    ) -> ConjunctInputs {
+        ConjunctInputs {
             observation: Observation {
                 rows_in,
                 rows_out: rows_in - skippable_rows,
                 skippable_rows,
                 row_groups_pruned: pruned,
                 row_groups_kept: kept,
+                nanos: 0,
             },
             unread_output_bytes_per_row,
             read_output_bytes_per_row: 0.0,
@@ -305,8 +340,62 @@ mod tests {
     }
 
     #[test]
-    fn optional_follows_the_gate() {
-        assert_eq!(place_optional(true), Placement::Skip);
-        assert_eq!(place_optional(false), Placement::RowFilter);
+    fn optional_is_skipped_while_paused_else_placed_like_required() {
+        let rows = 100_000;
+        let clustered = inputs(rows, rows / 2, 0, 0, 10.0, 1.0);
+        for current in [None, Some(Placement::RowFilter), Some(Placement::PostScan)] {
+            assert_eq!(place_optional(&clustered, true, current), Placement::Skip);
+        }
+        // Not paused: the evidence decides. A skipped conjunct is not a row
+        // filter, thus it moves to the row filter with the entry margin.
+        assert_eq!(
+            place_optional(&clustered, false, Some(Placement::Skip)),
+            Placement::RowFilter
+        );
+        let unmeasured = inputs(0, 0, 0, 0, 100.0, 0.0);
+        for current in [None, Some(Placement::Skip)] {
+            assert_eq!(
+                place_optional(&unmeasured, false, current),
+                Placement::PostScan
+            );
+        }
+        // No output column that the conjunct does not read: a row filter
+        // saves no decode time.
+        let reads_all_outputs = inputs(rows, rows / 2, 0, 0, 0.0, 0.0);
+        assert_eq!(
+            place_optional(&reads_all_outputs, false, Some(Placement::RowFilter)),
+            Placement::PostScan
+        );
+    }
+
+    fn measured(rows_in: u64, rows_out: u64, nanos: u64) -> Observation {
+        Observation {
+            rows_in,
+            rows_out,
+            nanos,
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn order_by_rows_removed_per_nanosecond() {
+        let rows = MIN_OBSERVED_ROWS * 10;
+        // Removes 90% at 1 ns for each row: 0.9 rows for each ns.
+        let cheap_selective = measured(rows, rows / 10, rows);
+        // Removes 95% at 16 ns for each row: about 0.06 rows for each ns.
+        let expensive = measured(rows, rows / 20, 16 * rows);
+        // Removes nothing at 1 ns for each row.
+        let useless = measured(rows, rows, rows);
+        let unmeasured = measured(MIN_OBSERVED_ROWS - 1, 0, 1);
+        assert_eq!(rank(&unmeasured), None);
+        assert_eq!(
+            evaluation_order(&[expensive, useless, cheap_selective]),
+            vec![2, 0, 1]
+        );
+        // Unmeasured conjuncts first, in the written order.
+        assert_eq!(
+            evaluation_order(&[expensive, unmeasured, cheap_selective, unmeasured]),
+            vec![1, 3, 2, 0]
+        );
     }
 }
diff --git a/datafusion/datasource-parquet/src/filter_placement/stats.rs b/datafusion/datasource-parquet/src/filter_placement/stats.rs
index 94775eb21fcc0..fb578ff83a688 100644
--- a/datafusion/datasource-parquet/src/filter_placement/stats.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/stats.rs
@@ -26,7 +26,7 @@ use std::time::Duration;
 use arrow::array::BooleanArray;
 use datafusion_physical_expr::PhysicalExpr;
 use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
-use datafusion_physical_expr::filter_stats::duration_nanos;
+use datafusion_physical_expr::filter_stats::{FilterCost, duration_nanos};
 use datafusion_physical_expr::utils::is_optional_filter;
 use parking_lot::Mutex;
 
@@ -48,9 +48,20 @@ pub(crate) struct Observation {
     pub(crate) row_groups_pruned: u64,
     /// Row groups that the conjunct alone did not prune with statistics.
     pub(crate) row_groups_kept: u64,
+    /// Evaluation time of the conjunct, in nanoseconds.
+    pub(crate) nanos: u64,
 }
 
 impl Observation {
+    /// The rows in, rows out and evaluation time.
+    pub(crate) fn cost(&self) -> FilterCost {
+        FilterCost {
+            rows_in: self.rows_in,
+            rows_out: self.rows_out,
+            nanos: self.nanos,
+        }
+    }
+
     /// Fraction of the evaluated rows that the decoder can skip, or `None`
     /// if the conjunct was not evaluated on any row.
     pub(crate) fn skippable_fraction(&self) -> Option<f64> {
@@ -79,6 +90,7 @@ pub(crate) struct ConjunctStats {
     skippable_rows: AtomicU64,
     row_groups_pruned: AtomicU64,
     row_groups_kept: AtomicU64,
+    nanos: AtomicU64,
 }
 
 impl ConjunctStats {
@@ -92,7 +104,7 @@ impl ConjunctStats {
     /// spans 64 or more rows of the file, thus the skippable rows are
     /// counted too low. The first predicate is measured exactly. See
     /// [`StageSelection`] for the post-scan filter.
-    pub(crate) fn record_evaluation(&self, result: &BooleanArray) {
+    pub(crate) fn record_evaluation(&self, result: &BooleanArray, nanos: u64) {
         let rows_in = result.len() as u64;
         if rows_in == 0 {
             return;
@@ -109,14 +121,23 @@ impl ConjunctStats {
         self.rows_in.fetch_add(rows_in, Ordering::Relaxed);
         self.rows_out.fetch_add(rows_out, Ordering::Relaxed);
         self.skippable_rows.fetch_add(skippable, Ordering::Relaxed);
+        self.nanos.fetch_add(nanos, Ordering::Relaxed);
     }
 
     /// Records one evaluation of the conjunct on `rows_in` rows, of which
-    /// `rows_out` passed and which made `skippable` rows skippable.
-    pub(crate) fn record(&self, rows_in: usize, rows_out: usize, skippable: usize) {
+    /// `rows_out` passed and which made `skippable` rows skippable, in
+    /// `nanos` nanoseconds.
+    pub(crate) fn record(
+        &self,
+        rows_in: usize,
+        rows_out: usize,
+        skippable: usize,
+        nanos: u64,
+    ) {
         if rows_in == 0 {
             return;
         }
+        self.nanos.fetch_add(nanos, Ordering::Relaxed);
         self.rows_in.fetch_add(rows_in as u64, Ordering::Relaxed);
         self.rows_out.fetch_add(rows_out as u64, Ordering::Relaxed);
         self.skippable_rows
@@ -131,6 +152,7 @@ impl ConjunctStats {
             skippable_rows: self.skippable_rows.load(Ordering::Relaxed),
             row_groups_pruned: self.row_groups_pruned.load(Ordering::Relaxed),
             row_groups_kept: self.row_groups_kept.load(Ordering::Relaxed),
+            nanos: self.nanos.load(Ordering::Relaxed),
         }
     }
 }
@@ -177,14 +199,21 @@ impl StageSelection {
     }
 
     /// Records in `stats` the evaluation of a conjunct with the result
-    /// `passed` (without nulls) for each row of the working batch.
-    pub(crate) fn record(&self, stats: &ConjunctStats, passed: &BooleanArray) {
+    /// `passed` (without nulls) for each row of the working batch, which
+    /// took `nanos` nanoseconds.
+    pub(crate) fn record(
+        &self,
+        stats: &ConjunctStats,
+        passed: &BooleanArray,
+        nanos: u64,
+    ) {
         let values = passed.values();
         match &self.complete_windows {
             None => stats.record(
                 self.input_rows,
                 values.count_set_bits(),
                 skippable_in(values),
+                nanos,
             ),
             Some(windows) => {
                 // The rows that the conjunct did not see pass.
@@ -200,6 +229,7 @@ impl StageSelection {
                     self.input_rows,
                     unseen + values.count_set_bits(),
                     skippable,
+                    nanos,
                 );
             }
         }
@@ -389,9 +419,9 @@ mod tests {
         let stats = ConjunctStats::default();
         let mut values = vec![Some(false); 128];
         values[0] = Some(true);
-        stats.record_evaluation(&bools(values));
-        stats.record_evaluation(&bools(vec![Some(true); 64]));
-        stats.record_evaluation(&bools(vec![None; 64]));
+        stats.record_evaluation(&bools(values), 10);
+        stats.record_evaluation(&bools(vec![Some(true); 64]), 10);
+        stats.record_evaluation(&bools(vec![None; 64]), 10);
         let observation = stats.observation();
         assert_eq!(
             observation,
@@ -401,6 +431,7 @@ mod tests {
                 skippable_rows: 128,
                 row_groups_pruned: 0,
                 row_groups_kept: 0,
+                nanos: 30,
             }
         );
         assert_eq!(observation.skippable_fraction(), Some(0.5));
@@ -417,7 +448,7 @@ mod tests {
         // Conjunct 1 keeps every fourth row: no empty window. Exact.
         let first: BooleanArray = (0..rows).map(|i| Some(i % 4 == 0)).collect();
         let first_stats = ConjunctStats::default();
-        selection.record(&first_stats, &first);
+        selection.record(&first_stats, &first, 0);
         assert_eq!(
             first_stats.observation(),
             Observation {
@@ -434,7 +465,7 @@ mod tests {
         // thus no window is empty.
         let second: BooleanArray = (0..64).map(|i| Some(i >= 32)).collect();
         let second_stats = ConjunctStats::default();
-        selection.record(&second_stats, &second);
+        selection.record(&second_stats, &second, 0);
         assert_eq!(
             second_stats.observation(),
             Observation {
@@ -454,7 +485,7 @@ mod tests {
         // Fails input rows 0..128: two empty windows.
         let passed: BooleanArray = (0..256).map(|i| Some(i >= 128)).collect();
         let stats = ConjunctStats::default();
-        selection.record(&stats, &passed);
+        selection.record(&stats, &passed, 0);
         assert_eq!(
             stats.observation(),
             Observation {
@@ -483,7 +514,7 @@ mod tests {
             .map(|i| Some(!(32..96).contains(&i) && i != 100))
             .collect();
         let stats = ConjunctStats::default();
-        selection.record(&stats, &second);
+        selection.record(&stats, &second, 0);
         assert_eq!(
             stats.observation(),
             Observation {
@@ -498,7 +529,7 @@ mod tests {
         selection.compact(&second);
         let third: BooleanArray = (0..95).map(|i| Some(i >= 64)).collect();
         let stats = ConjunctStats::default();
-        selection.record(&stats, &third);
+        selection.record(&stats, &third, 0);
         assert_eq!(stats.observation().skippable_rows, 0);
         assert_eq!(stats.observation().rows_out, 256 - 64);
     }
@@ -516,7 +547,7 @@ mod tests {
         // all of them leaves the odd rows (not seen) passing.
         let none: BooleanArray = (0..64).map(|_| Some(false)).collect();
         let stats = ConjunctStats::default();
-        selection.record(&stats, &none);
+        selection.record(&stats, &none, 0);
         assert_eq!(
             stats.observation(),
             Observation {
diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index bf71d25077fea..f8bf36f4f13dc 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -581,9 +581,10 @@ impl DecoderReadPlans {
         //
         // Either way every required conjunct is applied; nothing is silently
         // dropped. Optional conjuncts (see `split_optional`) are not needed
-        // for correctness. They never go to the post-scan filter: they are
-        // row filter predicates (as `optional_filter_mode` says) or they are
-        // used only for statistics pruning.
+        // for correctness. They are row filter predicates (as
+        // `optional_filter_mode` says), or the adaptive filter placement
+        // places them like the required conjuncts, or they are used only for
+        // statistics pruning. A rejected optional conjunct is not used.
         // ---------------------------------------------------------------
         // A pruning-only predicate is applied by a `FilterExec` above the
         // scan: the scan uses it only to prune.
@@ -6115,13 +6116,15 @@ mod test {
             assert_eq!(scan.count("filter_placement_changes"), 0);
         }
 
-        /// An optional filter that removes no rows is paused by its gate. At
-        /// the next row group boundary, the scan removes it from the
-        /// `RowFilter`: it is not evaluated and its column is not decoded for
-        /// the row filter. Without adaptive placement, the paused filter stays
-        /// a `RowFilter` predicate that lets all rows pass.
+        /// An optional filter that removes no rows is paused by its gate.
+        /// With adaptive placement, it starts in the post-scan filter (no
+        /// evidence for a row filter). While it is paused, the scan skips it
+        /// from the next row group boundary: it is not evaluated, and its
+        /// column is not decoded if it is not an output column. Without
+        /// adaptive placement, the paused filter stays a `RowFilter`
+        /// predicate that lets all rows pass.
         #[tokio::test]
-        async fn paused_optional_filter_is_removed_from_row_filter() {
+        async fn paused_optional_filter_is_skipped() {
             let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
             let (schema, file) = write_file(&store, |i| (i % 100) as i32).await;
             let inner = logical2physical(&col("a").gt_eq(lit(0)), &schema);
@@ -6133,27 +6136,46 @@ mod test {
                 ..Default::default()
             };
 
+            // The output columns are `a` and `v`.
             let adaptive =
                 scan(&store, &schema, &file, &predicate, true, true, options()).await;
             let gated =
                 scan(&store, &schema, &file, &predicate, true, false, options()).await;
+            // The output column is `v` only.
+            let metrics = ExecutionPlanMetricsSet::new();
+            let morselizer = ParquetMorselizerBuilder::new()
+                .with_store(Arc::clone(&store))
+                .with_schema(Arc::clone(&schema))
+                .with_projection_indices(&[1])
+                .with_predicate(Arc::clone(&predicate))
+                .with_pushdown_filters(true)
+                .with_filter_placement(true)
+                .with_placement_sites(fixed_sites())
+                .with_optional_filters(options())
+                .with_metrics(metrics.clone())
+                .build();
+            let stream = open_file(&morselizer, file.clone()).await.unwrap();
+            let (_, rows) = count_batches_and_rows(stream).await;
+            let not_output = Scan { rows, metrics };
 
-            for scan in [&adaptive, &gated] {
+            for scan in [&adaptive, &gated, &not_output] {
                 assert_eq!(scan.rows, TOTAL_ROWS);
                 assert!(scan.count("optional_filter_rows_skipped") > 0);
             }
             // Without placement, every row goes through the row filter.
             assert_eq!(gated.row_filter_rows(), TOTAL_ROWS);
             assert_eq!(gated.count("filter_placement_changes"), 0);
-            // With placement, the row groups where the filter is paused have
-            // no row filter.
-            assert!(adaptive.count("filter_placement_changes") > 0);
-            assert!(
-                adaptive.row_filter_rows() < TOTAL_ROWS,
-                "row filter rows: {}",
-                adaptive.row_filter_rows()
-            );
-            assert_eq!(adaptive.post_scan_rows(), 0);
+            // With placement, the filter is never a row filter, and the row
+            // groups where it is paused have no post-scan filter.
+            for scan in [&adaptive, &not_output] {
+                assert_eq!(scan.row_filter_rows(), 0);
+                assert!(scan.count("filter_placement_changes") > 0);
+                assert!(
+                    scan.post_scan_rows() < TOTAL_ROWS,
+                    "post-scan rows: {}",
+                    scan.post_scan_rows()
+                );
+            }
         }
 
         /// With a limit, the results are the same when the placement moves a
diff --git a/datafusion/datasource-parquet/src/optional_filter.rs b/datafusion/datasource-parquet/src/optional_filter.rs
index 9c4b6ccffa2e9..46e0404b5c8d4 100644
--- a/datafusion/datasource-parquet/src/optional_filter.rs
+++ b/datafusion/datasource-parquet/src/optional_filter.rs
@@ -73,7 +73,7 @@
 
 use std::collections::HashMap;
 use std::sync::Arc;
-use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
 use std::time::Duration;
 
 use arrow::array::BooleanArray;
@@ -187,6 +187,10 @@ pub(crate) struct OptionalFilterSaving {
     /// The removed rows that the decoder can skip, see
     /// [`skippable_rows`].
     skippable_rows: AtomicU64,
+    /// True if the filter is a row filter predicate (the default). The
+    /// adaptive filter placement can put the filter in the post-scan filter:
+    /// then it saves no decode time and has no stage cost.
+    row_filter: AtomicBool,
 }
 
 impl OptionalFilterSaving {
@@ -200,13 +204,20 @@ impl OptionalFilterSaving {
             unread_bytes_per_row,
             removed_rows: AtomicU64::new(0),
             skippable_rows: AtomicU64::new(0),
+            row_filter: AtomicBool::new(true),
         };
-        saving
-            .measured
-            .set_overhead_ns_per_row(ROW_FILTER_STAGE_NS_PER_ROW);
+        saving.update();
         saving
     }
 
+    /// Sets if the filter is a row filter predicate (`true`) or not (in the
+    /// post-scan filter, or not evaluated). Only a row filter predicate
+    /// saves decode time and has a stage cost.
+    pub(crate) fn set_row_filter(&self, row_filter: bool) {
+        self.row_filter.store(row_filter, Ordering::Relaxed);
+        self.update();
+    }
+
     /// The values that the gate reads, see
     /// [`OptionalFilterGate::with_measured_saving`](datafusion_physical_expr::optional_filter_gate::OptionalFilterGate::with_measured_saving).
     pub(crate) fn measured(&self) -> &Arc<MeasuredRowSaving> {
@@ -244,14 +255,21 @@ impl OptionalFilterSaving {
         self.skippable_rows.load(Ordering::Relaxed) as f64 / removed as f64
     }
 
-    /// Updates the measured saving with the current decode speed and
-    /// skippable fraction.
+    /// Updates the measured saving and overhead with the placement, the
+    /// current decode speed and the skippable fraction.
     fn update(&self) {
+        if !self.row_filter.load(Ordering::Relaxed) {
+            self.measured.set_ns_per_row(0.0);
+            self.measured.set_overhead_ns_per_row(0.0);
+            return;
+        }
         self.measured.set_ns_per_row(
             self.skippable_fraction()
                 * self.unread_bytes_per_row
                 * self.decode_cost.ns_per_byte(),
         );
+        self.measured
+            .set_overhead_ns_per_row(ROW_FILTER_STAGE_NS_PER_ROW);
     }
 }
 
@@ -356,6 +374,29 @@ mod tests {
     use super::*;
     use datafusion_physical_expr::expressions::{DynamicFilterPhysicalExpr, lit};
 
+    /// Only a row filter predicate saves decode time and has a stage cost.
+    #[test]
+    fn saving_depends_on_placement() {
+        let saving = OptionalFilterSaving::new(10.0, Arc::default());
+        // One window of 64 removed rows: all removed rows are skippable.
+        saving.record_evaluation(&BooleanArray::from(vec![false; 64]));
+        let row_filter = (
+            DEFAULT_DECODE_NS_PER_BYTE * 10.0,
+            ROW_FILTER_STAGE_NS_PER_ROW,
+        );
+        let measured = |s: &OptionalFilterSaving| {
+            (
+                s.measured().ns_per_row(),
+                s.measured().overhead_ns_per_row(),
+            )
+        };
+        assert_eq!(measured(&saving), row_filter);
+        saving.set_row_filter(false);
+        assert_eq!(measured(&saving), (0.0, 0.0));
+        saving.set_row_filter(true);
+        assert_eq!(measured(&saving), row_filter);
+    }
+
     #[test]
     fn sites_share_verdicts_by_expression_id() {
         let sites = OptionalFilterSites::default();
diff --git a/datafusion/datasource-parquet/src/push_decoder.rs b/datafusion/datasource-parquet/src/push_decoder.rs
index 42252c38bd024..4d8728195d858 100644
--- a/datafusion/datasource-parquet/src/push_decoder.rs
+++ b/datafusion/datasource-parquet/src/push_decoder.rs
@@ -69,12 +69,12 @@ use crate::ParquetFileMetrics;
 use crate::decoder_projection::{
     DecoderProjection, DecoderProjectionBuilder, PostScanSelection,
 };
-use crate::filter_placement::FilePlacement;
+use crate::filter_placement::{FilePlacement, PlacementChange};
 use crate::metrics::{ByteProgress, RowFilterSkippedFullyMatchedMetric};
 use crate::optional_filter::OptionalFilterSavings;
 use crate::row_filter::{
     OptionalFilterRowFilterContext, PrebuiltRowFilterCandidate,
-    prebuild_row_filter_candidates, row_filter_from_prebuilt,
+    prebuild_row_filter_candidates, row_filter_from_prebuilt, row_filter_in_order,
 };
 use crate::row_group_filter::RowGroupPruningStatistics;
 
@@ -554,20 +554,24 @@ impl RowFilterContext {
     /// Infallible by construction: [`Self::try_new`] only produces a context
     /// when the prebuilt candidate list is non-empty.
     ///
-    /// With adaptive filter placement, only the candidates that are placed
-    /// in the `RowFilter` are used.
+    /// With adaptive filter placement, the `RowFilter` has the candidates
+    /// that the placement does not manage, then the candidates that it
+    /// places in the `RowFilter`, in its evaluation order.
     pub(crate) fn build_row_filter(&self) -> RowFilter {
-        let placement = self.placement.as_ref();
-        row_filter_from_prebuilt(
-            self.prebuilt
-                .as_slice()
-                .iter()
-                .enumerate()
-                .filter(|(index, _)| placement.is_none_or(|p| p.in_row_filter(*index)))
-                .map(|(_, candidate)| candidate),
-            self.reorder_predicates,
-            &self.file_metrics,
-        )
+        let candidates = self.prebuilt.as_slice();
+        let Some(placement) = &self.placement else {
+            return row_filter_from_prebuilt(
+                candidates,
+                self.reorder_predicates,
+                &self.file_metrics,
+            );
+        };
+        let ordered = (0..candidates.len())
+            .filter(|index| !placement.manages(*index))
+            .chain(placement.row_filter_candidates())
+            .map(|index| &candidates[index])
+            .collect();
+        row_filter_in_order(ordered, &self.file_metrics)
     }
 }
 
@@ -937,9 +941,16 @@ impl PushDecoderStreamState {
         pruned_count: usize,
     ) -> Result<bool, DataFusionError> {
         // The adaptive filter placement for the next RG. `Some` when it
-        // changed: then the decoder needs the new projection and a new
-        // `RowFilter`.
-        let new_projection = self.update_placement()?;
+        // changed: then the stream needs the new projection (its post-scan
+        // filter), and the decoder needs a rebuild only if its projection
+        // mask or its `RowFilter` changed.
+        let (new_projection, row_filter_changed) = match self.update_placement()? {
+            Some((projection, change)) => (Some(projection), change.row_filter),
+            None => (None, false),
+        };
+        let mask_changed = new_projection.as_ref().is_some_and(|projection| {
+            projection.projection_mask() != self.decoder_projection.projection_mask()
+        });
 
         // `desired_filter` is `Some(true)` when the next RG needs a real
         // filter, `Some(false)` when it is fully-matched (filter is a no-op, so
@@ -950,10 +961,15 @@ impl PushDecoderStreamState {
             .as_ref()
             .and_then(|_| self.rg_plan.front().map(|e| !e.fully_matched));
         let filter_needs_toggle = desired_filter.is_some_and(|want| {
-            want != self.filter_installed || (want && new_projection.is_some())
+            want != self.filter_installed || (want && row_filter_changed)
         });
 
-        if pruned_count == 0 && !filter_needs_toggle && new_projection.is_none() {
+        if pruned_count == 0 && !filter_needs_toggle && !mask_changed {
+            // Only the post-scan filter changed (or nothing): the decoder
+            // stays.
+            if let Some(projection) = new_projection {
+                self.decoder_projection = projection;
+            }
             return Ok(false);
         }
         if self.rg_plan.is_empty() {
@@ -1012,7 +1028,9 @@ impl PushDecoderStreamState {
     /// projection. If the new post-scan conjuncts change that schema (this
     /// can happen with nested columns), the file keeps its current placement
     /// until its end.
-    fn update_placement(&mut self) -> Result<Option<DecoderProjection>> {
+    fn update_placement(
+        &mut self,
+    ) -> Result<Option<(DecoderProjection, PlacementChange)>> {
         let (Some(builder), Some(front)) =
             (self.projection_builder.as_ref(), self.rg_plan.front())
         else {
@@ -1025,18 +1043,18 @@ impl PushDecoderStreamState {
         else {
             return Ok(None);
         };
-        let previous = placement.placements();
+        let previous = placement.snapshot();
         let rows = placement.row_group_rows(front.rg_index);
-        if !placement.decide(rows) {
+        let Some(change) = placement.decide(rows) else {
             return Ok(None);
-        }
+        };
         let projection = builder.build(&placement.post_scan_conjuncts())?;
         if projection.filtered_schema() != self.decoder_projection.filtered_schema() {
             placement.restore_and_freeze(&previous);
             return Ok(None);
         }
         placement.set_decoder_mask(projection.projection_mask());
-        Ok(Some(projection))
+        Ok(Some((projection, change)))
     }
 
     /// Copies metrics from ArrowReaderMetrics (the metrics collected by the
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 8d76f57ede953..5993a4fb420e9 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -80,8 +80,10 @@ use parquet::file::metadata::ParquetMetaData;
 use datafusion_common::Result;
 use datafusion_common::cast::as_boolean_array;
 use datafusion_common::config::OptionalFilterMode;
+use datafusion_common::instant::Instant;
 use datafusion_common::tree_node::TreeNode;
 use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
+use datafusion_physical_expr::filter_stats::duration_nanos;
 use datafusion_physical_expr::optional_filter_gate::{GateDecision, OptionalFilterGate};
 use datafusion_physical_expr::utils::{is_optional_filter, reassign_expr_columns};
 use datafusion_physical_expr::{PhysicalExpr, split_conjunction};
@@ -160,7 +162,20 @@ pub(crate) struct OptionalFilterGateState {
 pub(crate) type SharedOptionalFilterGate = Arc<Mutex<OptionalFilterGateState>>;
 
 impl OptionalFilterGateState {
-    fn begin_batch(&mut self, num_rows: usize) -> GateDecision {
+    /// The shared state of `gate`, which reports to `metrics`.
+    pub(crate) fn shared(
+        gate: OptionalFilterGate,
+        metrics: OptionalFilterMetrics,
+    ) -> SharedOptionalFilterGate {
+        Arc::new(Mutex::new(Self {
+            gate,
+            metrics,
+            reported_pauses: 0,
+        }))
+    }
+
+    /// See [`OptionalFilterGate::begin_batch`]. Counts the skipped rows.
+    pub(crate) fn begin_batch(&mut self, num_rows: usize) -> GateDecision {
         let decision = self.gate.begin_batch();
         if decision == GateDecision::Skip {
             self.metrics.add_rows_skipped(num_rows);
@@ -169,12 +184,19 @@ impl OptionalFilterGateState {
         decision
     }
 
-    fn record(&mut self, rows_in: usize, rows_out: usize, elapsed: Duration) {
+    /// See [`OptionalFilterGate::record`]. Adds the evaluation time.
+    pub(crate) fn record(&mut self, rows_in: usize, rows_out: usize, elapsed: Duration) {
         self.gate.record(rows_in, rows_out, elapsed);
         self.metrics.add_eval_time(elapsed);
         self.report_pauses();
     }
 
+    /// The clock to measure the evaluation time, see
+    /// [`OptionalFilterGate::clock`].
+    pub(crate) fn now_nanos(&self) -> u64 {
+        self.gate.clock().now_nanos()
+    }
+
     /// True if the gate skips the next batch.
     pub(crate) fn is_paused(&self) -> bool {
         self.gate.is_paused()
@@ -255,6 +277,7 @@ impl ArrowPredicate for DatafusionArrowPredicate {
         }
         // The gate measures the evaluation time with its clock.
         let start = gate.as_ref().map(|gate| gate.gate.clock().now_nanos());
+        let measure_start = self.placement_stats.as_ref().map(|_| Instant::now());
 
         self.physical_expr
             .evaluate(&batch)
@@ -265,8 +288,13 @@ impl ArrowPredicate for DatafusionArrowPredicate {
                 let num_pruned = bool_arr.len() - num_matched;
                 self.rows_pruned.add(num_pruned);
                 self.rows_matched.add(num_matched);
-                if let Some(stats) = &self.placement_stats {
-                    stats.record_evaluation(&bool_arr);
+                if let (Some(stats), Some(measure_start)) =
+                    (&self.placement_stats, measure_start)
+                {
+                    stats.record_evaluation(
+                        &bool_arr,
+                        duration_nanos(measure_start.elapsed()),
+                    );
                 }
                 if let Some(saving) = &self.saving {
                     saving.record_evaluation(&bool_arr);
@@ -814,15 +842,14 @@ pub(crate) fn prebuild_row_filter_candidates(
                 gate = gate.with_measured_saving(Arc::clone(saving.measured()));
                 candidate.saving = Some(saving);
             }
-            candidate.gate = Some(Arc::new(Mutex::new(OptionalFilterGateState {
+            candidate.gate = Some(OptionalFilterGateState::shared(
                 gate,
-                metrics: OptionalFilterMetrics::new(
+                OptionalFilterMetrics::new(
                     optional.metrics,
                     optional.partition,
                     optional.filename,
                 ),
-                reported_pauses: 0,
-            })));
+            ));
             prebuilt.push(candidate);
         }
     }
@@ -889,12 +916,19 @@ pub(crate) fn row_filter_from_prebuilt<'a>(
     prebuilt: impl IntoIterator<Item = &'a PrebuiltRowFilterCandidate>,
     reorder_predicates: bool,
     file_metrics: &ParquetFileMetrics,
+) -> RowFilter {
+    row_filter_in_order(order_candidates(prebuilt, reorder_predicates), file_metrics)
+}
+
+/// A [`RowFilter`] with the predicates of `ordered`, in this order.
+pub(crate) fn row_filter_in_order(
+    ordered: Vec<&PrebuiltRowFilterCandidate>,
+    file_metrics: &ParquetFileMetrics,
 ) -> RowFilter {
     let rows_pruned = &file_metrics.pushdown_rows_pruned;
     let rows_matched = &file_metrics.pushdown_rows_matched;
     let time = &file_metrics.row_pushdown_eval_time;
 
-    let ordered = order_candidates(prebuilt, reorder_predicates);
     let total = ordered.len();
     let filters: Vec<Box<dyn ArrowPredicate>> = ordered
         .into_iter()
@@ -3192,21 +3226,20 @@ mod optional_filter_tests {
     }
 
     /// The adaptive filter placement of a file with a required conjunct and
-    /// a gated optional conjunct. The gate decisions use the durations
+    /// two gated optional conjuncts. The gate decisions use the durations
     /// passed to `record`, thus the test is deterministic.
     #[test]
     fn file_placement_follows_gate_and_measurements() {
-        use crate::filter_placement::{
-            FilePlacement, Placement, PlacementOptions, PlacementSites,
-        };
+        use crate::filter_placement::{FilePlacement, PlacementOptions, PlacementSites};
         use arrow::array::BooleanArray;
         use datafusion_physical_plan::metrics::Count;
 
         let (_file, metadata, file_schema) = test_file();
-        // a > 5 AND Optional(b >= 0). The optional conjunct removes no row.
+        // a > 5 AND Optional(b >= 0) AND Optional(a >= 0).
         let predicate = conjunction([
             col_op("a", Operator::Gt, 5, &file_schema),
             optional(col_op("b", Operator::GtEq, 0, &file_schema)),
+            optional(col_op("a", Operator::GtEq, 0, &file_schema)),
         ]);
         let options = options(OptionalFilterMode::Adaptive);
         let metrics = ExecutionPlanMetricsSet::new();
@@ -3215,8 +3248,7 @@ mod optional_filter_tests {
         // A slow decode: a row filter that skips rows saves a lot.
         let sites = Arc::new(PlacementSites::with_fixed_costs(1000.0, 0.0));
         let placement_options = PlacementOptions::new(true, sites, Some(&predicate));
-        // The output columns are `b`, `c` and `s`: the required conjunct
-        // reads no output column (no second decode in a row filter).
+        // The output columns are `b`, `c` and `s`.
         let output =
             ProjectionMask::leaves(metadata.file_metadata().schema_descr(), [1, 2, 3]);
         let batch_size = 100;
@@ -3232,59 +3264,105 @@ mod optional_filter_tests {
             changes.clone(),
         )
         .unwrap();
-        let required = candidates.iter().position(|c| !c.is_optional()).unwrap();
-        let gated = candidates.iter().position(|c| c.is_optional()).unwrap();
-        // Without measurements, the required conjunct starts in the
-        // post-scan filter. The gate is not paused.
-        assert!(!placement.in_row_filter(required));
-        assert!(placement.in_row_filter(gated));
-
-        // The gate pauses the optional filter: it removed no row in its
-        // first window (`sample_batches` = 2).
-        let gate = Arc::clone(candidates[gated].gate().unwrap());
-        for _ in 0..2 {
-            let mut gate = gate.lock();
-            assert_eq!(gate.begin_batch(batch_size), GateDecision::Evaluate);
-            gate.record(batch_size, batch_size, Duration::from_micros(1));
-        }
-        assert!(gate.lock().is_paused());
-        assert!(placement.decide(1000));
-        assert!(!placement.in_row_filter(gated));
-        assert!(!placement.in_row_filter(required));
-        assert_eq!(changes.value(), 1);
-
-        // The pause (`initial_pause_batches` = 4) ends while the scan skips
-        // the next row group (10 batches): the filter is a row filter again.
-        assert!(placement.decide(1000));
-        assert!(placement.in_row_filter(gated));
-        assert!(!gate.lock().is_paused());
+        let index = |expr: &str| {
+            candidates
+                .iter()
+                .position(|c| c.source_expr().to_string() == expr)
+                .unwrap()
+        };
+        let required = index("a@0 > 5");
+        let on_b = index("b@1 >= 0");
+        let on_a = index("a@0 >= 0");
+        let post_scan = |placement: &FilePlacement| {
+            placement
+                .post_scan_conjuncts()
+                .iter()
+                .map(|c| (c.expr.to_string(), c.gate.is_some()))
+                .collect::<Vec<_>>()
+        };
+        let entry = |expr: &str, gated: bool| (expr.to_string(), gated);
+        // Without measurements, all conjuncts start in the post-scan filter,
+        // in the written order. The optional conjuncts have their gate.
+        assert!(placement.row_filter_candidates().is_empty());
+        assert_eq!(
+            post_scan(&placement),
+            vec![
+                entry("a@0 > 5", false),
+                entry("b@1 >= 0", true),
+                entry("a@0 >= 0", true)
+            ]
+        );
 
-        // The post-scan filter measures the required conjunct.
-        let post_scan = placement.post_scan_conjuncts();
-        assert_eq!(post_scan.len(), 1);
-        assert_eq!(post_scan[0].expr.to_string(), "a@0 > 5");
-        let stats = post_scan[0].stats.clone().unwrap();
+        // The gates pause the optional filters: they removed no row in their
+        // first window (`sample_batches` = 2). They are skipped.
+        let pause = |candidate: usize| {
+            let gate = Arc::clone(candidates[candidate].gate().unwrap());
+            for _ in 0..2 {
+                let mut gate = gate.lock();
+                assert_eq!(gate.begin_batch(batch_size), GateDecision::Evaluate);
+                gate.record(batch_size, batch_size, Duration::from_micros(1));
+            }
+            assert!(gate.lock().is_paused());
+        };
+        pause(on_b);
+        pause(on_a);
+        let change = placement.decide(1000).unwrap();
+        assert!(!change.row_filter);
+        assert_eq!(post_scan(&placement), vec![entry("a@0 > 5", false)]);
+        assert_eq!(changes.value(), 1);
 
-        // It removes every second row: no run of removed rows that the
-        // decoder can skip, thus it stays in the post-scan filter.
+        // The pauses (`initial_pause_batches` = 4) end while the scan skips
+        // the next row group (10 batches).
+        assert!(placement.decide(1000).is_some());
+        assert_eq!(post_scan(&placement).len(), 3);
+
+        // The measurements, pooled over all files. The required conjunct
+        // removes every second row at 10 ns for each row: no run of removed
+        // rows that the decoder can skip, thus it stays in the post-scan
+        // filter.
+        let all_stats = placement.post_scan_conjuncts();
+        let stats_of = |expr: &str| {
+            all_stats
+                .iter()
+                .find(|c| c.expr.to_string() == expr)
+                .and_then(|c| c.stats.clone())
+                .unwrap()
+        };
         let scattered: BooleanArray = (0..20_000).map(|i| Some(i % 2 == 0)).collect();
-        stats.record_evaluation(&scattered);
-        assert!(!placement.decide(1000));
-        assert!(!placement.in_row_filter(required));
-
-        // More rows where it removes a long run of rows: it moves to the
-        // row filter.
+        stats_of("a@0 > 5").record_evaluation(&scattered, 200_000);
+        // The optional conjunct on `a` removes a long run of rows at 1 ns
+        // for each row: it saves decode time in a row filter, and it ranks
+        // before the required conjunct.
         let clustered: BooleanArray = (0..40_000).map(|i| Some(i >= 30_000)).collect();
-        stats.record_evaluation(&clustered);
-        assert!(placement.decide(1000));
-        assert!(placement.in_row_filter(required));
-        assert!(placement.post_scan_conjuncts().is_empty());
+        stats_of("a@0 >= 0").record_evaluation(&clustered, 40_000);
+        let change = placement.decide(1000).unwrap();
+        assert!(change.row_filter);
+        assert_eq!(placement.row_filter_candidates(), vec![on_a]);
+        // The optional conjunct on `b` is not measured yet: it comes first.
+        assert_eq!(
+            post_scan(&placement),
+            vec![entry("b@1 >= 0", true), entry("a@0 > 5", false)]
+        );
+
+        // The optional conjunct on `b` removes nothing: it ranks last. Only
+        // the post-scan order changes.
+        stats_of("b@1 >= 0")
+            .record_evaluation(&BooleanArray::from(vec![true; 20_000]), 20_000);
+        let change = placement.decide(1000).unwrap();
+        assert!(!change.row_filter);
+        assert_eq!(
+            post_scan(&placement),
+            vec![entry("a@0 > 5", false), entry("b@1 >= 0", true)]
+        );
+        assert!(placement.decide(1000).is_none());
+        assert_eq!(changes.value(), 4);
+        let _ = required;
 
         // A frozen placement does not change.
-        let before = placement.placements();
-        placement.restore_and_freeze(&[Placement::PostScan, Placement::PostScan]);
-        assert!(!placement.decide(1000));
-        assert_ne!(placement.placements(), before);
-        assert_eq!(changes.value(), 3);
+        let frozen = placement.snapshot();
+        placement.restore_and_freeze(&frozen);
+        stats_of("b@1 >= 0").record_evaluation(&clustered, 1);
+        assert!(placement.decide(1000).is_none());
+        assert_eq!(placement.snapshot(), frozen);
     }
 }

From aa00419f9dbf6138a60b48930af710fd60ea9de6 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 12:18:07 -0500
Subject: [PATCH 27/42] feat(parquet): add hysteresis to the placement changes
 of a file

The measurements are noisy near a decision boundary of the placement
model, and each change can rebuild the decoder (128 placement changes in
ClickBench Q25). After the n-th change of a file, the next change now
waits for 2^(n-1) - 1 row group boundaries, thus a file whose decision
flips again and again changes at most log2(row groups) + 1 times. This
applies to all decisions of the model (placement and evaluation order) in
the same way.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/filter_placement/mod.rs               | 35 ++++++++++++++++---
 .../datasource-parquet/src/row_filter.rs      |  9 ++++-
 2 files changed, 39 insertions(+), 5 deletions(-)

diff --git a/datafusion/datasource-parquet/src/filter_placement/mod.rs b/datafusion/datasource-parquet/src/filter_placement/mod.rs
index 51b9c9e6901e7..dd874cbbfdbd7 100644
--- a/datafusion/datasource-parquet/src/filter_placement/mod.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/mod.rs
@@ -51,9 +51,14 @@
 //! only the post-scan filter. The measurements are pooled over all files
 //! and partitions of the scan ([`PlacementSites`]).
 //!
-//! The placement of a file stops changing only when a change would change
-//! the schema of the decoded batches (possible with nested columns): the
-//! batch coalescer holds batches with the current schema.
+//! The measurements are noisy near a decision boundary, thus the changes of
+//! one file have hysteresis: after the `n`-th change, the next change waits
+//! for `2^(n-1) - 1` row group boundaries. A file whose decision flips again
+//! and again changes its placement at most `log2(row groups) + 1` times.
+//!
+//! The placement of a file stops changing when a change would change the
+//! schema of the decoded batches (possible with nested columns): the batch
+//! coalescer holds batches with the current schema.
 //!
 //! Conjuncts that `build_row_filter` rejects for a file always run in the
 //! post-scan filter (required) or are not used (optional). Optional
@@ -172,6 +177,10 @@ pub(crate) struct FilePlacement {
     decoded_bytes_per_row: f64,
     /// When true, the placement does not change any more.
     frozen: bool,
+    /// Number of changes of the placement of this file.
+    changes_made: u32,
+    /// Row group boundaries since the last change.
+    boundaries_since_change: usize,
     /// Number of row group boundaries where the placement changed.
     changes: Count,
 }
@@ -246,6 +255,8 @@ impl FilePlacement {
             row_group_rows: first_row_group_rows,
             decoded_bytes_per_row: 0.0,
             frozen: false,
+            changes_made: 0,
+            boundaries_since_change: 0,
             changes,
         };
         placement.place_all(first_row_group_rows, true);
@@ -256,7 +267,8 @@ impl FilePlacement {
     /// `next_row_group_rows` rows. Call at each row group boundary. Returns
     /// `Some` if the placement or the evaluation order changed: then rebuild
     /// the post-scan filter, and the decoder if its projection mask or its
-    /// `RowFilter` changed.
+    /// `RowFilter` changed. Too soon after a change, nothing changes (see
+    /// the [module documentation](self)).
     pub(crate) fn decide(
         &mut self,
         next_row_group_rows: usize,
@@ -278,6 +290,10 @@ impl FilePlacement {
             }
         }
         self.row_group_rows = next_row_group_rows;
+        self.boundaries_since_change += 1;
+        if self.boundaries_since_change <= self.change_hold() {
+            return None;
+        }
         let row_filter = self.row_filter_candidates();
         let before = self.snapshot();
         self.place_all(next_row_group_rows, false);
@@ -285,11 +301,22 @@ impl FilePlacement {
             return None;
         }
         self.changes.add(1);
+        self.changes_made += 1;
+        self.boundaries_since_change = 0;
         Some(PlacementChange {
             row_filter: self.row_filter_candidates() != row_filter,
         })
     }
 
+    /// Row group boundaries to wait after the last change before the next
+    /// change: `2^(n-1) - 1` after the `n`-th change.
+    fn change_hold(&self) -> usize {
+        1usize
+            .checked_shl(self.changes_made)
+            .map_or(usize::MAX, |doubled| doubled / 2)
+            .saturating_sub(1)
+    }
+
     /// Decides the placement of each conjunct and the evaluation order.
     fn place_all(&mut self, row_group_rows: usize, first: bool) {
         let decode_ns_per_byte = self.sites.decode().ns_per_byte();
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 5993a4fb420e9..c8352a08ae22f 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -3335,6 +3335,9 @@ mod optional_filter_tests {
         // before the required conjunct.
         let clustered: BooleanArray = (0..40_000).map(|i| Some(i >= 30_000)).collect();
         stats_of("a@0 >= 0").record_evaluation(&clustered, 40_000);
+        // After the second change, the next change waits for one row group
+        // boundary (hysteresis).
+        assert!(placement.decide(1000).is_none());
         let change = placement.decide(1000).unwrap();
         assert!(change.row_filter);
         assert_eq!(placement.row_filter_candidates(), vec![on_a]);
@@ -3345,9 +3348,13 @@ mod optional_filter_tests {
         );
 
         // The optional conjunct on `b` removes nothing: it ranks last. Only
-        // the post-scan order changes.
+        // the post-scan order changes. After the third change, the next
+        // change waits for 3 row group boundaries (hysteresis).
         stats_of("b@1 >= 0")
             .record_evaluation(&BooleanArray::from(vec![true; 20_000]), 20_000);
+        for _ in 0..3 {
+            assert!(placement.decide(1000).is_none());
+        }
         let change = placement.decide(1000).unwrap();
         assert!(!change.row_filter);
         assert_eq!(

From 30ec574c49e497010712a89b97c1f8db7f70dd55 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 12:26:27 -0500
Subject: [PATCH 28/42] feat(parquet): measure unmeasured conjuncts first and
 forget old filters

Three measurement gaps of the placement model:

* A conjunct in the post-scan filter sees only the rows that the row
  filter lets pass. When the row filter removes most rows, the conjunct
  never gets enough measurements for a better placement or order
  (ClickBench Q23: the TopK filter on `EventTime` behind the `URL LIKE`
  row filter saw about 1K of 100M rows, thus it stayed behind the
  expensive `LIKE`). A conjunct without enough measurements is now the
  first predicate of the row filter when the row filter is used, thus it
  is measured on all rows. This costs one row filter stage until the
  pooled measurements have `MIN_OBSERVED_ROWS` rows.
* The pooled measurements of a conjunct with a dynamic filter described
  all generations of the filter together (for example the `true`
  placeholder of a TopK filter). They are now cleared when the generation
  of the filter increases.
* For the same reason, the row group pruning result (the statistics prior,
  which overrides the measured evidence) is not recorded for a conjunct
  whose dynamic filters can still change: it describes the filter at file
  open.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/filter_placement/mod.rs               | 40 ++++++++++++++---
 .../src/filter_placement/model.rs             | 45 ++++++++++++++++++-
 .../src/filter_placement/stats.rs             | 40 +++++++++++++++++
 .../datasource-parquet/src/row_filter.rs      | 18 ++++----
 4 files changed, 127 insertions(+), 16 deletions(-)

diff --git a/datafusion/datasource-parquet/src/filter_placement/mod.rs b/datafusion/datasource-parquet/src/filter_placement/mod.rs
index dd874cbbfdbd7..9e71c6fb148fa 100644
--- a/datafusion/datasource-parquet/src/filter_placement/mod.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/mod.rs
@@ -70,9 +70,11 @@ mod stats;
 use std::sync::Arc;
 use std::time::Duration;
 
+use datafusion_physical_expr::expressions::DynamicFilterTracking;
 use datafusion_physical_expr::utils::is_optional_filter;
 use datafusion_physical_expr::{PhysicalExpr, split_conjunction};
 use datafusion_physical_expr_common::metrics::Count;
+use datafusion_physical_expr_common::physical_expr::snapshot_generation;
 use parquet::arrow::ProjectionMask;
 use parquet::file::metadata::ParquetMetaData;
 
@@ -81,7 +83,7 @@ use crate::optional_filter::{OptionalFilterSaving, compressed_bytes_per_row};
 use crate::row_filter::{PrebuiltRowFilterCandidate, SharedOptionalFilterGate};
 
 pub(crate) use model::Placement;
-use model::{ConjunctInputs, evaluation_order, place_optional, place_required};
+use model::{ConjunctInputs, evaluation_order, explore, place_optional, place_required};
 pub(crate) use stats::{ConjunctStats, PlacementSites, StageSelection};
 
 /// The adaptive placement settings of one scan.
@@ -113,6 +115,12 @@ impl PlacementOptions {
     }
 }
 
+/// True if `expr` has dynamic filters that are not complete, thus can
+/// still change.
+fn can_change(expr: &Arc<dyn PhysicalExpr>) -> bool {
+    DynamicFilterTracking::classify(expr).watcher().is_some()
+}
+
 /// A conjunct whose placement [`FilePlacement`] decides.
 #[derive(Debug)]
 struct ManagedConjunct {
@@ -132,6 +140,11 @@ struct ManagedConjunct {
     /// `Some` for an optional conjunct with a gate (the `adaptive` optional
     /// filter mode).
     optional: Option<OptionalConjunct>,
+    /// True if the conjunct has dynamic filters that were not complete at
+    /// file open, thus can change: then its
+    /// measurements are cleared at each change (see
+    /// [`ConjunctStats::observe_generation`]).
+    dynamic: bool,
     placement: Placement,
 }
 
@@ -240,6 +253,7 @@ impl FilePlacement {
                 unread_output_bytes_per_row,
                 read_output_bytes_per_row,
                 optional,
+                dynamic: can_change(candidate.source_expr()),
                 placement: Placement::RowFilter,
             });
         }
@@ -324,6 +338,11 @@ impl FilePlacement {
             self.sites.fetch().mean_nanos() / row_group_rows.max(1) as f64;
         let mut observations = Vec::with_capacity(self.conjuncts.len());
         for conjunct in &mut self.conjuncts {
+            if conjunct.dynamic {
+                conjunct
+                    .stats
+                    .observe_generation(snapshot_generation(&conjunct.expr));
+            }
             let observation = conjunct.stats.observation();
             observations.push(observation);
             let inputs = ConjunctInputs {
@@ -338,14 +357,23 @@ impl FilePlacement {
                 None => place_required(&inputs, current),
                 Some(optional) => {
                     let paused = optional.gate.lock().is_paused();
-                    let placement = place_optional(&inputs, paused, current);
-                    if let Some(saving) = &optional.saving {
-                        saving.set_row_filter(placement == Placement::RowFilter);
-                    }
-                    placement
+                    place_optional(&inputs, paused, current)
                 }
             };
         }
+        let row_filter_used = self
+            .conjuncts
+            .iter()
+            .any(|conjunct| conjunct.placement == Placement::RowFilter);
+        for (conjunct, observation) in self.conjuncts.iter_mut().zip(&observations) {
+            conjunct.placement =
+                explore(conjunct.placement, observation, row_filter_used);
+            if let Some(saving) =
+                conjunct.optional.as_ref().and_then(|o| o.saving.as_ref())
+            {
+                saving.set_row_filter(conjunct.placement == Placement::RowFilter);
+            }
+        }
         self.order = evaluation_order(&observations);
     }
 
diff --git a/datafusion/datasource-parquet/src/filter_placement/model.rs b/datafusion/datasource-parquet/src/filter_placement/model.rs
index 35f48b96b8e53..77ab75de634f6 100644
--- a/datafusion/datasource-parquet/src/filter_placement/model.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/model.rs
@@ -165,6 +165,28 @@ pub(crate) fn place_optional(
     place_required(inputs, current)
 }
 
+/// The placement of a conjunct that has no [`rank`] yet, when the row
+/// filter has other predicates: it is measured first. A conjunct that runs
+/// after the row filter sees only the rows that the row filter lets pass,
+/// thus it could never get the measurements for a better placement or
+/// order (for example a TopK filter behind an expensive `LIKE` row filter,
+/// which could remove most rows before the `LIKE`). As first predicate of
+/// the row filter it is measured on all rows. This costs one row filter
+/// stage until [`MIN_OBSERVED_ROWS`] rows are measured (pooled over the
+/// scan), then the evidence decides.
+pub(crate) fn explore(
+    placement: Placement,
+    observation: &Observation,
+    row_filter_used: bool,
+) -> Placement {
+    if row_filter_used && placement == Placement::PostScan && rank(observation).is_none()
+    {
+        Placement::RowFilter
+    } else {
+        placement
+    }
+}
+
 /// The rank of a conjunct in the evaluation order of its stage: the rows
 /// that it removed for each nanosecond of evaluation time
 /// ([`FilterCost::rows_removed_per_nano`], the ranking key of the adaptive
@@ -180,7 +202,7 @@ pub(crate) fn rank(observation: &Observation) -> Option<f64> {
 /// The evaluation order of conjuncts, as indexes into `observations`. The
 /// same rule for required and optional conjuncts: a larger [`rank`] first.
 /// A conjunct without a rank comes first, in the written order, so that it
-/// is measured on all rows of its stage.
+/// is measured on all rows of its stage (see also [`explore`]).
 pub(crate) fn evaluation_order(observations: &[Observation]) -> Vec<usize> {
     let mut order: Vec<usize> = (0..observations.len()).collect();
     // A stable sort keeps the written order for equal keys.
@@ -377,6 +399,27 @@ mod tests {
         }
     }
 
+    #[test]
+    fn unmeasured_conjunct_explores_the_row_filter() {
+        let unmeasured = measured(MIN_OBSERVED_ROWS - 1, 0, 1);
+        let known = measured(MIN_OBSERVED_ROWS, 0, 1);
+        // Only a post-scan conjunct without a rank, and only when the row
+        // filter has other predicates.
+        assert_eq!(
+            explore(Placement::PostScan, &unmeasured, true),
+            Placement::RowFilter
+        );
+        assert_eq!(
+            explore(Placement::PostScan, &unmeasured, false),
+            Placement::PostScan
+        );
+        assert_eq!(
+            explore(Placement::PostScan, &known, true),
+            Placement::PostScan
+        );
+        assert_eq!(explore(Placement::Skip, &unmeasured, true), Placement::Skip);
+    }
+
     #[test]
     fn order_by_rows_removed_per_nanosecond() {
         let rows = MIN_OBSERVED_ROWS * 10;
diff --git a/datafusion/datasource-parquet/src/filter_placement/stats.rs b/datafusion/datasource-parquet/src/filter_placement/stats.rs
index fb578ff83a688..0ceaafd16349a 100644
--- a/datafusion/datasource-parquet/src/filter_placement/stats.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/stats.rs
@@ -91,6 +91,9 @@ pub(crate) struct ConjunctStats {
     row_groups_pruned: AtomicU64,
     row_groups_kept: AtomicU64,
     nanos: AtomicU64,
+    /// The generation of the conjunct that the evaluation measurements are
+    /// for, see [`Self::observe_generation`].
+    generation: AtomicU64,
 }
 
 impl ConjunctStats {
@@ -144,6 +147,28 @@ impl ConjunctStats {
             .fetch_add(skippable as u64, Ordering::Relaxed);
     }
 
+    /// Clears the evaluation measurements (not the statistics pruning
+    /// result) when `generation` is newer than the generation that they are
+    /// for. `generation` is the sum of the generations of the dynamic
+    /// filters in the conjunct ([`snapshot_generation`]): a new generation
+    /// is a new filter, and the measurements of the old filter do not
+    /// describe it. Generations only increase, thus a caller with an older
+    /// value does not clear the measurements again.
+    ///
+    /// [`snapshot_generation`]: datafusion_physical_expr_common::physical_expr::snapshot_generation
+    pub(crate) fn observe_generation(&self, generation: u64) {
+        if self.generation.fetch_max(generation, Ordering::Relaxed) < generation {
+            for counter in [
+                &self.rows_in,
+                &self.rows_out,
+                &self.skippable_rows,
+                &self.nanos,
+            ] {
+                counter.store(0, Ordering::Relaxed);
+            }
+        }
+    }
+
     /// The pooled values. The values are not read at the same instant.
     pub(crate) fn observation(&self) -> Observation {
         Observation {
@@ -414,6 +439,21 @@ mod tests {
         values.into_iter().collect()
     }
 
+    #[test]
+    fn new_generation_clears_evaluation_measurements() {
+        let stats = ConjunctStats::default();
+        stats.observe_generation(3);
+        stats.record_evaluation(&bools(vec![Some(false); 64]), 10);
+        // The same or an older generation keeps the measurements.
+        stats.observe_generation(3);
+        stats.observe_generation(2);
+        assert_eq!(stats.observation().rows_in, 64);
+        // A new generation clears them.
+        stats.observe_generation(4);
+        let observation = stats.observation();
+        assert_eq!((observation.rows_in, observation.nanos), (0, 0));
+    }
+
     #[test]
     fn conjunct_stats_accumulate() {
         let stats = ConjunctStats::default();
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index c8352a08ae22f..f1abff4935c5f 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -3340,15 +3340,14 @@ mod optional_filter_tests {
         assert!(placement.decide(1000).is_none());
         let change = placement.decide(1000).unwrap();
         assert!(change.row_filter);
-        assert_eq!(placement.row_filter_candidates(), vec![on_a]);
-        // The optional conjunct on `b` is not measured yet: it comes first.
-        assert_eq!(
-            post_scan(&placement),
-            vec![entry("b@1 >= 0", true), entry("a@0 > 5", false)]
-        );
+        // The optional conjunct on `b` is not measured yet: now that the
+        // row filter is used, it is the first row filter predicate, so that
+        // it is measured on all rows.
+        assert_eq!(placement.row_filter_candidates(), vec![on_b, on_a]);
+        assert_eq!(post_scan(&placement), vec![entry("a@0 > 5", false)]);
 
-        // The optional conjunct on `b` removes nothing: it ranks last. Only
-        // the post-scan order changes. After the third change, the next
+        // The optional conjunct on `b` removes nothing: no evidence for the
+        // row filter, and it ranks last. After the third change, the next
         // change waits for 3 row group boundaries (hysteresis).
         stats_of("b@1 >= 0")
             .record_evaluation(&BooleanArray::from(vec![true; 20_000]), 20_000);
@@ -3356,7 +3355,8 @@ mod optional_filter_tests {
             assert!(placement.decide(1000).is_none());
         }
         let change = placement.decide(1000).unwrap();
-        assert!(!change.row_filter);
+        assert!(change.row_filter);
+        assert_eq!(placement.row_filter_candidates(), vec![on_a]);
         assert_eq!(
             post_scan(&placement),
             vec![entry("a@0 > 5", false), entry("b@1 >= 0", true)]

From 3f9098e6da5c7ac0a9c2e0d066b0f66dda15d16f Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 12:45:36 -0500
Subject: [PATCH 29/42] test(parquet): gate windows of MIN_OBSERVED_ROWS rows
 in row filter tests

The gate decides only on windows of at least `MIN_OBSERVED_ROWS` rows.

PR: #25682

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/datasource-parquet/src/row_filter.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index f1abff4935c5f..510b0351d4949 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -3192,7 +3192,7 @@ mod optional_filter_tests {
                     break;
                 }
                 assert_eq!(state.gate.begin_batch(), GateDecision::Evaluate);
-                state.gate.record(100, 100, Duration::ZERO);
+                state.gate.record(8192, 8192, Duration::ZERO);
             }
             assert!(state.gate.is_paused());
         }

From 15f1b67869bb9b93e077b8dc5d5051fe9956d970 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 12:45:37 -0500
Subject: [PATCH 30/42] refactor(parquet): use the shared MIN_OBSERVED_ROWS in
 the placement model

The filter placement and the optional filter gates now use the same
sample size (`filter_stats::MIN_OBSERVED_ROWS`). The tests of gated
conjuncts use windows of this size.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/datasource-parquet/src/decoder_projection.rs  | 9 ++++++---
 .../datasource-parquet/src/filter_placement/model.rs     | 5 ++---
 datafusion/datasource-parquet/src/row_filter.rs          | 5 +++--
 3 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/datafusion/datasource-parquet/src/decoder_projection.rs b/datafusion/datasource-parquet/src/decoder_projection.rs
index ffc58de794a71..9554829d4a2d4 100644
--- a/datafusion/datasource-parquet/src/decoder_projection.rs
+++ b/datafusion/datasource-parquet/src/decoder_projection.rs
@@ -768,13 +768,16 @@ mod tests {
             rows_matched: Count::new(),
             eval_time: Time::new(),
         };
-        let input = || batch((0..200).map(Some).collect());
+        // Batches where half of the rows reach the optional conjunct: a
+        // window of 2 batches has more than `MIN_OBSERVED_ROWS` rows.
+        let rows = 82 * 200;
+        let input = || batch((0..rows).map(|i| Some(i % 200)).collect());
         for _ in 0..2 {
-            assert_eq!(survivors(&filter, input()).len(), 100);
+            assert_eq!(survivors(&filter, input()).len(), rows as usize / 2);
         }
         assert!(gate.lock().is_paused());
         // The paused gate skips the optional conjunct: same result.
-        assert_eq!(survivors(&filter, input()).len(), 100);
+        assert_eq!(survivors(&filter, input()).len(), rows as usize / 2);
     }
 
     #[test]
diff --git a/datafusion/datasource-parquet/src/filter_placement/model.rs b/datafusion/datasource-parquet/src/filter_placement/model.rs
index 77ab75de634f6..afefb6d085196 100644
--- a/datafusion/datasource-parquet/src/filter_placement/model.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/model.rs
@@ -18,6 +18,8 @@
 //! The placement decision for one conjunct. Pure functions, see the
 //! [module documentation](super) for the model.
 
+use datafusion_physical_expr::filter_stats::MIN_OBSERVED_ROWS;
+
 use super::stats::Observation;
 use crate::row_filter_cost::ROW_FILTER_STAGE_NS_PER_ROW;
 
@@ -32,9 +34,6 @@ pub(crate) enum Placement {
     Skip,
 }
 
-/// Minimum number of evaluated rows before a conjunct can move to the row
-/// filter.
-pub(crate) const MIN_OBSERVED_ROWS: u64 = 8192;
 
 /// A conjunct that the statistics pruned in at least this fraction of the
 /// row groups stays in the post-scan filter. See [`place_required`].
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 510b0351d4949..40813ddf06f0c 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -3299,8 +3299,9 @@ mod optional_filter_tests {
             let gate = Arc::clone(candidates[candidate].gate().unwrap());
             for _ in 0..2 {
                 let mut gate = gate.lock();
-                assert_eq!(gate.begin_batch(batch_size), GateDecision::Evaluate);
-                gate.record(batch_size, batch_size, Duration::from_micros(1));
+                // A window needs `MIN_OBSERVED_ROWS` rows.
+                assert_eq!(gate.begin_batch(8192), GateDecision::Evaluate);
+                gate.record(8192, 8192, Duration::from_micros(1));
             }
             assert!(gate.lock().is_paused());
         };

From 33a188af8e071fb25d9dafa47c8d2a1ec473628b Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Sat, 26 Sep 2026 12:05:40 -0500
Subject: [PATCH 31/42] feat(parquet): let measured evidence decide the
 placement, drop the statistics prior

The statistics prior (a conjunct that the row group statistics pruned in
at least half of the row groups stays in the post-scan filter) overrode
the measured evidence. A prior must yield to evidence. Before a conjunct
has `MIN_OBSERVED_ROWS` measured rows, it is in the post-scan filter
anyway, thus a prior that yields to evidence never changes a decision.
This removes the prior from the placement model
(`CLUSTERED_PRUNED_FRACTION`) and the pruning counts from the pooled
statistics. The model does not need the per-conjunct pruning statistics
of #25726 any more.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/filter_placement/model.rs             | 73 ++++---------------
 .../src/filter_placement/stats.rs             | 21 +-----
 2 files changed, 14 insertions(+), 80 deletions(-)

diff --git a/datafusion/datasource-parquet/src/filter_placement/model.rs b/datafusion/datasource-parquet/src/filter_placement/model.rs
index afefb6d085196..57606fe403d4b 100644
--- a/datafusion/datasource-parquet/src/filter_placement/model.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/model.rs
@@ -34,11 +34,6 @@ pub(crate) enum Placement {
     Skip,
 }
 
-
-/// A conjunct that the statistics pruned in at least this fraction of the
-/// row groups stays in the post-scan filter. See [`place_required`].
-pub(crate) const CLUSTERED_PRUNED_FRACTION: f64 = 0.5;
-
 /// A row filter conjunct moves to the post-scan filter only if its cost is
 /// larger than this multiple of its benefit. The same margin as the
 /// optional filter gate.
@@ -113,12 +108,6 @@ impl ConjunctInputs {
 /// group, `None` at the first decision. A change needs a 10% margin, so that
 /// a conjunct does not move at each row group when the benefit and the cost
 /// are almost equal.
-///
-/// A conjunct that the row group statistics pruned in at least
-/// [`CLUSTERED_PRUNED_FRACTION`] of the row groups stays in the post-scan
-/// filter (the statistics prior). The data is then clustered on its
-/// columns: the row groups that are left pass most rows, and the page
-/// index already skips the pages that the conjunct removes.
 pub(crate) fn place_required(
     inputs: &ConjunctInputs,
     current: Option<Placement>,
@@ -126,13 +115,6 @@ pub(crate) fn place_required(
     let Some(benefit) = inputs.benefit_ns_per_row() else {
         return current.unwrap_or(Placement::PostScan);
     };
-    let clustered = inputs
-        .observation
-        .pruned_fraction()
-        .is_some_and(|pruned| pruned >= CLUSTERED_PRUNED_FRACTION);
-    if clustered {
-        return Placement::PostScan;
-    }
     let cost = inputs.cost_ns_per_row();
     let row_filter = match current {
         Some(Placement::RowFilter) => cost <= benefit * LEAVE_ROW_FILTER_MARGIN,
@@ -221,8 +203,6 @@ mod tests {
     fn inputs(
         rows_in: u64,
         skippable_rows: u64,
-        pruned: u64,
-        kept: u64,
         unread_output_bytes_per_row: f64,
         fetch_ns_per_row: f64,
     ) -> ConjunctInputs {
@@ -231,8 +211,6 @@ mod tests {
                 rows_in,
                 rows_out: rows_in - skippable_rows,
                 skippable_rows,
-                row_groups_pruned: pruned,
-                row_groups_kept: kept,
                 nanos: 0,
             },
             unread_output_bytes_per_row,
@@ -246,16 +224,9 @@ mod tests {
     fn starts_post_scan() {
         // Without measurements a conjunct is a post-scan filter, also with
         // wide output columns that it does not read.
-        let wide = inputs(0, 0, 0, 0, 100.0, 0.0);
+        let wide = inputs(0, 0, 100.0, 0.0);
         assert_eq!(place_required(&wide, None), Placement::PostScan);
-        let few = inputs(
-            MIN_OBSERVED_ROWS - 1,
-            MIN_OBSERVED_ROWS - 1,
-            0,
-            0,
-            100.0,
-            0.0,
-        );
+        let few = inputs(MIN_OBSERVED_ROWS - 1, MIN_OBSERVED_ROWS - 1, 100.0, 0.0);
         assert_eq!(place_required(&few, None), Placement::PostScan);
         // Too few measured rows: no change.
         assert_eq!(
@@ -269,7 +240,7 @@ mod tests {
         let rows = 100_000;
         // Half of the rows are skippable: benefit = 0.5 * 10 * 4 = 20 ns,
         // cost = 8 + 1 = 9 ns.
-        let clustered = inputs(rows, rows / 2, 0, 0, 10.0, 1.0);
+        let clustered = inputs(rows, rows / 2, 10.0, 1.0);
         assert_eq!(place_required(&clustered, None), Placement::RowFilter);
         assert_eq!(
             place_required(&clustered, Some(Placement::PostScan)),
@@ -277,21 +248,21 @@ mod tests {
         );
         // No skippable rows (for example a uniform 50% filter): nothing to
         // save, thus post-scan. Also without any fetch latency.
-        let scattered = inputs(rows, 0, 0, 0, 10.0, 0.0);
+        let scattered = inputs(rows, 0, 10.0, 0.0);
         assert_eq!(
             place_required(&scattered, Some(Placement::RowFilter)),
             Placement::PostScan
         );
         // The saving is smaller than the fixed stage cost: benefit =
         // 0.1 * 10 * 4 = 4 ns.
-        let small = inputs(rows, rows / 10, 0, 0, 10.0, 0.0);
+        let small = inputs(rows, rows / 10, 10.0, 0.0);
         assert_eq!(
             place_required(&small, Some(Placement::PostScan)),
             Placement::PostScan
         );
         // A high fetch latency (a remote object store) is more than the
         // saving: benefit = 20 ns, cost = 8 + 30 ns.
-        let remote = inputs(rows, rows / 2, 0, 0, 10.0, 30.0);
+        let remote = inputs(rows, rows / 2, 10.0, 30.0);
         assert_eq!(
             place_required(&remote, Some(Placement::RowFilter)),
             Placement::PostScan
@@ -303,7 +274,7 @@ mod tests {
         let rows = 100_000;
         // benefit = 0.5 * 10 * 4 = 20 ns. Without read output columns,
         // cost = 8 ns: row filter.
-        let mut shared = inputs(rows, rows / 2, 0, 0, 10.0, 0.0);
+        let mut shared = inputs(rows, rows / 2, 10.0, 0.0);
         assert_eq!(place_required(&shared, None), Placement::RowFilter);
         // The conjunct also reads 4 bytes of output columns for each row:
         // the rows that pass (50%) decode them again, cost = 8 + 0.5 * 4 *
@@ -315,30 +286,12 @@ mod tests {
         assert_eq!(place_required(&shared, None), Placement::PostScan);
     }
 
-    #[test]
-    fn statistics_prior_keeps_post_scan() {
-        let rows = 100_000;
-        // The statistics pruned most row groups: post-scan, also when the
-        // measurements favor the row filter.
-        let clustered_by_stats = inputs(rows, rows / 2, 6, 4, 10.0, 0.0);
-        assert_eq!(
-            place_required(&clustered_by_stats, None),
-            Placement::PostScan
-        );
-        assert_eq!(
-            place_required(&clustered_by_stats, Some(Placement::RowFilter)),
-            Placement::PostScan
-        );
-        let not_clustered = inputs(rows, rows / 2, 4, 6, 10.0, 0.0);
-        assert_eq!(place_required(&not_clustered, None), Placement::RowFilter);
-    }
-
     #[test]
     fn margin_prevents_flapping() {
         let rows = 100_000;
         // benefit = 0.5 * 10 * 4 = 20 ns; cost = 8 + 13 = 21 ns is inside
         // the margin.
-        let close = inputs(rows, rows / 2, 0, 0, 10.0, 13.0);
+        let close = inputs(rows, rows / 2, 10.0, 13.0);
         assert_eq!(
             place_required(&close, Some(Placement::RowFilter)),
             Placement::RowFilter
@@ -348,12 +301,12 @@ mod tests {
             Placement::PostScan
         );
         // Outside the margin in both directions: cost = 23 ns and 17 ns.
-        let expensive = inputs(rows, rows / 2, 0, 0, 10.0, 15.0);
+        let expensive = inputs(rows, rows / 2, 10.0, 15.0);
         assert_eq!(
             place_required(&expensive, Some(Placement::RowFilter)),
             Placement::PostScan
         );
-        let cheap = inputs(rows, rows / 2, 0, 0, 10.0, 9.0);
+        let cheap = inputs(rows, rows / 2, 10.0, 9.0);
         assert_eq!(
             place_required(&cheap, Some(Placement::PostScan)),
             Placement::RowFilter
@@ -363,7 +316,7 @@ mod tests {
     #[test]
     fn optional_is_skipped_while_paused_else_placed_like_required() {
         let rows = 100_000;
-        let clustered = inputs(rows, rows / 2, 0, 0, 10.0, 1.0);
+        let clustered = inputs(rows, rows / 2, 10.0, 1.0);
         for current in [None, Some(Placement::RowFilter), Some(Placement::PostScan)] {
             assert_eq!(place_optional(&clustered, true, current), Placement::Skip);
         }
@@ -373,7 +326,7 @@ mod tests {
             place_optional(&clustered, false, Some(Placement::Skip)),
             Placement::RowFilter
         );
-        let unmeasured = inputs(0, 0, 0, 0, 100.0, 0.0);
+        let unmeasured = inputs(0, 0, 100.0, 0.0);
         for current in [None, Some(Placement::Skip)] {
             assert_eq!(
                 place_optional(&unmeasured, false, current),
@@ -382,7 +335,7 @@ mod tests {
         }
         // No output column that the conjunct does not read: a row filter
         // saves no decode time.
-        let reads_all_outputs = inputs(rows, rows / 2, 0, 0, 0.0, 0.0);
+        let reads_all_outputs = inputs(rows, rows / 2, 0.0, 0.0);
         assert_eq!(
             place_optional(&reads_all_outputs, false, Some(Placement::RowFilter)),
             Placement::PostScan
diff --git a/datafusion/datasource-parquet/src/filter_placement/stats.rs b/datafusion/datasource-parquet/src/filter_placement/stats.rs
index 0ceaafd16349a..76e41466a76de 100644
--- a/datafusion/datasource-parquet/src/filter_placement/stats.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/stats.rs
@@ -44,10 +44,6 @@ pub(crate) struct Observation {
     /// Rows in windows where no row passed the conjunct, see
     /// [`skippable_rows`](crate::row_filter_cost::skippable_rows) and [`StageSelection`].
     pub(crate) skippable_rows: u64,
-    /// Row groups that the conjunct alone pruned with statistics.
-    pub(crate) row_groups_pruned: u64,
-    /// Row groups that the conjunct alone did not prune with statistics.
-    pub(crate) row_groups_kept: u64,
     /// Evaluation time of the conjunct, in nanoseconds.
     pub(crate) nanos: u64,
 }
@@ -73,13 +69,6 @@ impl Observation {
     pub(crate) fn pass_ratio(&self) -> Option<f64> {
         (self.rows_in > 0).then(|| self.rows_out as f64 / self.rows_in as f64)
     }
-
-    /// Fraction of the row groups that the conjunct pruned with statistics,
-    /// or `None` if there is no statistics pruning result.
-    pub(crate) fn pruned_fraction(&self) -> Option<f64> {
-        let total = self.row_groups_pruned + self.row_groups_kept;
-        (total > 0).then(|| self.row_groups_pruned as f64 / total as f64)
-    }
 }
 
 /// The pooled [`Observation`] of one conjunct. Lock-free.
@@ -88,8 +77,6 @@ pub(crate) struct ConjunctStats {
     rows_in: AtomicU64,
     rows_out: AtomicU64,
     skippable_rows: AtomicU64,
-    row_groups_pruned: AtomicU64,
-    row_groups_kept: AtomicU64,
     nanos: AtomicU64,
     /// The generation of the conjunct that the evaluation measurements are
     /// for, see [`Self::observe_generation`].
@@ -175,8 +162,6 @@ impl ConjunctStats {
             rows_in: self.rows_in.load(Ordering::Relaxed),
             rows_out: self.rows_out.load(Ordering::Relaxed),
             skippable_rows: self.skippable_rows.load(Ordering::Relaxed),
-            row_groups_pruned: self.row_groups_pruned.load(Ordering::Relaxed),
-            row_groups_kept: self.row_groups_kept.load(Ordering::Relaxed),
             nanos: self.nanos.load(Ordering::Relaxed),
         }
     }
@@ -450,8 +435,7 @@ mod tests {
         assert_eq!(stats.observation().rows_in, 64);
         // A new generation clears them.
         stats.observe_generation(4);
-        let observation = stats.observation();
-        assert_eq!((observation.rows_in, observation.nanos), (0, 0));
+        assert_eq!(stats.observation(), Observation::default());
     }
 
     #[test]
@@ -469,14 +453,11 @@ mod tests {
                 rows_in: 256,
                 rows_out: 65,
                 skippable_rows: 128,
-                row_groups_pruned: 0,
-                row_groups_kept: 0,
                 nanos: 30,
             }
         );
         assert_eq!(observation.skippable_fraction(), Some(0.5));
         assert_eq!(Observation::default().skippable_fraction(), None);
-        assert_eq!(Observation::default().pruned_fraction(), None);
     }
 
     /// A conjunct after a compaction is measured in the positions of the

From 86531c1d7bedddae6e56ad7b8ff287cb80d270d0 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 12:51:11 -0500
Subject: [PATCH 32/42] feat(parquet): order row filter predicates by measured
 rows removed per nanosecond

`order_candidates` put the gated optional predicates after all required
predicates, and `prebuild_row_filter_candidates` built them in that order
too. This is a rule for one kind of filter, not a measurement.

Now all predicates of the `RowFilter` (without adaptive filter placement)
use one rule: before the measurements, the written order of the predicate
(or by `required_bytes` with `reorder_predicates`); once a predicate has
`MIN_OBSERVED_ROWS` measured rows, by rows removed for each nanosecond
(`FilterCost::rows_removed_per_nano`, the key of the adaptive conjunct
order of `FilterExec`). An unmeasured predicate comes first, so that it
is measured on all rows. Each predicate measures its rows and time in the
file. At each row group boundary the scan rebuilds the `RowFilter` when
the order changes, with hysteresis: after the n-th change, the next
change waits for 2^(n-1) - 1 row group boundaries.

`row_filter_cost` now has `rank`, `evaluation_order` and
`ChangeHysteresis`, for the adaptive filter placement too.

PR: #25682

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../datasource-parquet/src/push_decoder.rs    | 105 ++++++++++--
 .../datasource-parquet/src/row_filter.rs      | 162 ++++++++++++------
 .../datasource-parquet/src/row_filter_cost.rs | 150 ++++++++++++++++
 3 files changed, 345 insertions(+), 72 deletions(-)

diff --git a/datafusion/datasource-parquet/src/push_decoder.rs b/datafusion/datasource-parquet/src/push_decoder.rs
index 4d8728195d858..4f885d7becd25 100644
--- a/datafusion/datasource-parquet/src/push_decoder.rs
+++ b/datafusion/datasource-parquet/src/push_decoder.rs
@@ -73,9 +73,10 @@ use crate::filter_placement::{FilePlacement, PlacementChange};
 use crate::metrics::{ByteProgress, RowFilterSkippedFullyMatchedMetric};
 use crate::optional_filter::OptionalFilterSavings;
 use crate::row_filter::{
-    OptionalFilterRowFilterContext, PrebuiltRowFilterCandidate,
-    prebuild_row_filter_candidates, row_filter_from_prebuilt, row_filter_in_order,
+    OptionalFilterRowFilterContext, PrebuiltRowFilterCandidate, candidate_order,
+    prebuild_row_filter_candidates, row_filter_in_order,
 };
+use crate::row_filter_cost::ChangeHysteresis;
 use crate::row_group_filter::RowGroupPruningStatistics;
 
 /// Shared options applied to the [`ParquetPushDecoderBuilder`] for a file
@@ -407,6 +408,11 @@ pub(crate) struct RowFilterContext {
     pub(crate) placement: Option<FilePlacement>,
     /// See [`Self::start_reader`].
     decode_measurement: DecodeMeasurement,
+    /// Without adaptive filter placement: the evaluation order of the
+    /// candidates in the installed `RowFilter` (see [`Self::refresh_order`]).
+    order: Vec<usize>,
+    /// Hysteresis on the changes of `order`.
+    order_changes: ChangeHysteresis,
 }
 
 /// Which output batches measure the decode speed, see
@@ -463,6 +469,7 @@ impl RowFilterContext {
                         unfiltered: true,
                     };
                     Self {
+                        order: candidate_order(&prebuilt, reorder_predicates),
                         prebuilt: PrebuiltRowFilterCandidateList::new(prebuilt),
                         reorder_predicates,
                         file_metrics,
@@ -470,6 +477,7 @@ impl RowFilterContext {
                         optional_savings,
                         placement: None,
                         decode_measurement,
+                        order_changes: ChangeHysteresis::default(),
                     }
                 });
                 (context, rejected)
@@ -556,23 +564,38 @@ impl RowFilterContext {
     ///
     /// With adaptive filter placement, the `RowFilter` has the candidates
     /// that the placement does not manage, then the candidates that it
-    /// places in the `RowFilter`, in its evaluation order.
+    /// places in the `RowFilter`, in its evaluation order. Without it, all
+    /// candidates in [`Self::refresh_order`] order.
     pub(crate) fn build_row_filter(&self) -> RowFilter {
         let candidates = self.prebuilt.as_slice();
-        let Some(placement) = &self.placement else {
-            return row_filter_from_prebuilt(
-                candidates,
-                self.reorder_predicates,
-                &self.file_metrics,
-            );
+        let order: Vec<usize> = match &self.placement {
+            None => self.order.clone(),
+            Some(placement) => (0..candidates.len())
+                .filter(|index| !placement.manages(*index))
+                .chain(placement.row_filter_candidates())
+                .collect(),
         };
-        let ordered = (0..candidates.len())
-            .filter(|index| !placement.manages(*index))
-            .chain(placement.row_filter_candidates())
-            .map(|index| &candidates[index])
-            .collect();
+        let ordered = order.into_iter().map(|index| &candidates[index]).collect();
         row_filter_in_order(ordered, &self.file_metrics)
     }
+
+    /// Without adaptive filter placement: at a row group boundary, updates
+    /// the evaluation order of the candidates from their measurements (see
+    /// [`candidate_order`]). Returns true if it changed: then rebuild the
+    /// `RowFilter`. With adaptive filter placement, the placement decides
+    /// the order.
+    pub(crate) fn refresh_order(&mut self) -> bool {
+        if self.placement.is_some() || !self.order_changes.boundary() {
+            return false;
+        }
+        let order = candidate_order(self.prebuilt.as_slice(), self.reorder_predicates);
+        if order == self.order {
+            return false;
+        }
+        self.order = order;
+        self.order_changes.changed();
+        true
+    }
 }
 
 impl PushDecoderStreamState {
@@ -946,7 +969,12 @@ impl PushDecoderStreamState {
         // mask or its `RowFilter` changed.
         let (new_projection, row_filter_changed) = match self.update_placement()? {
             Some((projection, change)) => (Some(projection), change.row_filter),
-            None => (None, false),
+            None => (
+                None,
+                self.row_filter_context
+                    .as_mut()
+                    .is_some_and(RowFilterContext::refresh_order),
+            ),
         };
         let mask_changed = new_projection.as_ref().is_some_and(|projection| {
             projection.projection_mask() != self.decoder_projection.projection_mask()
@@ -1201,6 +1229,53 @@ mod tests {
         ))
     }
 
+    /// Without adaptive filter placement, the `RowFilter` order follows
+    /// the measurements, with hysteresis on the changes.
+    #[test]
+    fn row_filter_order_follows_measurements() {
+        let (meta, schema) = build_three_rg_file();
+        let predicate: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
+            gt_predicate(0),
+            Operator::And,
+            gt_predicate(2000),
+        ));
+        let metrics = ExecutionPlanMetricsSet::new();
+        let (context, rejected) = RowFilterContext::try_new(
+            &predicate,
+            &schema,
+            &meta,
+            false,
+            ParquetFileMetrics::new(0, "file", &metrics),
+            None,
+            None,
+        );
+        assert!(rejected.is_empty());
+        let mut context = context.unwrap();
+        // Before any measurement: the written order.
+        assert_eq!(context.order, vec![0, 1]);
+        assert!(!context.refresh_order());
+
+        // `v > 2000` removes more rows for each nanosecond: it goes first.
+        let rows = 10 * 8192;
+        let candidates = context.prebuilt.as_slice();
+        candidates[0].cost().record(rows, rows, rows as u64);
+        candidates[1].cost().record(rows, rows / 3, rows as u64);
+        assert!(context.refresh_order());
+        assert_eq!(context.order, vec![1, 0]);
+
+        // A flip back waits for the hysteresis: after the first change the
+        // hold is 0 boundaries, after the second 1 boundary.
+        let candidates = context.prebuilt.as_slice();
+        candidates[0].cost().record(10 * rows, 0, rows as u64);
+        assert!(context.refresh_order());
+        assert_eq!(context.order, vec![0, 1]);
+        let candidates = context.prebuilt.as_slice();
+        candidates[1].cost().record(100 * rows, 0, rows as u64);
+        assert!(!context.refresh_order());
+        assert!(context.refresh_order());
+        assert_eq!(context.order, vec![1, 0]);
+    }
+
     #[test]
     fn row_group_pruner_skips_only_disqualified_row_groups() {
         let (meta, schema) = build_three_rg_file();
diff --git a/datafusion/datasource-parquet/src/row_filter.rs b/datafusion/datasource-parquet/src/row_filter.rs
index 40813ddf06f0c..4aedea7e6b70c 100644
--- a/datafusion/datasource-parquet/src/row_filter.rs
+++ b/datafusion/datasource-parquet/src/row_filter.rs
@@ -83,7 +83,7 @@ use datafusion_common::config::OptionalFilterMode;
 use datafusion_common::instant::Instant;
 use datafusion_common::tree_node::TreeNode;
 use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
-use datafusion_physical_expr::filter_stats::duration_nanos;
+use datafusion_physical_expr::filter_stats::{FilterCost, duration_nanos};
 use datafusion_physical_expr::optional_filter_gate::{GateDecision, OptionalFilterGate};
 use datafusion_physical_expr::utils::{is_optional_filter, reassign_expr_columns};
 use datafusion_physical_expr::{PhysicalExpr, split_conjunction};
@@ -102,6 +102,7 @@ use crate::projection_read_plan::{
     ParquetReadPlan, PushdownChecker, PushdownColumns, assemble_read_plan,
     build_read_plan_with_cast_clipping,
 };
+use crate::row_filter_cost::{MeasuredCost, evaluation_order};
 
 /// A "compiled" predicate passed to `ParquetRecordBatchStream` to perform
 /// row-level filtering during parquet decoding.
@@ -142,6 +143,9 @@ pub(crate) struct DatafusionArrowPredicate {
     /// removes and that the decoder can skip. `None` for all filters
     /// without a gate, and when the output projection is not known.
     saving: Option<Arc<OptionalFilterSaving>>,
+    /// The measured cost of the predicate, for the evaluation order (see
+    /// [`RowFilterContext::refresh_order`](crate::push_decoder::RowFilterContext::refresh_order)).
+    cost: Arc<MeasuredCost>,
 }
 
 /// The [`OptionalFilterGate`] of one optional filter in one file, with its
@@ -252,6 +256,7 @@ impl DatafusionArrowPredicate {
             gate: None,
             placement_stats: None,
             saving: None,
+            cost: Arc::default(),
         })
     }
 }
@@ -277,7 +282,7 @@ impl ArrowPredicate for DatafusionArrowPredicate {
         }
         // The gate measures the evaluation time with its clock.
         let start = gate.as_ref().map(|gate| gate.gate.clock().now_nanos());
-        let measure_start = self.placement_stats.as_ref().map(|_| Instant::now());
+        let measure_start = Instant::now();
 
         self.physical_expr
             .evaluate(&batch)
@@ -288,13 +293,10 @@ impl ArrowPredicate for DatafusionArrowPredicate {
                 let num_pruned = bool_arr.len() - num_matched;
                 self.rows_pruned.add(num_pruned);
                 self.rows_matched.add(num_matched);
-                if let (Some(stats), Some(measure_start)) =
-                    (&self.placement_stats, measure_start)
-                {
-                    stats.record_evaluation(
-                        &bool_arr,
-                        duration_nanos(measure_start.elapsed()),
-                    );
+                let nanos = duration_nanos(measure_start.elapsed());
+                self.cost.record(num_rows, num_matched, nanos);
+                if let Some(stats) = &self.placement_stats {
+                    stats.record_evaluation(&bool_arr, nanos);
                 }
                 if let Some(saving) = &self.saving {
                     saving.record_evaluation(&bool_arr);
@@ -631,6 +633,9 @@ pub(crate) struct PrebuiltRowFilterCandidate {
     source_expr: Arc<dyn PhysicalExpr>,
     /// See [`DatafusionArrowPredicate::placement_stats`].
     placement_stats: Option<Arc<ConjunctStats>>,
+    /// The measured cost of the predicate in the file, shared by all
+    /// `RowFilter`s built from this candidate.
+    cost: Arc<MeasuredCost>,
 }
 
 impl PrebuiltRowFilterCandidate {
@@ -644,6 +649,7 @@ impl PrebuiltRowFilterCandidate {
     }
 
     /// True if this is a gated optional filter.
+    #[cfg(test)]
     pub(crate) fn is_optional(&self) -> bool {
         self.gate.is_some()
     }
@@ -665,6 +671,12 @@ impl PrebuiltRowFilterCandidate {
         &self.source_expr
     }
 
+    /// The measured cost of the predicate in the file.
+    #[cfg(test)]
+    pub(crate) fn cost(&self) -> &MeasuredCost {
+        &self.cost
+    }
+
     /// Records the evaluations of this candidate in `stats`, for the
     /// adaptive filter placement.
     pub(crate) fn set_placement_stats(&mut self, stats: Arc<ConjunctStats>) {
@@ -726,8 +738,8 @@ pub(crate) struct OptionalFilterRowFilterContext<'a> {
 ///   gates of one filter share their pauses in all files and partitions of
 ///   the scan (see
 ///   [`OptionalFilterSites`](crate::optional_filter::OptionalFilterSites)).
-///   Optional candidates come after all required candidates, see
-///   [`row_filter_from_prebuilt`].
+///   Required and optional candidates are ordered by the same rule, see
+///   [`candidate_order`].
 ///
 /// An optional conjunct that cannot be evaluated as an `ArrowPredicate` for
 /// this file (for example because of schema evolution) is not used, in all
@@ -877,47 +889,56 @@ impl PrebuiltRowFilterCandidate {
             position,
             source_expr,
             placement_stats: None,
+            cost: Arc::default(),
         })
     }
 }
 
-/// Returns the candidates in evaluation order: the order of `prebuilt`, or
-/// ordered by `required_bytes` if `reorder_predicates` is true. Gated
-/// optional candidates always come last (see [`row_filter_from_prebuilt`]).
-fn order_candidates<'a>(
-    prebuilt: impl IntoIterator<Item = &'a PrebuiltRowFilterCandidate>,
+/// Returns the indexes of `candidates` in evaluation order. Required and
+/// optional candidates use the same rule: by the measured rows removed for
+/// each nanosecond ([`evaluation_order`]) once measured; before that, the
+/// written order of the predicate, or by `required_bytes` if
+/// `reorder_predicates` is true.
+pub(crate) fn candidate_order(
+    candidates: &[PrebuiltRowFilterCandidate],
     reorder_predicates: bool,
-) -> Vec<&'a PrebuiltRowFilterCandidate> {
-    // Collect references into a working list we can sort without disturbing
-    // the shared cache.
-    let mut ordered: Vec<&PrebuiltRowFilterCandidate> = prebuilt.into_iter().collect();
-    if reorder_predicates {
-        ordered.sort_unstable_by_key(|c| (c.is_optional(), c.required_bytes));
-    } else {
-        // `prebuild_row_filter_candidates` already puts the optional
-        // candidates last. A stable sort keeps the other order.
-        ordered.sort_by_key(|c| c.is_optional());
-    }
-    ordered
+) -> Vec<usize> {
+    let mut before_measurement: Vec<usize> = (0..candidates.len()).collect();
+    before_measurement.sort_by_key(|&i| {
+        let candidate = &candidates[i];
+        (
+            if reorder_predicates {
+                candidate.required_bytes
+            } else {
+                0
+            },
+            candidate.position,
+        )
+    });
+    let costs: Vec<FilterCost> = before_measurement
+        .iter()
+        .map(|&i| candidates[i].cost.cost())
+        .collect();
+    evaluation_order(&costs)
+        .into_iter()
+        .map(|i| before_measurement[i])
+        .collect()
 }
 
 /// Wrap a list of prebuilt candidates into a fresh [`RowFilter`], assigning
-/// per-predicate metric counters and (optionally) reordering by
-/// `required_bytes`. This is the cheap per-row-group rebuild path — no tree
-/// walks, no column resolution, only counter allocation.
-///
-/// Gated optional candidates (see [`prebuild_row_filter_candidates`]) always
-/// come after the required candidates, also when `reorder_predicates` is
-/// true. The required candidates must run in all cases. When they run first,
-/// the optional candidates see only the rows that the required candidates
-/// let pass: the gate then measures the selectivity that the optional filter
-/// adds, and a skipped optional filter costs nothing.
-pub(crate) fn row_filter_from_prebuilt<'a>(
-    prebuilt: impl IntoIterator<Item = &'a PrebuiltRowFilterCandidate>,
+/// per-predicate metric counters, in the order of [`candidate_order`]. This
+/// is the cheap per-row-group rebuild path — no tree walks, no column
+/// resolution, only counter allocation.
+pub(crate) fn row_filter_from_prebuilt(
+    prebuilt: &[PrebuiltRowFilterCandidate],
     reorder_predicates: bool,
     file_metrics: &ParquetFileMetrics,
 ) -> RowFilter {
-    row_filter_in_order(order_candidates(prebuilt, reorder_predicates), file_metrics)
+    let ordered = candidate_order(prebuilt, reorder_predicates)
+        .into_iter()
+        .map(|i| &prebuilt[i])
+        .collect();
+    row_filter_in_order(ordered, file_metrics)
 }
 
 /// A [`RowFilter`] with the predicates of `ordered`, in this order.
@@ -950,6 +971,7 @@ pub(crate) fn row_filter_in_order(
                 gate: candidate.gate.clone(),
                 placement_stats: candidate.placement_stats.clone(),
                 saving: candidate.saving.clone(),
+                cost: Arc::clone(&candidate.cost),
             }) as Box<dyn ArrowPredicate>
         })
         .collect();
@@ -2760,8 +2782,9 @@ mod optional_filter_tests {
         candidates: &[PrebuiltRowFilterCandidate],
         reorder: bool,
     ) -> Vec<(String, bool)> {
-        order_candidates(candidates, reorder)
+        candidate_order(candidates, reorder)
             .into_iter()
+            .map(|i| &candidates[i])
             .map(|c| (c.physical_expr.to_string(), c.is_optional()))
             .collect()
     }
@@ -2831,7 +2854,8 @@ mod optional_filter_tests {
         assert!(prebuilt.0.is_none());
         assert!(prebuilt.1.is_empty());
 
-        // adaptive: required first, then each optional inner with a gate.
+        // adaptive: each optional inner with a gate, in the written order
+        // (before any measurement).
         let options_adaptive = options(OptionalFilterMode::Adaptive);
         let candidates = prebuild(
             &predicate,
@@ -2843,20 +2867,22 @@ mod optional_filter_tests {
         assert_eq!(
             describe(&candidates, false),
             vec![
-                entry("a@0 > 5", false),
                 entry("b@0 > 3", true),
+                entry("a@0 > 5", false),
                 entry("c@0 < 10", true),
             ]
         );
     }
 
+    /// Before any measurement, the candidates are in the written order, or
+    /// ordered by `required_bytes` with `reorder_predicates`. Once
+    /// measured, they are ordered by rows removed for each nanosecond, the
+    /// same rule for required and optional candidates.
     #[test]
-    fn optional_candidates_stay_last_when_reordered() {
+    fn candidate_order_uses_measurements() {
         let (_file, metadata, file_schema) = test_file();
         let metrics = ExecutionPlanMetricsSet::new();
-        // `a` is the largest column, thus a reorder puts it last among the
-        // required conjuncts. The optional conjunct on the small column `b`
-        // still comes after all required conjuncts.
+        // `a` is the largest column.
         let predicate = conjunction([
             optional(col_op("b", Operator::Gt, 3, &file_schema)),
             col_op("a", Operator::Gt, 5, &file_schema),
@@ -2865,22 +2891,44 @@ mod optional_filter_tests {
         let options = options(OptionalFilterMode::Adaptive);
         let candidates =
             prebuild(&predicate, &file_schema, &metadata, &options, &metrics);
-        assert_eq!(
-            describe(&candidates, true),
-            vec![
-                entry("c@0 < 10", false),
-                entry("a@0 > 5", false),
-                entry("b@0 > 3", true),
-            ]
-        );
         assert_eq!(
             describe(&candidates, false),
             vec![
+                entry("b@0 > 3", true),
                 entry("a@0 > 5", false),
                 entry("c@0 < 10", false),
-                entry("b@0 > 3", true),
             ]
         );
+        assert_eq!(
+            describe(&candidates, true).last(),
+            Some(&entry("a@0 > 5", false))
+        );
+
+        // Measurements: `a > 5` removes 90% at 1 ns for each row, `b > 3`
+        // removes nothing, `c < 10` removes 50% at 1 ns for each row.
+        let rows = 10 * 8192;
+        let index = |expr: &str| {
+            candidates
+                .iter()
+                .position(|c| c.physical_expr.to_string() == expr)
+                .unwrap()
+        };
+        candidates[index("a@0 > 5")]
+            .cost
+            .record(rows, rows / 10, rows as u64);
+        candidates[index("b@0 > 3")]
+            .cost
+            .record(rows, rows, rows as u64);
+        candidates[index("c@0 < 10")]
+            .cost
+            .record(rows, rows / 2, rows as u64);
+        let measured = vec![
+            entry("a@0 > 5", false),
+            entry("c@0 < 10", false),
+            entry("b@0 > 3", true),
+        ];
+        assert_eq!(describe(&candidates, false), measured);
+        assert_eq!(describe(&candidates, true), measured);
     }
 
     #[test]
diff --git a/datafusion/datasource-parquet/src/row_filter_cost.rs b/datafusion/datasource-parquet/src/row_filter_cost.rs
index 4b9c7fcc34247..9f8730d51f01a 100644
--- a/datafusion/datasource-parquet/src/row_filter_cost.rs
+++ b/datafusion/datasource-parquet/src/row_filter_cost.rs
@@ -23,9 +23,16 @@
 //! * The *saving*: the decoder does not decode the other columns of the
 //!   rows that the predicate removes only when these rows make long runs.
 //!   [`skippable_rows`] counts the rows in such runs.
+//!
+//! It also has the order of the predicates: by measured rows removed for
+//! each nanosecond ([`rank`], [`evaluation_order`]), with hysteresis on the
+//! changes of a file ([`ChangeHysteresis`]).
+
+use std::sync::atomic::{AtomicU64, Ordering};
 
 use arrow::array::{Array, BooleanArray};
 use arrow::buffer::BooleanBuffer;
+use datafusion_physical_expr::filter_stats::{FilterCost, MIN_OBSERVED_ROWS};
 
 /// The fixed cost of a row filter stage for each row that it evaluates,
 /// in nanoseconds. It does not depend on the predicate: the decoder builds a
@@ -71,10 +78,153 @@ pub(crate) fn skippable_in(passed: &BooleanBuffer) -> usize {
     full + remainder
 }
 
+/// The measured rows in, rows out and evaluation time of one predicate.
+/// Lock-free.
+#[derive(Debug, Default)]
+pub(crate) struct MeasuredCost {
+    rows_in: AtomicU64,
+    rows_out: AtomicU64,
+    nanos: AtomicU64,
+}
+
+impl MeasuredCost {
+    /// Records one evaluation on `rows_in` rows, of which `rows_out` passed,
+    /// in `nanos` nanoseconds.
+    pub(crate) fn record(&self, rows_in: usize, rows_out: usize, nanos: u64) {
+        self.rows_in.fetch_add(rows_in as u64, Ordering::Relaxed);
+        self.rows_out.fetch_add(rows_out as u64, Ordering::Relaxed);
+        self.nanos.fetch_add(nanos, Ordering::Relaxed);
+    }
+
+    /// The measured values. They are not read at the same instant.
+    pub(crate) fn cost(&self) -> FilterCost {
+        FilterCost {
+            rows_in: self.rows_in.load(Ordering::Relaxed),
+            rows_out: self.rows_out.load(Ordering::Relaxed),
+            nanos: self.nanos.load(Ordering::Relaxed),
+        }
+    }
+}
+
+/// The rank of a predicate in the evaluation order of its stage: the rows
+/// that it removed for each nanosecond of evaluation time
+/// ([`FilterCost::rows_removed_per_nano`], the ranking key of the adaptive
+/// conjunct order of `FilterExec`). `None` before [`MIN_OBSERVED_ROWS`]
+/// evaluated rows.
+pub(crate) fn rank(cost: &FilterCost) -> Option<f64> {
+    if cost.rows_in < MIN_OBSERVED_ROWS {
+        return None;
+    }
+    cost.rows_removed_per_nano()
+}
+
+/// The evaluation order of predicates, as indexes into `costs`. The same
+/// rule for required and optional predicates: a larger [`rank`] first. A
+/// predicate without a rank comes first, in the given order, so that it is
+/// measured on all rows of its stage. Thus before any measurement the order
+/// is the given order.
+pub(crate) fn evaluation_order(costs: &[FilterCost]) -> Vec<usize> {
+    let mut order: Vec<usize> = (0..costs.len()).collect();
+    // A stable sort keeps the given order for equal keys.
+    order.sort_by(|&a, &b| {
+        let key = |i: usize| rank(&costs[i]).unwrap_or(f64::INFINITY);
+        key(b).total_cmp(&key(a))
+    });
+    order
+}
+
+/// Hysteresis on the changes of the row filter (its predicates or their
+/// order) of one file. Each change rebuilds the decoder, and the
+/// measurements are noisy near a decision boundary: after the `n`-th change,
+/// the next change waits for `2^(n-1) - 1` row group boundaries. A file
+/// whose decision flips again and again changes at most
+/// `log2(row groups) + 1` times.
+#[derive(Debug, Default)]
+pub(crate) struct ChangeHysteresis {
+    /// Number of changes.
+    changes: u32,
+    /// Row group boundaries since the last change.
+    boundaries_since_change: usize,
+}
+
+impl ChangeHysteresis {
+    /// Call at each row group boundary. Returns true if a change is
+    /// allowed at this boundary.
+    pub(crate) fn boundary(&mut self) -> bool {
+        self.boundaries_since_change += 1;
+        let hold = 1usize
+            .checked_shl(self.changes)
+            .map_or(usize::MAX, |doubled| doubled / 2)
+            .saturating_sub(1);
+        self.boundaries_since_change > hold
+    }
+
+    /// Call when a change happened at this boundary.
+    pub(crate) fn changed(&mut self) {
+        self.changes += 1;
+        self.boundaries_since_change = 0;
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
+    fn measured(rows_in: u64, rows_out: u64, nanos: u64) -> FilterCost {
+        FilterCost {
+            rows_in,
+            rows_out,
+            nanos,
+        }
+    }
+
+    #[test]
+    fn order_by_rows_removed_per_nanosecond() {
+        let rows = MIN_OBSERVED_ROWS * 10;
+        // Removes 90% at 1 ns for each row: 0.9 rows for each ns.
+        let cheap_selective = measured(rows, rows / 10, rows);
+        // Removes 95% at 16 ns for each row: about 0.06 rows for each ns.
+        let expensive = measured(rows, rows / 20, 16 * rows);
+        // Removes nothing at 1 ns for each row.
+        let useless = measured(rows, rows, rows);
+        let unmeasured = measured(MIN_OBSERVED_ROWS - 1, 0, 1);
+        assert_eq!(rank(&unmeasured), None);
+        assert_eq!(
+            evaluation_order(&[expensive, useless, cheap_selective]),
+            vec![2, 0, 1]
+        );
+        // Unmeasured predicates first, in the given order.
+        assert_eq!(
+            evaluation_order(&[expensive, unmeasured, cheap_selective, unmeasured]),
+            vec![1, 3, 2, 0]
+        );
+        assert_eq!(evaluation_order(&[unmeasured; 3]), vec![0, 1, 2]);
+    }
+
+    #[test]
+    fn hysteresis_doubles_the_hold() {
+        let mut hysteresis = ChangeHysteresis::default();
+        // Boundaries where a change is allowed, with a change at each of
+        // them.
+        let mut allowed = vec![];
+        for boundary in 1..=16 {
+            if hysteresis.boundary() {
+                allowed.push(boundary);
+                hysteresis.changed();
+            }
+        }
+        // Holds of 0, 0, 1, 3 and 7 boundaries after the changes.
+        assert_eq!(allowed, vec![1, 2, 4, 8, 16]);
+    }
+
+    #[test]
+    fn measured_cost_accumulates() {
+        let cost = MeasuredCost::default();
+        cost.record(100, 10, 50);
+        cost.record(100, 30, 150);
+        assert_eq!(cost.cost(), measured(200, 40, 200));
+    }
+
     fn bools(values: impl IntoIterator<Item = Option<bool>>) -> BooleanArray {
         values.into_iter().collect()
     }

From 0322fd62c4c4daaa0b11963dd2f87af51f4fdd79 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 12:51:49 -0500
Subject: [PATCH 33/42] refactor(parquet): share the ranking and the hysteresis
 with the row filter order

The adaptive filter placement and the `RowFilter` order without placement
now use the same `row_filter_cost::rank`, `evaluation_order` and
`ChangeHysteresis`, thus the same rule and the same sample size.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/filter_placement/mod.rs               | 36 +++++---------
 .../src/filter_placement/model.rs             | 49 ++-----------------
 2 files changed, 17 insertions(+), 68 deletions(-)

diff --git a/datafusion/datasource-parquet/src/filter_placement/mod.rs b/datafusion/datasource-parquet/src/filter_placement/mod.rs
index 9e71c6fb148fa..87d1a84c1ee3f 100644
--- a/datafusion/datasource-parquet/src/filter_placement/mod.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/mod.rs
@@ -39,7 +39,7 @@
 //! `adaptive` optional filter mode) is skipped while its gate is paused.
 //! In each stage (the row filter and the post-scan filter), the conjuncts
 //! are evaluated in the order of their measured rows removed for each
-//! nanosecond ([`model::evaluation_order`]), thus a cheap conjunct that
+//! nanosecond ([`evaluation_order`]), thus a cheap conjunct that
 //! removes many rows runs before an expensive one, required or optional.
 //!
 //! [`FilePlacement`] decides at file open and again at each row group
@@ -83,7 +83,10 @@ use crate::optional_filter::{OptionalFilterSaving, compressed_bytes_per_row};
 use crate::row_filter::{PrebuiltRowFilterCandidate, SharedOptionalFilterGate};
 
 pub(crate) use model::Placement;
-use model::{ConjunctInputs, evaluation_order, explore, place_optional, place_required};
+use model::{ConjunctInputs, explore, place_optional, place_required};
+use stats::Observation;
+
+use crate::row_filter_cost::{ChangeHysteresis, evaluation_order};
 pub(crate) use stats::{ConjunctStats, PlacementSites, StageSelection};
 
 /// The adaptive placement settings of one scan.
@@ -178,7 +181,7 @@ pub(crate) struct PlacementChange {
 pub(crate) struct FilePlacement {
     conjuncts: Vec<ManagedConjunct>,
     /// The evaluation order of `conjuncts` (indexes), see
-    /// [`model::evaluation_order`].
+    /// [`evaluation_order`].
     order: Vec<usize>,
     sites: Arc<PlacementSites>,
     metadata: Arc<ParquetMetaData>,
@@ -190,10 +193,8 @@ pub(crate) struct FilePlacement {
     decoded_bytes_per_row: f64,
     /// When true, the placement does not change any more.
     frozen: bool,
-    /// Number of changes of the placement of this file.
-    changes_made: u32,
-    /// Row group boundaries since the last change.
-    boundaries_since_change: usize,
+    /// Hysteresis on the changes of the placement of this file.
+    hysteresis: ChangeHysteresis,
     /// Number of row group boundaries where the placement changed.
     changes: Count,
 }
@@ -269,8 +270,7 @@ impl FilePlacement {
             row_group_rows: first_row_group_rows,
             decoded_bytes_per_row: 0.0,
             frozen: false,
-            changes_made: 0,
-            boundaries_since_change: 0,
+            hysteresis: ChangeHysteresis::default(),
             changes,
         };
         placement.place_all(first_row_group_rows, true);
@@ -304,8 +304,7 @@ impl FilePlacement {
             }
         }
         self.row_group_rows = next_row_group_rows;
-        self.boundaries_since_change += 1;
-        if self.boundaries_since_change <= self.change_hold() {
+        if !self.hysteresis.boundary() {
             return None;
         }
         let row_filter = self.row_filter_candidates();
@@ -315,22 +314,12 @@ impl FilePlacement {
             return None;
         }
         self.changes.add(1);
-        self.changes_made += 1;
-        self.boundaries_since_change = 0;
+        self.hysteresis.changed();
         Some(PlacementChange {
             row_filter: self.row_filter_candidates() != row_filter,
         })
     }
 
-    /// Row group boundaries to wait after the last change before the next
-    /// change: `2^(n-1) - 1` after the `n`-th change.
-    fn change_hold(&self) -> usize {
-        1usize
-            .checked_shl(self.changes_made)
-            .map_or(usize::MAX, |doubled| doubled / 2)
-            .saturating_sub(1)
-    }
-
     /// Decides the placement of each conjunct and the evaluation order.
     fn place_all(&mut self, row_group_rows: usize, first: bool) {
         let decode_ns_per_byte = self.sites.decode().ns_per_byte();
@@ -374,7 +363,8 @@ impl FilePlacement {
                 saving.set_row_filter(conjunct.placement == Placement::RowFilter);
             }
         }
-        self.order = evaluation_order(&observations);
+        let costs: Vec<_> = observations.iter().map(Observation::cost).collect();
+        self.order = evaluation_order(&costs);
     }
 
     /// The placement and the evaluation order of the managed conjuncts.
diff --git a/datafusion/datasource-parquet/src/filter_placement/model.rs b/datafusion/datasource-parquet/src/filter_placement/model.rs
index 57606fe403d4b..6a82d69c1430e 100644
--- a/datafusion/datasource-parquet/src/filter_placement/model.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/model.rs
@@ -21,6 +21,7 @@
 use datafusion_physical_expr::filter_stats::MIN_OBSERVED_ROWS;
 
 use super::stats::Observation;
+use crate::row_filter_cost;
 use crate::row_filter_cost::ROW_FILTER_STAGE_NS_PER_ROW;
 
 /// Where the scan evaluates one conjunct of its predicate.
@@ -168,30 +169,10 @@ pub(crate) fn explore(
     }
 }
 
-/// The rank of a conjunct in the evaluation order of its stage: the rows
-/// that it removed for each nanosecond of evaluation time
-/// ([`FilterCost::rows_removed_per_nano`], the ranking key of the adaptive
-/// conjunct order of `FilterExec`). `None` before [`MIN_OBSERVED_ROWS`]
-/// evaluated rows.
+/// The rank of a conjunct in the evaluation order of its stage, see
+/// [`row_filter_cost::rank`].
 pub(crate) fn rank(observation: &Observation) -> Option<f64> {
-    if observation.rows_in < MIN_OBSERVED_ROWS {
-        return None;
-    }
-    observation.cost().rows_removed_per_nano()
-}
-
-/// The evaluation order of conjuncts, as indexes into `observations`. The
-/// same rule for required and optional conjuncts: a larger [`rank`] first.
-/// A conjunct without a rank comes first, in the written order, so that it
-/// is measured on all rows of its stage (see also [`explore`]).
-pub(crate) fn evaluation_order(observations: &[Observation]) -> Vec<usize> {
-    let mut order: Vec<usize> = (0..observations.len()).collect();
-    // A stable sort keeps the written order for equal keys.
-    order.sort_by(|&a, &b| {
-        let key = |i: usize| rank(&observations[i]).unwrap_or(f64::INFINITY);
-        key(b).total_cmp(&key(a))
-    });
-    order
+    row_filter_cost::rank(&observation.cost())
 }
 
 #[cfg(test)]
@@ -371,26 +352,4 @@ mod tests {
         );
         assert_eq!(explore(Placement::Skip, &unmeasured, true), Placement::Skip);
     }
-
-    #[test]
-    fn order_by_rows_removed_per_nanosecond() {
-        let rows = MIN_OBSERVED_ROWS * 10;
-        // Removes 90% at 1 ns for each row: 0.9 rows for each ns.
-        let cheap_selective = measured(rows, rows / 10, rows);
-        // Removes 95% at 16 ns for each row: about 0.06 rows for each ns.
-        let expensive = measured(rows, rows / 20, 16 * rows);
-        // Removes nothing at 1 ns for each row.
-        let useless = measured(rows, rows, rows);
-        let unmeasured = measured(MIN_OBSERVED_ROWS - 1, 0, 1);
-        assert_eq!(rank(&unmeasured), None);
-        assert_eq!(
-            evaluation_order(&[expensive, useless, cheap_selective]),
-            vec![2, 0, 1]
-        );
-        // Unmeasured conjuncts first, in the written order.
-        assert_eq!(
-            evaluation_order(&[expensive, unmeasured, cheap_selective, unmeasured]),
-            vec![1, 3, 2, 0]
-        );
-    }
 }

From 102627d53ef8ea50582fc2b9bbf17e2894825f7f Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 12:53:51 -0500
Subject: [PATCH 34/42] feat(parquet): add the measured post-scan copy to the
 benefit of a row filter

The placement model compared the decode time that a row filter saves with
the cost of a row filter stage, but the post-scan filter has a cost that
a row filter does not have: after a selective conjunct it copies the
working batch to the rows that pass (the compaction). The model did not
see it, thus it kept selective conjuncts in the post-scan filter where a
row filter was faster (TPC-DS Q72 and Q38 `catalog_sales`).

The post-scan filter now times each copy and adds it to the measurements
of the conjunct after which it copies. The model adds this time, for
each row of the input batches of the post-scan filter, to the benefit of
a row filter. There is no new constant: the term is measured.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/decoder_projection.rs                 | 10 ++++
 .../src/filter_placement/model.rs             | 29 ++++++++++--
 .../src/filter_placement/stats.rs             | 47 +++++++++++++++++--
 3 files changed, 77 insertions(+), 9 deletions(-)

diff --git a/datafusion/datasource-parquet/src/decoder_projection.rs b/datafusion/datasource-parquet/src/decoder_projection.rs
index 9554829d4a2d4..0a063b5ef17a5 100644
--- a/datafusion/datasource-parquet/src/decoder_projection.rs
+++ b/datafusion/datasource-parquet/src/decoder_projection.rs
@@ -246,7 +246,13 @@ impl PostScanFilter {
                 if let Some(stages) = stages.as_mut() {
                     stages.compact(&folded);
                 }
+                // The placement model measures the copy: a row filter does
+                // not make it.
+                let copy_start = stats.as_ref().map(|_| Instant::now());
                 working = filter_record_batch(&working, &folded)?;
+                if let (Some(stats), Some(copy_start)) = (stats, copy_start) {
+                    stats.record_copy(duration_nanos(copy_start.elapsed()));
+                }
                 acc = None;
             } else {
                 acc = Some(folded);
@@ -722,6 +728,10 @@ mod tests {
             (first.rows_in, first.rows_out, first.skippable_rows),
             (256, 64, 0)
         );
+        // The first conjunct keeps 25% of the rows: the loop copies the
+        // working batch after it, and measures the copy for it.
+        assert_eq!(first.post_scan_rows, 256);
+        assert!(first.copy_nanos > 0);
         // The rows that the first conjunct removed pass: no empty window.
         let second = second.observation();
         assert_eq!(
diff --git a/datafusion/datasource-parquet/src/filter_placement/model.rs b/datafusion/datasource-parquet/src/filter_placement/model.rs
index 6a82d69c1430e..2d5e86b821d38 100644
--- a/datafusion/datasource-parquet/src/filter_placement/model.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/model.rs
@@ -67,14 +67,19 @@ pub(crate) struct ConjunctInputs {
 }
 
 impl ConjunctInputs {
-    /// Decode time, in nanoseconds for each evaluated row, that a row
-    /// filter saves. `None` without enough measurements.
+    /// Time, in nanoseconds for each evaluated row, that a row filter
+    /// saves: the decode of the unread output columns for the skippable
+    /// rows, and the measured copies of the post-scan filter after the
+    /// conjunct. `None` without enough measurements.
     pub(crate) fn benefit_ns_per_row(&self) -> Option<f64> {
         if self.observation.rows_in < MIN_OBSERVED_ROWS {
             return None;
         }
         let skippable = self.observation.skippable_fraction()?;
-        Some(skippable * self.unread_output_bytes_per_row * self.decode_ns_per_byte)
+        Some(
+            skippable * self.unread_output_bytes_per_row * self.decode_ns_per_byte
+                + self.observation.copy_ns_per_row(),
+        )
     }
 
     /// The extra time, in nanoseconds for each evaluated row, of a row
@@ -99,6 +104,7 @@ impl ConjunctInputs {
 ///
 /// ```text
 /// benefit = skippable rows / rows in * unread output bytes * decode ns for each byte
+///         + measured post-scan copy ns for each row
 /// cost    = fixed stage cost
 ///         + fetch latency / rows of the next row group
 ///         + rows out / rows in * read output bytes * decode ns for each byte
@@ -192,7 +198,7 @@ mod tests {
                 rows_in,
                 rows_out: rows_in - skippable_rows,
                 skippable_rows,
-                nanos: 0,
+                ..Default::default()
             },
             unread_output_bytes_per_row,
             read_output_bytes_per_row: 0.0,
@@ -250,6 +256,21 @@ mod tests {
         );
     }
 
+    /// The copies of the post-scan filter after the conjunct are part of
+    /// the benefit of a row filter.
+    #[test]
+    fn measured_post_scan_copy() {
+        let rows = 100_000;
+        // benefit = 0.1 * 10 * 4 = 4 ns, cost = 8 ns: post-scan.
+        let mut copying = inputs(rows, rows / 10, 10.0, 0.0);
+        assert_eq!(place_required(&copying, None), Placement::PostScan);
+        // The copies after the conjunct took 6 ns for each row: benefit =
+        // 10 ns, more than 8 ns / 0.9.
+        copying.observation.post_scan_rows = rows;
+        copying.observation.copy_nanos = 6 * rows;
+        assert_eq!(place_required(&copying, None), Placement::RowFilter);
+    }
+
     #[test]
     fn double_decode_of_read_output_columns() {
         let rows = 100_000;
diff --git a/datafusion/datasource-parquet/src/filter_placement/stats.rs b/datafusion/datasource-parquet/src/filter_placement/stats.rs
index 76e41466a76de..008a556e278df 100644
--- a/datafusion/datasource-parquet/src/filter_placement/stats.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/stats.rs
@@ -33,8 +33,8 @@ use parking_lot::Mutex;
 use crate::optional_filter::DEFAULT_DECODE_NS_PER_BYTE;
 use crate::row_filter_cost::{SKIP_WINDOW_ROWS, skippable_in, skippable_rows};
 
-/// What the scan measured for one conjunct: in the row filter or in the
-/// post-scan filter, and in the row group statistics pruning.
+/// What the scan measured for one conjunct, in the row filter or in the
+/// post-scan filter.
 #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
 pub(crate) struct Observation {
     /// Rows that the conjunct was evaluated on.
@@ -46,9 +46,26 @@ pub(crate) struct Observation {
     pub(crate) skippable_rows: u64,
     /// Evaluation time of the conjunct, in nanoseconds.
     pub(crate) nanos: u64,
+    /// Rows of the input batches of the post-scan filter where the
+    /// conjunct was evaluated.
+    pub(crate) post_scan_rows: u64,
+    /// Time of the copies (compactions of the working batch) of the
+    /// post-scan filter after the conjunct, in nanoseconds.
+    pub(crate) copy_nanos: u64,
 }
 
 impl Observation {
+    /// The time of the copies that the post-scan filter makes after the
+    /// conjunct, in nanoseconds for each row of its input batches. 0 before
+    /// a post-scan evaluation. A row filter does not copy: the decoder
+    /// produces only the rows that pass.
+    pub(crate) fn copy_ns_per_row(&self) -> f64 {
+        if self.post_scan_rows == 0 {
+            return 0.0;
+        }
+        self.copy_nanos as f64 / self.post_scan_rows as f64
+    }
+
     /// The rows in, rows out and evaluation time.
     pub(crate) fn cost(&self) -> FilterCost {
         FilterCost {
@@ -78,6 +95,8 @@ pub(crate) struct ConjunctStats {
     rows_out: AtomicU64,
     skippable_rows: AtomicU64,
     nanos: AtomicU64,
+    post_scan_rows: AtomicU64,
+    copy_nanos: AtomicU64,
     /// The generation of the conjunct that the evaluation measurements are
     /// for, see [`Self::observe_generation`].
     generation: AtomicU64,
@@ -134,9 +153,14 @@ impl ConjunctStats {
             .fetch_add(skippable as u64, Ordering::Relaxed);
     }
 
-    /// Clears the evaluation measurements (not the statistics pruning
-    /// result) when `generation` is newer than the generation that they are
-    /// for. `generation` is the sum of the generations of the dynamic
+    /// Records a copy (a compaction of the working batch) of the post-scan
+    /// filter after the conjunct, which took `nanos` nanoseconds.
+    pub(crate) fn record_copy(&self, nanos: u64) {
+        self.copy_nanos.fetch_add(nanos, Ordering::Relaxed);
+    }
+
+    /// Clears the measurements when `generation` is newer than the
+    /// generation that they are for. `generation` is the sum of the generations of the dynamic
     /// filters in the conjunct ([`snapshot_generation`]): a new generation
     /// is a new filter, and the measurements of the old filter do not
     /// describe it. Generations only increase, thus a caller with an older
@@ -150,6 +174,8 @@ impl ConjunctStats {
                 &self.rows_out,
                 &self.skippable_rows,
                 &self.nanos,
+                &self.post_scan_rows,
+                &self.copy_nanos,
             ] {
                 counter.store(0, Ordering::Relaxed);
             }
@@ -163,6 +189,8 @@ impl ConjunctStats {
             rows_out: self.rows_out.load(Ordering::Relaxed),
             skippable_rows: self.skippable_rows.load(Ordering::Relaxed),
             nanos: self.nanos.load(Ordering::Relaxed),
+            post_scan_rows: self.post_scan_rows.load(Ordering::Relaxed),
+            copy_nanos: self.copy_nanos.load(Ordering::Relaxed),
         }
     }
 }
@@ -218,6 +246,9 @@ impl StageSelection {
         nanos: u64,
     ) {
         let values = passed.values();
+        stats
+            .post_scan_rows
+            .fetch_add(self.input_rows as u64, Ordering::Relaxed);
         match &self.complete_windows {
             None => stats.record(
                 self.input_rows,
@@ -454,6 +485,7 @@ mod tests {
                 rows_out: 65,
                 skippable_rows: 128,
                 nanos: 30,
+                ..Default::default()
             }
         );
         assert_eq!(observation.skippable_fraction(), Some(0.5));
@@ -476,6 +508,7 @@ mod tests {
                 rows_in: 256,
                 rows_out: 64,
                 skippable_rows: 0,
+                post_scan_rows: 256,
                 ..Default::default()
             }
         );
@@ -493,6 +526,7 @@ mod tests {
                 rows_in: 256,
                 rows_out: 256 - 32,
                 skippable_rows: 0,
+                post_scan_rows: 256,
                 ..Default::default()
             }
         );
@@ -513,6 +547,7 @@ mod tests {
                 rows_in: 256,
                 rows_out: 128,
                 skippable_rows: 128,
+                post_scan_rows: 256,
                 ..Default::default()
             }
         );
@@ -542,6 +577,7 @@ mod tests {
                 rows_in: 256,
                 rows_out: 256 - 65,
                 skippable_rows: 64,
+                post_scan_rows: 256,
                 ..Default::default()
             }
         );
@@ -575,6 +611,7 @@ mod tests {
                 rows_in: 256,
                 rows_out: 192,
                 skippable_rows: 0,
+                post_scan_rows: 256,
                 ..Default::default()
             }
         );

From b964d740443ea58df9cd2fc601b11b86d77ec711 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 13:08:50 -0500
Subject: [PATCH 35/42] test(parquet): size the optional filter integration
 tests for gate windows

A gate decides only on windows of at least `MIN_OBSERVED_ROWS` rows, thus
the tests use batches of 4096 rows (and more row groups for the test with
15-row row groups). The row group size is a multiple of the period of `a`
and of the write batch size, so that the page index cannot prune a page
with only a tail of the row group.

PR: #25682

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/core/tests/parquet/optional_filters.rs | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/datafusion/core/tests/parquet/optional_filters.rs b/datafusion/core/tests/parquet/optional_filters.rs
index 73fdd45a9d4b0..6e72e1213f13f 100644
--- a/datafusion/core/tests/parquet/optional_filters.rs
+++ b/datafusion/core/tests/parquet/optional_filters.rs
@@ -64,9 +64,13 @@ use tempfile::NamedTempFile;
 use crate::parquet::utils::MetricsFinder;
 
 const ROW_GROUPS: usize = 10;
-const ROWS_PER_ROW_GROUP: usize = 2000;
-/// Each row filter evaluation sees one batch of this many rows.
-const BATCH_SIZE: usize = 100;
+/// More than 6 batches. A multiple of 100 (the period of `a`) and of 1024
+/// (the write batch size of the writer), thus no page of `a` holds only a
+/// tail of values that the page index could prune.
+const ROWS_PER_ROW_GROUP: usize = 25_600;
+/// Each row filter evaluation sees one batch of this many rows. A gate
+/// decides on windows of at least 2 batches and `MIN_OBSERVED_ROWS` rows.
+const BATCH_SIZE: usize = 4096;
 
 /// A file with `ROW_GROUPS` row groups. Column `a` is `i % 100` (each row
 /// group has all values `0..100`, thus statistics cannot prune it), `rg` is
@@ -490,10 +494,11 @@ async fn optional_filter_is_not_evaluated_post_scan() {
 /// filter from the row filter for complete row groups, and counts the rows of
 /// these row groups as skipped. It must count the rows that the row groups
 /// have, not the batch size: here each row group has 15 rows and the batch
-/// size is 8192.
+/// size is 8192. There are enough row groups for a window of the gate
+/// (`MIN_OBSERVED_ROWS` rows).
 #[tokio::test]
 async fn skipped_rows_of_placement_are_the_rows_of_the_row_groups() {
-    let row_groups = 20;
+    let row_groups = 1000;
     let rows_per_row_group = 15;
     let total_rows = row_groups * rows_per_row_group;
     let (file, schema) = write_file_with(row_groups, rows_per_row_group);

From 48be981c5e7c332336389772232510222090937b Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 14:27:35 -0500
Subject: [PATCH 36/42] feat(parquet): order the post-scan conjuncts by their
 measurements at each batch

The post-scan filter evaluated its conjuncts in the order of the
placement, which changes only at a row group boundary. A file range with
one or two row groups (TPC-DS SF1 `store_sales` has one file for 12
partitions) kept the written order until its end, before any pooled
measurement: in TPC-DS Q24 the join filter on `ss_ticket_number` (a hash
lookup, 12 ns for each row) ran on all rows before the filter on
`ss_store_sk` (2 ns for each row) that removes all of them.

Changing the post-scan order needs no decoder rebuild, thus the post-scan
filter now orders its conjuncts at each batch with the same rule as the
placement (`evaluation_order`: rows removed for each nanosecond, from the
pooled measurements, unmeasured conjuncts first in their order).

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/decoder_projection.rs                 | 80 ++++++++++++++++---
 1 file changed, 70 insertions(+), 10 deletions(-)

diff --git a/datafusion/datasource-parquet/src/decoder_projection.rs b/datafusion/datasource-parquet/src/decoder_projection.rs
index 0a063b5ef17a5..ce8e9dfa54c36 100644
--- a/datafusion/datasource-parquet/src/decoder_projection.rs
+++ b/datafusion/datasource-parquet/src/decoder_projection.rs
@@ -42,7 +42,7 @@ use arrow::datatypes::SchemaRef;
 use datafusion_common::cast::as_boolean_array;
 use datafusion_common::instant::Instant;
 use datafusion_common::{Result, internal_err};
-use datafusion_physical_expr::filter_stats::duration_nanos;
+use datafusion_physical_expr::filter_stats::{FilterCost, duration_nanos};
 use datafusion_physical_expr::optional_filter_gate::GateDecision;
 use datafusion_physical_expr::projection::{ProjectionExprs, Projector};
 use datafusion_physical_expr::split_conjunction;
@@ -59,6 +59,7 @@ use crate::filter_placement::{ConjunctStats, StageSelection};
 use crate::opener::{VirtualColumnsState, append_fields};
 use crate::projection_read_plan::build_projection_read_plan;
 use crate::row_filter::SharedOptionalFilterGate;
+use crate::row_filter_cost::evaluation_order;
 
 /// Stream-schema column indices referenced by `projection`, in ascending
 /// order, or `None` when the projection already reads every column of
@@ -138,9 +139,9 @@ pub(crate) enum PostScanSelection {
 /// into [`ParquetFileMetrics`] for `EXPLAIN ANALYZE`.
 pub(crate) struct PostScanFilter {
     /// The `AND` conjuncts of the predicate, rebased onto the decoder's stream
-    /// schema, in the order they are evaluated, each with the measurements
-    /// of the adaptive filter placement (if the placement manages it). Never
-    /// empty.
+    /// schema, each with the measurements of the adaptive filter placement
+    /// (if the placement manages it). Never empty. They are evaluated in the
+    /// order of [`Self::evaluation_order`].
     conjuncts: Vec<PostScanConjunct>,
     rows_pruned: Count,
     rows_matched: Count,
@@ -180,15 +181,12 @@ impl PostScanFilter {
             .then(|| StageSelection::new(input_rows));
 
         let last = self.conjuncts.len() - 1;
-        for (
-            position,
-            PostScanConjunct {
+        for (position, index) in self.evaluation_order().into_iter().enumerate() {
+            let PostScanConjunct {
                 expr: conjunct,
                 stats,
                 gate,
-            },
-        ) in self.conjuncts.iter().enumerate()
-        {
+            } = &self.conjuncts[index];
             let rows_in = working.num_rows();
             // An optional conjunct that its gate skips lets all rows pass.
             let mut gate = gate.as_ref().map(|gate| gate.lock());
@@ -270,6 +268,30 @@ impl PostScanFilter {
         })
     }
 
+    /// The order to evaluate the conjuncts for the next batch, as indexes
+    /// into `conjuncts`: by the pooled measurements of the conjuncts (rows
+    /// removed for each nanosecond, see
+    /// [`evaluation_order`](crate::row_filter_cost::evaluation_order)), the
+    /// conjuncts without enough measurements first in their order. The
+    /// post-scan filter can change its order at each batch for free, thus a
+    /// file with few row groups does not keep the written order until its
+    /// end (TPC-DS Q24: an expensive join filter ran before a cheap one that
+    /// removes all rows).
+    fn evaluation_order(&self) -> Vec<usize> {
+        let costs: Vec<FilterCost> = self
+            .conjuncts
+            .iter()
+            .map(|conjunct| {
+                conjunct
+                    .stats
+                    .as_ref()
+                    .map(|stats| stats.observation().cost())
+                    .unwrap_or_default()
+            })
+            .collect();
+        evaluation_order(&costs)
+    }
+
     /// Record one batch's contribution to the rows-matched / rows-pruned
     /// metrics. `rows_pruned` stays "total rows in, minus final survivors"
     /// regardless of how many intermediate compactions the loop performed.
@@ -790,6 +812,44 @@ mod tests {
         assert_eq!(survivors(&filter, input()).len(), rows as usize / 2);
     }
 
+    /// The conjuncts run in the order of their pooled measurements: the
+    /// conjunct that removes more rows for each nanosecond first. The
+    /// result does not change.
+    #[test]
+    fn conjuncts_run_in_measured_order() {
+        let rows = 16 * 8192;
+        let expensive = Arc::new(ConjunctStats::default());
+        let cheap = Arc::new(ConjunctStats::default());
+        let filter = PostScanFilter {
+            conjuncts: vec![
+                PostScanConjunct {
+                    expr: gt("a", 0, 49),
+                    stats: Some(Arc::clone(&expensive)),
+                    gate: None,
+                },
+                PostScanConjunct {
+                    expr: gt("b", 1, 149),
+                    stats: Some(Arc::clone(&cheap)),
+                    gate: None,
+                },
+            ],
+            rows_pruned: Count::new(),
+            rows_matched: Count::new(),
+            eval_time: Time::new(),
+        };
+        // Unmeasured: the written order.
+        assert_eq!(filter.evaluation_order(), vec![0, 1]);
+        // The second conjunct removes more rows for each nanosecond.
+        expensive.record(rows, rows / 2, 0, 20 * rows as u64);
+        cheap.record(rows, rows / 4, 0, rows as u64);
+        assert_eq!(filter.evaluation_order(), vec![1, 0]);
+        let input = batch((0..200).map(Some).collect());
+        assert_eq!(
+            survivors(&filter, input),
+            (150..200).map(Some).collect::<Vec<_>>()
+        );
+    }
+
     #[test]
     fn compacts_between_conjuncts_without_losing_rows() {
         // 100 rows; `a > 79` keeps 20 (20% — below the compaction threshold, so

From e6a798e40097a6dfbc408d7a7edfe554bc39a378 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Fri, 25 Sep 2026 19:37:30 -0500
Subject: [PATCH 37/42] fix(parquet): fix rustdoc links of the adaptive filter
 placement

`cargo doc` with `-D warnings` rejected a link to the removed
`ConjunctSiteKey` and two redundant explicit link targets.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/datasource-parquet/src/decoder_projection.rs     | 2 +-
 datafusion/datasource-parquet/src/filter_placement/stats.rs | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/datafusion/datasource-parquet/src/decoder_projection.rs b/datafusion/datasource-parquet/src/decoder_projection.rs
index ce8e9dfa54c36..e1e140aa12319 100644
--- a/datafusion/datasource-parquet/src/decoder_projection.rs
+++ b/datafusion/datasource-parquet/src/decoder_projection.rs
@@ -271,7 +271,7 @@ impl PostScanFilter {
     /// The order to evaluate the conjuncts for the next batch, as indexes
     /// into `conjuncts`: by the pooled measurements of the conjuncts (rows
     /// removed for each nanosecond, see
-    /// [`evaluation_order`](crate::row_filter_cost::evaluation_order)), the
+    /// [`evaluation_order`]), the
     /// conjuncts without enough measurements first in their order. The
     /// post-scan filter can change its order at each batch for free, thus a
     /// file with few row groups does not keep the written order until its
diff --git a/datafusion/datasource-parquet/src/filter_placement/stats.rs b/datafusion/datasource-parquet/src/filter_placement/stats.rs
index 008a556e278df..f007f9416fe97 100644
--- a/datafusion/datasource-parquet/src/filter_placement/stats.rs
+++ b/datafusion/datasource-parquet/src/filter_placement/stats.rs
@@ -42,7 +42,7 @@ pub(crate) struct Observation {
     /// Rows that passed the conjunct.
     pub(crate) rows_out: u64,
     /// Rows in windows where no row passed the conjunct, see
-    /// [`skippable_rows`](crate::row_filter_cost::skippable_rows) and [`StageSelection`].
+    /// [`skippable_rows`] and [`StageSelection`].
     pub(crate) skippable_rows: u64,
     /// Evaluation time of the conjunct, in nanoseconds.
     pub(crate) nanos: u64,
@@ -420,7 +420,7 @@ impl PlacementSites {
 
     /// The statistics of the conjunct at `position` in `conjuncts` (the
     /// conjuncts of the root `AND` chain of a file predicate). The scan
-    /// predicate has `scan_count` conjuncts. See [`ConjunctSiteKey`].
+    /// predicate has `scan_count` conjuncts. See [`ConjunctKey`].
     pub(crate) fn stats_for(
         &self,
         conjuncts: &[&Arc<dyn PhysicalExpr>],

From 4a3820eacc9a12b616fdf61869baa2200195a4d2 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Sat, 26 Sep 2026 12:07:16 -0500
Subject: [PATCH 38/42] refactor(physical-expr): make PRE_SELECTION_THRESHOLD
 public

The post-scan filter of the Parquet scan uses the pre-selection threshold
of `AND` to decide when to compact its working batch (next commit). This
is the same change as in the FilterExec reordering PR (#25721), which
models the same rule. When one of the two PRs merges, this commit becomes
empty in the other.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/physical-expr/src/expressions/binary.rs | 6 +++++-
 datafusion/physical-expr/src/expressions/mod.rs    | 2 +-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/datafusion/physical-expr/src/expressions/binary.rs b/datafusion/physical-expr/src/expressions/binary.rs
index a0a6548518737..484e11e857e7c 100644
--- a/datafusion/physical-expr/src/expressions/binary.rs
+++ b/datafusion/physical-expr/src/expressions/binary.rs
@@ -1236,7 +1236,11 @@ enum ShortCircuitStrategy {
 /// the side that cannot short-circuit the operator is rare:
 /// - for `AND`, when the proportion of `true` is less than or equal to 0.2
 /// - for `OR`, when the proportion of `false` is less than or equal to 0.2
-const PRE_SELECTION_THRESHOLD: f32 = 0.2;
+///
+/// Public only so that the adaptive conjunct ordering of `FilterExec` can
+/// model the same rule. Not part of the stable API.
+#[doc(hidden)]
+pub const PRE_SELECTION_THRESHOLD: f32 = 0.2;
 
 /// Checks if a logical operator (`AND`/`OR`) can short-circuit evaluation based on the left-hand side (lhs) result.
 ///
diff --git a/datafusion/physical-expr/src/expressions/mod.rs b/datafusion/physical-expr/src/expressions/mod.rs
index d36f734afef39..af348989e4dd7 100644
--- a/datafusion/physical-expr/src/expressions/mod.rs
+++ b/datafusion/physical-expr/src/expressions/mod.rs
@@ -42,7 +42,7 @@ pub use crate::PhysicalSortExpr;
 /// Module with some convenient methods used in expression building
 pub use crate::aggregate::stats::StatsType;
 
-pub use binary::{BinaryExpr, binary, similar_to};
+pub use binary::{BinaryExpr, PRE_SELECTION_THRESHOLD, binary, similar_to};
 pub use case::{CaseExpr, case};
 pub use cast::{CastExpr, cast};
 pub use column::{Column, col, with_new_schema};

From 262713058d57dbac93e9c9ce068ed250d9f36ee7 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Sat, 26 Sep 2026 12:07:21 -0500
Subject: [PATCH 39/42] fix(parquet): compact the post-scan batch with the
 pre-selection threshold of AND

The post-scan filter copied the working batch (all its columns) to the
surviving rows when a conjunct kept at most 80% of them. The caller then
copies the surviving rows again when it applies the final mask. For a
cheap range predicate that keeps about half of the rows, the first copy
costs much more than the evaluation that it saves on the next conjunct.

TPC-DS Q82, `inventory` scan with `inv_quantity_on_hand BETWEEN 100 AND
500`, SF1, all 11.7M rows (EXPLAIN ANALYZE, 3 runs):

| | post-scan filter eval | scan compute |
|---|---|---|
| threshold 0.8 | 28.4 to 29.2 ms | 104 to 106 ms |
| threshold 0.2 | 4.5 to 4.6 ms | 74 to 78 ms |
| main, `FilterExec` above the scan | 11.7 to 14.1 ms (`FilterExec`) | 64 to 66 ms |

The threshold is now `PRE_SELECTION_THRESHOLD` (0.2), the threshold of
`AND` in `BinaryExpr` that a `FilterExec` uses. Thus the post-scan filter
makes the same copy decision as the `FilterExec` that it replaces. The
old comment gave a `CASE` dynamic filter as the reason for 0.8; the
partitioned hash join dynamic filters are no longer `CASE` expressions,
and optional filters have gates and a measured order.

Without a compaction, the working batch also has the rows that the
earlier conjuncts removed. The measurements of a later conjunct (its
placement statistics and its gate) now count these rows as passing, as
they do after a compaction. Before, a later conjunct got the credit for
rows that it did not remove.

Wall time, min of 16 runs, pushdown on: TPC-DS Q82 1.04x -> 0.96x of
main. TPC-H Q14 1.08x -> 0.94x, Q15 1.09x -> 1.00x, Q12 0.94x ->
0.84x. No other TPC-H, TPC-DS or ClickBench query changed outside the
A/A noise. A threshold of 0 (never compact) is slower on TPC-H Q20
(1.14x) and TPC-DS Q42, Q52, Q55 (1.2x), thus the compaction stays.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../src/decoder_projection.rs                 | 116 ++++++++++++++----
 1 file changed, 91 insertions(+), 25 deletions(-)

diff --git a/datafusion/datasource-parquet/src/decoder_projection.rs b/datafusion/datasource-parquet/src/decoder_projection.rs
index e1e140aa12319..8ec4dc68fd0bc 100644
--- a/datafusion/datasource-parquet/src/decoder_projection.rs
+++ b/datafusion/datasource-parquet/src/decoder_projection.rs
@@ -42,6 +42,7 @@ use arrow::datatypes::SchemaRef;
 use datafusion_common::cast::as_boolean_array;
 use datafusion_common::instant::Instant;
 use datafusion_common::{Result, internal_err};
+use datafusion_physical_expr::expressions::PRE_SELECTION_THRESHOLD;
 use datafusion_physical_expr::filter_stats::{FilterCost, duration_nanos};
 use datafusion_physical_expr::optional_filter_gate::GateDecision;
 use datafusion_physical_expr::projection::{ProjectionExprs, Projector};
@@ -89,13 +90,15 @@ fn projector_input_indices(
 /// repaid by the (smaller) saving on the conjuncts that follow, so the masks are
 /// just combined with a cheap bitwise `AND` instead.
 ///
-/// The threshold is higher than the equivalent constant in `FilterExec`'s
-/// adaptive evaluator because the post-scan filter's conjunct mix is skewed:
-/// the conjuncts that land here at `pushdown_filters = false` are typically a
-/// cheap static range predicate followed by a *much* more expensive dynamic
-/// filter (a `CASE` over per-partition hash-table probes), so even a modest
-/// reduction in the row count reaching the later conjunct pays for the copy.
-const COMPACTION_SELECTIVITY_THRESHOLD: f64 = 0.8;
+/// This is the pre-selection threshold of `AND` in `BinaryExpr`, which a
+/// `FilterExec` uses: the post-scan filter makes the same copy decision as
+/// the `FilterExec` that it replaces. The copy is of all the columns of the
+/// working batch (the output columns and the filter columns), and the caller
+/// copies the surviving rows again when it applies the final mask. With a
+/// higher threshold, a cheap range predicate that keeps half of the rows
+/// (TPC-DS Q82 on `inventory`) made the post-scan filter cost 6x the
+/// predicate evaluation, and 2x the `FilterExec` of main.
+const COMPACTION_SELECTIVITY_THRESHOLD: f64 = PRE_SELECTION_THRESHOLD as f64;
 
 /// Outcome of running the post-scan predicate over one decoded batch.
 ///
@@ -188,10 +191,15 @@ impl PostScanFilter {
                 gate,
             } = &self.conjuncts[index];
             let rows_in = working.num_rows();
+            // The rows of the working batch that the conjuncts before this one
+            // let pass. Without a compaction, the working batch also has the
+            // rows that they removed: the measurements of this conjunct do
+            // not count those rows.
+            let live_in = acc.as_ref().map_or(rows_in, |live| live.true_count());
             // An optional conjunct that its gate skips lets all rows pass.
             let mut gate = gate.as_ref().map(|gate| gate.lock());
             if let Some(gate) = gate.as_mut()
-                && gate.begin_batch(rows_in) == GateDecision::Skip
+                && gate.begin_batch(live_in) == GateDecision::Skip
             {
                 continue;
             }
@@ -209,14 +217,30 @@ impl PostScanFilter {
                 Some(_) => prep_null_mask_filter(mask),
                 None => mask.clone(),
             };
-            if let (Some(stages), Some(stats), Some(measure_start)) =
-                (stages.as_ref(), stats, measure_start)
+            let nanos = measure_start.map(|start| duration_nanos(start.elapsed()));
+            let gate_nanos = gate
+                .as_ref()
+                .zip(start)
+                .map(|(gate, start)| gate.now_nanos().saturating_sub(start));
+            // For the measurements, the rows that the conjuncts before this
+            // one removed pass: this conjunct did not remove them.
+            let measured =
+                (nanos.is_some() || gate_nanos.is_some()).then(|| match &acc {
+                    Some(live) => {
+                        BooleanArray::new(mask.values() | &!live.values(), None)
+                    }
+                    None => mask.clone(),
+                });
+            if let (Some(stages), Some(stats), Some(nanos), Some(measured)) =
+                (stages.as_ref(), stats, nanos, measured.as_ref())
             {
-                stages.record(stats, &mask, duration_nanos(measure_start.elapsed()));
+                stages.record(stats, measured, nanos);
             }
-            if let (Some(gate), Some(start)) = (gate.as_mut(), start) {
-                let elapsed = gate.now_nanos().saturating_sub(start);
-                gate.record(rows_in, mask.true_count(), Duration::from_nanos(elapsed));
+            if let (Some(gate), Some(gate_nanos), Some(measured)) =
+                (gate.as_mut(), gate_nanos, measured.as_ref())
+            {
+                let live_out = measured.true_count() - (rows_in - live_in);
+                gate.record(live_in, live_out, Duration::from_nanos(gate_nanos));
             }
             drop(gate);
             // An all-true conjunct leaves the accumulated selection untouched.
@@ -708,27 +732,27 @@ mod tests {
     /// removed count as passing.
     #[test]
     fn measures_skippable_rows_on_input_positions() {
-        // (a % 4) = 0: keeps every fourth row, no empty window. The loop then
-        // compacts the batch to 64 rows.
-        let every_fourth: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
+        // (a % 8) = 0: keeps every eighth row, no empty window. The loop then
+        // compacts the batch to 32 rows.
+        let every_eighth: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
             Arc::new(BinaryExpr::new(
                 Arc::new(Column::new("a", 0)),
                 Operator::Modulo,
-                Arc::new(Literal::new(ScalarValue::Int32(Some(4)))),
+                Arc::new(Literal::new(ScalarValue::Int32(Some(8)))),
             )),
             Operator::Eq,
             Arc::new(Literal::new(ScalarValue::Int32(Some(0)))),
         ));
-        // b > 127: removes the first 32 rows of the compacted batch, that is
+        // b > 127: removes the first 16 rows of the compacted batch, that is
         // the live rows of the input rows 0..128. In the compacted batch this
-        // is half of one window of 64 rows.
+        // is part of one window of 64 rows.
         let upper_half = gt("b", 1, 127);
         let first = Arc::new(ConjunctStats::default());
         let second = Arc::new(ConjunctStats::default());
         let filter = PostScanFilter {
             conjuncts: vec![
                 PostScanConjunct {
-                    expr: every_fourth,
+                    expr: every_eighth,
                     stats: Some(Arc::clone(&first)),
                     gate: None,
                 },
@@ -743,14 +767,14 @@ mod tests {
             eval_time: Time::new(),
         };
         let rows = survivors(&filter, batch((0..256).map(Some).collect()));
-        assert_eq!(rows, (128..256).step_by(4).map(Some).collect::<Vec<_>>());
+        assert_eq!(rows, (128..256).step_by(8).map(Some).collect::<Vec<_>>());
 
         let first = first.observation();
         assert_eq!(
             (first.rows_in, first.rows_out, first.skippable_rows),
-            (256, 64, 0)
+            (256, 32, 0)
         );
-        // The first conjunct keeps 25% of the rows: the loop copies the
+        // The first conjunct keeps 12.5% of the rows: the loop copies the
         // working batch after it, and measures the copy for it.
         assert_eq!(first.post_scan_rows, 256);
         assert!(first.copy_nanos > 0);
@@ -758,7 +782,49 @@ mod tests {
         let second = second.observation();
         assert_eq!(
             (second.rows_in, second.rows_out, second.skippable_rows),
-            (256, 256 - 32, 0)
+            (256, 256 - 16, 0)
+        );
+    }
+
+    /// Without a compaction, the working batch still has the rows that an
+    /// earlier conjunct removed. A later conjunct does not get the credit
+    /// for these rows: they count as passing in its measurements.
+    #[test]
+    fn measures_only_live_rows_without_compaction() {
+        // a > 127 keeps half of the rows: no compaction.
+        let first = Arc::new(ConjunctStats::default());
+        // b > 63 removes the rows 0..64, which the first conjunct removed.
+        let second = Arc::new(ConjunctStats::default());
+        let filter = PostScanFilter {
+            conjuncts: vec![
+                PostScanConjunct {
+                    expr: gt("a", 0, 127),
+                    stats: Some(Arc::clone(&first)),
+                    gate: None,
+                },
+                PostScanConjunct {
+                    expr: gt("b", 1, 63),
+                    stats: Some(Arc::clone(&second)),
+                    gate: None,
+                },
+            ],
+            rows_pruned: Count::new(),
+            rows_matched: Count::new(),
+            eval_time: Time::new(),
+        };
+        let rows = survivors(&filter, batch((0..256).map(Some).collect()));
+        assert_eq!(rows, (128..256).map(Some).collect::<Vec<_>>());
+
+        let first = first.observation();
+        assert_eq!(
+            (first.rows_in, first.rows_out, first.skippable_rows),
+            (256, 128, 128)
+        );
+        assert_eq!(first.copy_nanos, 0);
+        let second = second.observation();
+        assert_eq!(
+            (second.rows_in, second.rows_out, second.skippable_rows),
+            (256, 256, 0)
         );
     }
 

From a9aa65539ac11276b9b85f47e8c51da4ac2de14d Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Sun, 27 Sep 2026 23:45:00 -0500
Subject: [PATCH 40/42] fix(parquet): no scan equivalences from a pruning-only
 predicate

#25780 on main added `FileSource::exact_filter`: the part of the filter
that every output row satisfies, the only part that the scan derives
equivalences from. Its `ParquetSource` version returns the pushable
conjuncts when `pushdown_filters` is on, and nothing otherwise. This PR
changes both cases:

- A pruning-only predicate (a filter that stays in a `FilterExec` above
  a scan that cannot give the target partitions) is used only to prune,
  also with `pushdown_filters = true`. `exact_filter` returned it, thus
  the scan claimed that `a` is constant for `a = 5`, the
  order-preserving repartition merged on `b` only, and
  `ORDER BY b LIMIT 1` returned 2 instead of 1. It now returns `None`.
- With `pushdown_filters = false` the scan applies the accepted
  conjuncts in the post-scan filter. They are exact, thus
  `exact_filter` returns them. This keeps the plans of this PR (for
  example no `SortExec` for `ORDER BY b` with `b = 2`).

Tests: a new case in `push_down_filter_parquet.slt` (a plan pin and two
results that were wrong: `2` for `LIMIT 1`, and `5 2 / 5 1` for the
order) and `exact_filter` checks in the pruning-only unit test. The plan
of the #25780 case with `pushdown_filters = false` changes: the scan
applies `a = 5` and there is no `FilterExec`.

PR: #22384

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/datasource-parquet/src/source.rs   | 22 ++++-
 .../test_files/push_down_filter_parquet.slt   | 84 +++++++++++++++++--
 .../library-user-guide/upgrading/56.0.0.md    |  4 +-
 3 files changed, 96 insertions(+), 14 deletions(-)

diff --git a/datafusion/datasource-parquet/src/source.rs b/datafusion/datasource-parquet/src/source.rs
index c0d4727b31777..7ecf7c0f82f25 100644
--- a/datafusion/datasource-parquet/src/source.rs
+++ b/datafusion/datasource-parquet/src/source.rs
@@ -709,11 +709,18 @@ impl FileSource for ParquetSource {
         self.predicate.clone()
     }
 
-    /// The predicate is applied to every row only when filter pushdown is
-    /// enabled, and then only the conjuncts that can become a `RowFilter`.
-    /// Otherwise the predicate is used only for pruning.
+    /// The scan applies each conjunct of its predicate to every row: as a
+    /// `RowFilter` predicate when filter pushdown is enabled, else (and for
+    /// the conjuncts that the `RowFilter` cannot evaluate) in the post-scan
+    /// filter. Only the conjuncts that can be pushed down are returned, the
+    /// same test that `try_pushdown_filters` uses before it replies
+    /// `PushedDown::Yes`.
+    ///
+    /// A pruning-only predicate (see
+    /// [`FileSource::try_pushdown_pruning_filters`]) is used only to prune:
+    /// a `FilterExec` above the scan applies it. Thus it is not exact.
     fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
-        if !self.pushdown_filters() {
+        if self.pruning_only_predicate {
             return None;
         }
         let predicate = self.predicate.as_ref()?;
@@ -2210,6 +2217,9 @@ mod tests {
             pruning.predicate.as_ref().unwrap().to_string(),
             "value@0 > 1"
         );
+        // A pruning-only predicate is not exact: a `FilterExec` above the
+        // scan applies it.
+        assert!(pruning.exact_filter().is_none());
 
         // It uses later filters only to prune too.
         let prop = pruning
@@ -2231,6 +2241,10 @@ mod tests {
         assert!(matches!(prop.filters[..], [PushedDown::Yes]));
         let applied = downcast(&prop.updated_node.unwrap());
         assert!(!applied.pruning_only_predicate);
+        // The scan applies the accepted filter to every row, also with
+        // `pushdown_filters = false` (in the post-scan filter).
+        assert!(!applied.pushdown_filters());
+        assert_eq!(applied.exact_filter().unwrap().to_string(), "value@0 > 1");
         assert!(
             applied
                 .try_pushdown_pruning_filters(&[filter(2)], &config)
diff --git a/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt b/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
index f343395a71ca4..ad7e2b9350e42 100644
--- a/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
+++ b/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
@@ -1337,11 +1337,11 @@ RESET datafusion.execution.parquet.pushdown_filters;
 statement ok
 drop table t;
 
-# A predicate that the Parquet scan keeps only for pruning
-# (pushdown_filters = false) must not make the scan claim equivalences.
-# Each file is sorted by (a, b). If the scan claims that `a` is constant,
-# the order-preserving repartition below the FilterExec merges on `b` only
-# and the rows with a = 5 come out in the wrong order.
+# A predicate that the Parquet scan keeps only for pruning must not make the
+# scan claim equivalences. Each file is sorted by (a, b). If the scan claims
+# that `a` is constant, the order-preserving repartition below the
+# FilterExec merges on `b` only and the rows with a = 5 come out in the
+# wrong order.
 
 statement ok
 COPY (SELECT * FROM (VALUES (3, 9), (5, 1)) v(a, b))
@@ -1371,15 +1371,14 @@ set datafusion.execution.parquet.pushdown_filters = false;
 statement ok
 set datafusion.explain.physical_plan_only = true;
 
-# The repartition merges on (a, b), the real file ordering
+# With pushdown_filters = false the scan applies `a = 5` in its post-scan
+# filter, thus `a` is constant in its output and each file is ordered by `b`.
 query TT
 EXPLAIN SELECT b FROM pruning_only_eq WHERE a = 5 ORDER BY b LIMIT 1;
 ----
 physical_plan
 01)SortPreservingMergeExec: [b@0 ASC NULLS LAST], fetch=1
-02)--FilterExec: a@0 = 5, projection=[b@1], fetch=1
-03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2, preserve_order=true, sort_exprs=a@0 ASC NULLS LAST, b@1 ASC NULLS LAST
-04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/pruning_only_eq/f1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/pruning_only_eq/f2.parquet]]}, projection=[a, b], output_ordering=[a@0 ASC NULLS LAST, b@1 ASC NULLS LAST], file_type=parquet, predicate=a@0 = 5, pruning_predicate=a_null_count@2 != row_count@3 AND a_min@0 <= 5 AND 5 <= a_max@1, required_guarantees=[a in (5)]
+02)--DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/pruning_only_eq/f1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/pruning_only_eq/f2.parquet]]}, projection=[b], limit=1, file_type=parquet, predicate=a@0 = 5, pruning_predicate=a_null_count@2 != row_count@3 AND a_min@0 <= 5 AND 5 <= a_max@1, required_guarantees=[a in (5)]
 
 query I
 SELECT b FROM pruning_only_eq WHERE a = 5 ORDER BY b LIMIT 1;
@@ -1427,6 +1426,73 @@ SELECT a, b FROM pruning_only_col_eq WHERE a = b ORDER BY a;
 3 3
 5 5
 
+# With `pushdown_filters = true`, a scan that cannot give the target
+# partitions keeps the filter in a `FilterExec` above it and gets it only
+# to prune (a pruning-only predicate). The scan must not claim equivalences
+# from this predicate. The first file has more rows than one batch, thus a
+# round-robin repartition is useful and the filter stays above the scan. The
+# table is created with `pushdown_filters = true`, thus its scan has filter
+# pushdown on. The page index is off, so that the scan reads the rows with
+# a = 3 before the row (5, 1) in the same batch, and the repartition merges
+# the batch of (5, 1) with the batch of (5, 2).
+statement ok
+set datafusion.execution.parquet.pushdown_filters = true;
+
+statement ok
+COPY (
+  SELECT a, b FROM (
+    SELECT 3 AS a, 9 AS b FROM generate_series(1, 32768)
+    UNION ALL
+    SELECT 5 AS a, 1 AS b
+  ) ORDER BY a, b
+)
+TO 'test_files/scratch/push_down_filter_parquet/pruning_only_eq_pushdown/f1.parquet'
+STORED AS PARQUET;
+
+statement ok
+COPY (SELECT * FROM (VALUES (5, 2)) v(a, b))
+TO 'test_files/scratch/push_down_filter_parquet/pruning_only_eq_pushdown/f2.parquet'
+STORED AS PARQUET;
+
+statement ok
+set datafusion.execution.parquet.enable_page_index = false;
+
+statement ok
+CREATE EXTERNAL TABLE pruning_only_eq_pushdown (a INT, b INT)
+STORED AS PARQUET
+LOCATION 'test_files/scratch/push_down_filter_parquet/pruning_only_eq_pushdown/'
+WITH ORDER (a ASC, b ASC);
+
+# The repartition merges on (a, b), the real file ordering
+query TT
+EXPLAIN SELECT b FROM pruning_only_eq_pushdown WHERE a = 5 ORDER BY b LIMIT 1;
+----
+physical_plan
+01)SortPreservingMergeExec: [b@0 ASC NULLS LAST], fetch=1
+02)--FilterExec: a@0 = 5, projection=[b@1], fetch=1
+03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2, preserve_order=true, sort_exprs=a@0 ASC NULLS LAST, b@1 ASC NULLS LAST
+04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/pruning_only_eq_pushdown/f1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/pruning_only_eq_pushdown/f2.parquet]]}, projection=[a, b], output_ordering=[a@0 ASC NULLS LAST, b@1 ASC NULLS LAST], file_type=parquet, predicate=a@0 = 5, pruning_predicate=a_null_count@2 != row_count@3 AND a_min@0 <= 5 AND 5 <= a_max@1, required_guarantees=[a in (5)]
+
+query I
+SELECT b FROM pruning_only_eq_pushdown WHERE a = 5 ORDER BY b LIMIT 1;
+----
+1
+
+query II
+SELECT a, b FROM pruning_only_eq_pushdown WHERE a = 5 ORDER BY b;
+----
+5 1
+5 2
+
+statement ok
+drop table pruning_only_eq_pushdown;
+
+statement ok
+RESET datafusion.execution.parquet.enable_page_index;
+
+statement ok
+set datafusion.execution.parquet.pushdown_filters = false;
+
 statement ok
 drop table pruning_only_eq;
 
diff --git a/docs/source/library-user-guide/upgrading/56.0.0.md b/docs/source/library-user-guide/upgrading/56.0.0.md
index 79067402a7dfa..aabdc464670e4 100644
--- a/docs/source/library-user-guide/upgrading/56.0.0.md
+++ b/docs/source/library-user-guide/upgrading/56.0.0.md
@@ -645,7 +645,9 @@ non-matching rows through, and equivalences derived from it gave wrong query
 results.
 
 `exact_filter` returns `None` by default. `ParquetSource` returns the pushed-down
-conjuncts when `pushdown_filters` is enabled.
+conjuncts: it applies them to every row, as a `RowFilter` or after the decode.
+It returns `None` for a pruning-only predicate (see
+`FileSource::try_pushdown_pruning_filters`).
 
 **Migration guide:**
 

From 4cd30d739bd3fcbaabd63e79199a69fd17848ffd Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Mon, 28 Sep 2026 00:46:52 -0500
Subject: [PATCH 41/42] fix(parquet): no scan equivalences from optional
 conjuncts

`ParquetSource::exact_filter` (#25780) returns the conjuncts that every
output row satisfies. An optional conjunct is not one of them: the scan
does not evaluate it after the decode (this PR), and it drops it when
the `RowFilter` cannot evaluate it. `exact_filter` now skips optional
conjuncts, so that the scan claims no equivalence from them.

Test: `exact_filter_excludes_optional_conjuncts` (failed before this
change: `a@0 = 1 AND Optional(b@1 = 2)` with `pushdown_filters = false`).

PR: #25722

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/datasource-parquet/src/source.rs | 53 ++++++++++++++++++++-
 1 file changed, 52 insertions(+), 1 deletion(-)

diff --git a/datafusion/datasource-parquet/src/source.rs b/datafusion/datasource-parquet/src/source.rs
index 7ecf7c0f82f25..dfa8286c87564 100644
--- a/datafusion/datasource-parquet/src/source.rs
+++ b/datafusion/datasource-parquet/src/source.rs
@@ -50,7 +50,7 @@ use datafusion_datasource::file_scan_config::FileScanConfig;
 use datafusion_functions::core::file_row_index::FileRowIndexFunc;
 use datafusion_physical_expr::expressions::{Column, DynamicFilterTracking};
 use datafusion_physical_expr::projection::ProjectionExprs;
-use datafusion_physical_expr::utils::split_conjunction;
+use datafusion_physical_expr::utils::{is_optional_filter, split_conjunction};
 use datafusion_physical_expr::{EquivalenceProperties, conjunction};
 use datafusion_physical_expr_adapter::DefaultPhysicalExprAdapterFactory;
 use datafusion_physical_expr_adapter::rewrite::{
@@ -719,6 +719,10 @@ impl FileSource for ParquetSource {
     /// A pruning-only predicate (see
     /// [`FileSource::try_pushdown_pruning_filters`]) is used only to prune:
     /// a `FilterExec` above the scan applies it. Thus it is not exact.
+    ///
+    /// Optional conjuncts (see `split_optional`) are not exact either: the
+    /// scan does not evaluate them after the decode, and it drops them when
+    /// the `RowFilter` cannot evaluate them.
     fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
         if self.pruning_only_predicate {
             return None;
@@ -727,6 +731,7 @@ impl FileSource for ParquetSource {
         let pushable_schema = self.table_schema.schema_without_virtual_columns();
         let exact = split_conjunction(predicate)
             .into_iter()
+            .filter(|expr| !is_optional_filter(expr))
             .filter(|expr| can_expr_be_pushed_down_with_schemas(expr, pushable_schema))
             .cloned()
             .collect::<Vec<_>>();
@@ -2253,6 +2258,52 @@ mod tests {
         );
     }
 
+    /// An optional conjunct is not exact: the scan does not evaluate it
+    /// after the decode, and drops it when the `RowFilter` cannot evaluate
+    /// it. Thus it must not give equivalences.
+    #[test]
+    fn exact_filter_excludes_optional_conjuncts() {
+        use arrow::datatypes::{DataType, Field, Schema};
+        use datafusion_common::config::ConfigOptions;
+        use datafusion_expr::{col, lit as logical_lit};
+        use datafusion_physical_expr::expressions::OptionalFilterPhysicalExpr;
+        use datafusion_physical_expr::planner::logical2physical;
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("a", DataType::Int64, false),
+            Field::new("b", DataType::Int64, false),
+        ]));
+        let required = logical2physical(&col("a").eq(logical_lit(1i64)), &schema);
+        let optional: Arc<dyn PhysicalExpr> = Arc::new(OptionalFilterPhysicalExpr::new(
+            logical2physical(&col("b").eq(logical_lit(2i64)), &schema),
+        ));
+        for pushdown_filters in [false, true] {
+            let mut config = ConfigOptions::default();
+            config.execution.parquet.pushdown_filters = pushdown_filters;
+            let prop = ParquetSource::new(Arc::clone(&schema))
+                .try_pushdown_filters(
+                    vec![Arc::clone(&required), Arc::clone(&optional)],
+                    &config,
+                )
+                .unwrap();
+            let source = prop.updated_node.unwrap();
+            let source = source.downcast_ref::<ParquetSource>().unwrap();
+            assert_eq!(
+                source.exact_filter().unwrap().to_string(),
+                "a@0 = 1",
+                "pushdown_filters = {pushdown_filters}"
+            );
+        }
+
+        // A predicate with only optional conjuncts is not exact.
+        let prop = ParquetSource::new(Arc::clone(&schema))
+            .try_pushdown_filters(vec![Arc::clone(&optional)], &ConfigOptions::default())
+            .unwrap();
+        let source = prop.updated_node.unwrap();
+        let source = source.downcast_ref::<ParquetSource>().unwrap();
+        assert!(source.exact_filter().is_none());
+    }
+
     #[test]
     fn test_try_pushdown_filters_rejects_virtual_column_refs() {
         // Virtual columns are produced by the reader and cannot be referenced

From bd867a01052b3bbc04fdd1b68ddd7e39a65465a3 Mon Sep 17 00:00:00 2001
From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Date: Mon, 28 Sep 2026 01:24:16 -0500
Subject: [PATCH 42/42] docs(parquet): optional conjuncts of the placement are
 not exact

With adaptive filter placement, an optional conjunct can be in the
post-scan filter (its gate asks for each batch) or skipped. Say so in the
documentation of `ParquetSource::exact_filter`, which already excludes
optional conjuncts.

PR: #25727

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 datafusion/datasource-parquet/src/source.rs | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/datafusion/datasource-parquet/src/source.rs b/datafusion/datasource-parquet/src/source.rs
index 85521c3f99763..cdec14a84f999 100644
--- a/datafusion/datasource-parquet/src/source.rs
+++ b/datafusion/datasource-parquet/src/source.rs
@@ -768,9 +768,11 @@ impl FileSource for ParquetSource {
     /// [`FileSource::try_pushdown_pruning_filters`]) is used only to prune:
     /// a `FilterExec` above the scan applies it. Thus it is not exact.
     ///
-    /// Optional conjuncts (see `split_optional`) are not exact either: the
-    /// scan does not evaluate them after the decode, and it drops them when
-    /// the `RowFilter` cannot evaluate them.
+    /// Optional conjuncts (see `split_optional`) are not exact either: in
+    /// the `adaptive` mode their gate can skip them (also in the post-scan
+    /// filter of the adaptive filter placement, and the placement can skip
+    /// them), in the `pruning_only` mode the scan does not evaluate them, and
+    /// the scan drops them when the `RowFilter` cannot evaluate them.
     fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
         if self.pruning_only_predicate {
             return None;