Skip to content

Commit a6076ce

Browse files
adriangbclaude
andcommitted
docs: upgrade guide for FilterConjunct in physical filter pushdown
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 372e016 commit a6076ce

1 file changed

Lines changed: 89 additions & 0 deletions

File tree

  • docs/source/library-user-guide/upgrading

‎docs/source/library-user-guide/upgrading/56.0.0.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -695,3 +695,92 @@ impl FileSource for MySource {
695695
// ...
696696
}
697697
```
698+
699+
### Physical filter pushdown passes `FilterConjunct`s
700+
701+
Physical filter pushdown now carries each filter as a `FilterConjunct` (from
702+
`datafusion_physical_expr::filter`). A `FilterConjunct` is a boolean
703+
`PhysicalExpr` together with its properties. At this time, the only property
704+
is `is_optional`: an optional conjunct (for example, a dynamic filter from a
705+
hash join, TopK or aggregate) can be skipped without an effect on the query
706+
result. The properties now move through pushdown together with each filter.
707+
708+
These public APIs changed:
709+
710+
- `ExecutionPlan::gather_filters_for_pushdown` receives
711+
`parent_filters: Vec<FilterConjunct>` instead of
712+
`Vec<Arc<dyn PhysicalExpr>>`.
713+
- `ChildFilterDescription::from_child`, `from_child_with_column_mapping`,
714+
`from_child_with_allowed_indices`, `all_unsupported`,
715+
`FilterDescription::from_children` and `FilterDescription::all_unsupported`
716+
take `FilterConjunct`s.
717+
- `PushedDownPredicate::predicate` is a `FilterConjunct`.
718+
`PushedDownPredicate::supported`, `PushedDownPredicate::unsupported` and
719+
`PushedDown::wrap_expression` take a `FilterConjunct`, and
720+
`PushedDownPredicate::into_inner` returns one.
721+
- `ChildFilterPushdownResult::filter` is a `FilterConjunct`.
722+
- `DataSource::try_pushdown_filters` and `FileSource::try_pushdown_filters`
723+
are deprecated. Implement and call `try_pushdown_filter` instead. It takes a
724+
`PhysicalFilter` (an ordered list of `FilterConjunct`s).
725+
- `FileSource` has a new `physical_filter` method. `ParquetSource::physical_filter`
726+
now comes from this trait method and returns an owned `PhysicalFilter`.
727+
728+
**Migration guide:**
729+
730+
In `gather_filters_for_pushdown`, change the type of `parent_filters`. Use
731+
`FilterConjunct::expr` to read the expression. When a node rewrites a parent
732+
filter (for example, to remap its columns), use `FilterConjunct::with_expr` to
733+
keep the properties of the filter:
734+
735+
```rust,ignore
736+
fn gather_filters_for_pushdown(
737+
&self,
738+
_phase: FilterPushdownPhase,
739+
parent_filters: Vec<FilterConjunct>,
740+
_config: &ConfigOptions,
741+
) -> Result<FilterDescription> {
742+
let parent_filters = parent_filters
743+
.into_iter()
744+
.map(|filter| {
745+
let rewritten = rewrite(filter.expr())?;
746+
// Before: PushedDownPredicate::supported(rewritten)
747+
Ok(PushedDownPredicate::supported(filter.with_expr(rewritten)))
748+
})
749+
.collect::<Result<Vec<_>>>()?;
750+
// ...
751+
}
752+
```
753+
754+
Code that makes a filter from an expression can use
755+
`FilterConjunct::required(expr)` or `FilterConjunct::from(expr)`.
756+
757+
**Custom `DataSource` and `FileSource` implementations: check them manually.**
758+
The compiler does not warn when an implementation overrides a deprecated
759+
trait method. It warns only when code calls the method. Thus, a source that
760+
overrides `try_pushdown_filters` compiles without a warning. Such a source
761+
continues to work: the default `try_pushdown_filter` calls
762+
`try_pushdown_filters` with the expressions. But the source does not get the
763+
conjunct properties, and it applies optional conjuncts as required conjuncts.
764+
The deprecated method will be removed in a future release, so search your code
765+
for `fn try_pushdown_filters` and migrate each implementation now.
766+
767+
To migrate, rename `try_pushdown_filters` to `try_pushdown_filter` and read
768+
the expressions from the `PhysicalFilter`:
769+
770+
```rust,ignore
771+
fn try_pushdown_filter(
772+
&self,
773+
filter: PhysicalFilter,
774+
config: &ConfigOptions,
775+
) -> Result<FilterPushdownPropagation<Arc<dyn FileSource>>> {
776+
let filters: Vec<Arc<dyn PhysicalExpr>> = filter
777+
.into_conjuncts()
778+
.into_iter()
779+
.map(FilterConjunct::into_expr)
780+
.collect();
781+
// ... the previous body of `try_pushdown_filters` ...
782+
}
783+
```
784+
785+
A source that keeps the conjuncts (and thus their properties) should also
786+
implement `FileSource::physical_filter`.

0 commit comments

Comments
 (0)