@@ -695,3 +695,92 @@ impl FileSource for MySource {
695695 // ...
696696}
697697```
698+
699+ ### Physical filter pushdown passes ` FilterConjunct ` s
700+
701+ Physical filter pushdown now carries each filter as a ` FilterConjunct ` (from
702+ ` datafusion_physical_expr::filter ` ). A ` FilterConjunct ` is a boolean
703+ ` PhysicalExpr ` together with its properties. At this time, the only property
704+ is ` is_optional ` : an optional conjunct (for example, a dynamic filter from a
705+ hash join, TopK or aggregate) can be skipped without an effect on the query
706+ result. The properties now move through pushdown together with each filter.
707+
708+ These public APIs changed:
709+
710+ - ` ExecutionPlan::gather_filters_for_pushdown ` receives
711+ ` parent_filters: Vec<FilterConjunct> ` instead of
712+ ` Vec<Arc<dyn PhysicalExpr>> ` .
713+ - ` ChildFilterDescription::from_child ` , ` from_child_with_column_mapping ` ,
714+ ` from_child_with_allowed_indices ` , ` all_unsupported ` ,
715+ ` FilterDescription::from_children ` and ` FilterDescription::all_unsupported `
716+ take ` FilterConjunct ` s.
717+ - ` PushedDownPredicate::predicate ` is a ` FilterConjunct ` .
718+ ` PushedDownPredicate::supported ` , ` PushedDownPredicate::unsupported ` and
719+ ` PushedDown::wrap_expression ` take a ` FilterConjunct ` , and
720+ ` PushedDownPredicate::into_inner ` returns one.
721+ - ` ChildFilterPushdownResult::filter ` is a ` FilterConjunct ` .
722+ - ` DataSource::try_pushdown_filters ` and ` FileSource::try_pushdown_filters `
723+ are deprecated. Implement and call ` try_pushdown_filter ` instead. It takes a
724+ ` PhysicalFilter ` (an ordered list of ` FilterConjunct ` s).
725+ - ` FileSource ` has a new ` physical_filter ` method. ` ParquetSource::physical_filter `
726+ now comes from this trait method and returns an owned ` PhysicalFilter ` .
727+
728+ ** Migration guide:**
729+
730+ In ` gather_filters_for_pushdown ` , change the type of ` parent_filters ` . Use
731+ ` FilterConjunct::expr ` to read the expression. When a node rewrites a parent
732+ filter (for example, to remap its columns), use ` FilterConjunct::with_expr ` to
733+ keep the properties of the filter:
734+
735+ ``` rust,ignore
736+ fn gather_filters_for_pushdown(
737+ &self,
738+ _phase: FilterPushdownPhase,
739+ parent_filters: Vec<FilterConjunct>,
740+ _config: &ConfigOptions,
741+ ) -> Result<FilterDescription> {
742+ let parent_filters = parent_filters
743+ .into_iter()
744+ .map(|filter| {
745+ let rewritten = rewrite(filter.expr())?;
746+ // Before: PushedDownPredicate::supported(rewritten)
747+ Ok(PushedDownPredicate::supported(filter.with_expr(rewritten)))
748+ })
749+ .collect::<Result<Vec<_>>>()?;
750+ // ...
751+ }
752+ ```
753+
754+ Code that makes a filter from an expression can use
755+ ` FilterConjunct::required(expr) ` or ` FilterConjunct::from(expr) ` .
756+
757+ ** Custom ` DataSource ` and ` FileSource ` implementations: check them manually.**
758+ The compiler does not warn when an implementation overrides a deprecated
759+ trait method. It warns only when code calls the method. Thus, a source that
760+ overrides ` try_pushdown_filters ` compiles without a warning. Such a source
761+ continues to work: the default ` try_pushdown_filter ` calls
762+ ` try_pushdown_filters ` with the expressions. But the source does not get the
763+ conjunct properties, and it applies optional conjuncts as required conjuncts.
764+ The deprecated method will be removed in a future release, so search your code
765+ for ` fn try_pushdown_filters ` and migrate each implementation now.
766+
767+ To migrate, rename ` try_pushdown_filters ` to ` try_pushdown_filter ` and read
768+ the expressions from the ` PhysicalFilter ` :
769+
770+ ``` rust,ignore
771+ fn try_pushdown_filter(
772+ &self,
773+ filter: PhysicalFilter,
774+ config: &ConfigOptions,
775+ ) -> Result<FilterPushdownPropagation<Arc<dyn FileSource>>> {
776+ let filters: Vec<Arc<dyn PhysicalExpr>> = filter
777+ .into_conjuncts()
778+ .into_iter()
779+ .map(FilterConjunct::into_expr)
780+ .collect();
781+ // ... the previous body of `try_pushdown_filters` ...
782+ }
783+ ```
784+
785+ A source that keeps the conjuncts (and thus their properties) should also
786+ implement ` FileSource::physical_filter ` .
0 commit comments