Repository navigation
Commit 8f4c51f
authored
fix: keep filters with correlated subqueries above extension nodes (#25294)
## Which issue does this PR close?
- Closes #15046.
## Rationale for this change
An extension node can name columns whose predicates must not be pushed
past it, via
`UserDefinedLogicalNode::prevent_predicate_push_down_columns`. A
predicate whose only dependency on those columns runs through a
correlated subquery is pushed down anyway, and the resulting plan does
not just perform worse — it fails to execute.
Using the `NoopPlan` node already present in the optimizer's own tests,
which refuses push-down of column `c`:
```
Filter: EXISTS (SELECT sq.a FROM sq WHERE outer_ref(test.c) = sq.a)
NoopPlan
TableScan: test
```
optimizing that plan today gives
```
Optimizer rule 'push_down_filter' failed
Invalid (non-executable) plan after Optimizer rule: push_down_filter
In/Exist/SetComparison subquery can only be used in Projection, Filter, TableScan,
Window functions, Aggregate and Join plan nodes, but was used in [NoopPlan]
```
The predicate is moved below the node that asked to keep it, lands
somewhere subqueries are not permitted, and the invariant check turns
that into a query failure.
The cause is the one #15046 describes. The check is
```rust
!expr.column_refs().iter().any(|c| prevent_cols.contains(&c.name))
```
and `column_refs` collects only `Expr::Column`. A subquery records the
outer columns it correlates on in `Subquery::outer_ref_columns`, which
`Expr`'s traversal does not descend into, so `EXISTS (... WHERE outer.c
= ...)` reports no column references at all and looks unconditionally
safe to push.
## What changes are included in this PR?
A `references_any_column` helper in `push_down_filter.rs` that asks
`column_refs` first and then, for `Expr::Exists`, `Expr::InSubquery` and
`Expr::ScalarSubquery`, also looks at the `outer_ref_columns` those
carry. The extension-node branch uses it instead of `column_refs` alone.
Nothing else changes.
**I deliberately did not change `column_refs` itself**, which is what
the issue title asks for. It has many callers whose behaviour would
shift — `optimize_projections`, CSE, window and join handling all use
it, and several of them want "columns produced by this plan's own
schema", not outer references belonging to a parent. Widening it looks
like a separate change with its own risk, so this PR fixes the consumer
that has a demonstrable failure. Happy to look at the broader change
instead if you would rather that were the fix.
## What is the testing strategy for this PR?
New unit test
`push_down_filter::tests::user_defined_plan_outer_referenced_column`,
next to the existing `user_defined_plan` test and reusing its
`NoopPlan`. It asserts the `EXISTS` predicate stays above the node.
The test discriminates: with the outer-reference collection
short-circuited out, it fails with the invalid-plan error quoted above;
with the fix it passes. `datafusion-optimizer` (808) and
`datafusion-expr` (262) pass, as does `sqllogictest`.
## Are there any user-facing changes?
No public API changes. Plans change only where an extension node
declares `prevent_predicate_push_down_columns` and a predicate
correlates a subquery on one of those columns — previously an invalid
plan, now a correct one. No in-tree node declares those columns outside
tests, so no bundled plan snapshots move.
Two adjacent things found while working on this, neither touched here:
- `find_out_reference_exprs` does not see these outer references either,
for the same reason, so it cannot be used as the collection mechanism.
That is the same traversal gap as #16147, which @AdamGS has taken.
- The equivalent guard for correlated subqueries in other push-down
branches was not audited; this PR only covers the extension-node branch
that #15046 reports.1 parent c0e872f commit 8f4c51f
1 file changed
Lines changed: 132 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
30 | 32 | | |
31 | 33 | | |
32 | 34 | | |
| |||
783 | 785 | | |
784 | 786 | | |
785 | 787 | | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
786 | 827 | | |
787 | 828 | | |
788 | 829 | | |
| |||
1301 | 1342 | | |
1302 | 1343 | | |
1303 | 1344 | | |
1304 | | - | |
1305 | | - | |
1306 | | - | |
1307 | | - | |
1308 | | - | |
1309 | | - | |
| 1345 | + | |
1310 | 1346 | | |
1311 | 1347 | | |
1312 | 1348 | | |
| |||
1523 | 1559 | | |
1524 | 1560 | | |
1525 | 1561 | | |
1526 | | - | |
1527 | | - | |
| 1562 | + | |
| 1563 | + | |
1528 | 1564 | | |
1529 | 1565 | | |
1530 | 1566 | | |
1531 | | - | |
1532 | | - | |
1533 | | - | |
| 1567 | + | |
| 1568 | + | |
| 1569 | + | |
1534 | 1570 | | |
1535 | 1571 | | |
1536 | 1572 | | |
| |||
2185 | 2221 | | |
2186 | 2222 | | |
2187 | 2223 | | |
| 2224 | + | |
| 2225 | + | |
| 2226 | + | |
| 2227 | + | |
| 2228 | + | |
| 2229 | + | |
| 2230 | + | |
| 2231 | + | |
| 2232 | + | |
| 2233 | + | |
| 2234 | + | |
| 2235 | + | |
| 2236 | + | |
| 2237 | + | |
| 2238 | + | |
| 2239 | + | |
| 2240 | + | |
| 2241 | + | |
| 2242 | + | |
| 2243 | + | |
| 2244 | + | |
| 2245 | + | |
| 2246 | + | |
| 2247 | + | |
| 2248 | + | |
| 2249 | + | |
| 2250 | + | |
| 2251 | + | |
| 2252 | + | |
| 2253 | + | |
| 2254 | + | |
| 2255 | + | |
| 2256 | + | |
| 2257 | + | |
| 2258 | + | |
| 2259 | + | |
| 2260 | + | |
| 2261 | + | |
| 2262 | + | |
| 2263 | + | |
| 2264 | + | |
| 2265 | + | |
| 2266 | + | |
| 2267 | + | |
| 2268 | + | |
| 2269 | + | |
| 2270 | + | |
| 2271 | + | |
| 2272 | + | |
| 2273 | + | |
| 2274 | + | |
| 2275 | + | |
| 2276 | + | |
| 2277 | + | |
| 2278 | + | |
| 2279 | + | |
| 2280 | + | |
| 2281 | + | |
| 2282 | + | |
| 2283 | + | |
| 2284 | + | |
| 2285 | + | |
| 2286 | + | |
| 2287 | + | |
| 2288 | + | |
| 2289 | + | |
| 2290 | + | |
| 2291 | + | |
| 2292 | + | |
| 2293 | + | |
| 2294 | + | |
| 2295 | + | |
| 2296 | + | |
| 2297 | + | |
| 2298 | + | |
| 2299 | + | |
| 2300 | + | |
| 2301 | + | |
| 2302 | + | |
| 2303 | + | |
| 2304 | + | |
| 2305 | + | |
| 2306 | + | |
| 2307 | + | |
2188 | 2308 | | |
2189 | 2309 | | |
2190 | 2310 | | |
| |||
0 commit comments