Skip to content

Commit 982fca6

Browse files
authored
fix: keep aggregate limits when combining or rebuilding AggregateExec (#25983)
## Which issue does this PR close? - No separate issue. Both problems are regressions from #25696. ## Rationale for this change 1. Some `GROUP BY ... LIMIT` queries fail to plan. With one target partition: ```sql SET datafusion.execution.target_partitions = 1; CREATE TABLE t AS SELECT * FROM (VALUES (1, 10), (2, 20), (3, 30), (4, 40), (5, 50), (6, 60)) AS v(a, b); SELECT b, a FROM t GROUP BY b, a LIMIT 2; SELECT a % 3 FROM t GROUP BY a % 3 LIMIT 2; ``` both fail with: ``` DataFusion error: CombinePartialFinalAggregate caused by Internal error: The AggregateKind should stay either (a) General (b) DistinctLimit introduced with the previous optimizer pass `LimitedDistinctAggregation`, it's impossible to have other variant. ``` When the final aggregate's grouping differs from the partial's (expression keys, or columns not at their output positions), `LimitedDistinctAggregation` limits only the final aggregate. `CombinePartialFinalAggregate` treats that pair as impossible and returns an error. 2. On Parquet tables the TopK aggregate limit is dropped. `EXPLAIN SELECT DISTINCT k FROM pt ORDER BY k LIMIT 10` keeps `lim=[10]` on the aggregates for a memory table but not for a Parquet table, so the query falls back to a full hash aggregation. MIN/MAX TopK aggregates (`ORDER BY max(x) DESC NULLS LAST LIMIT n`) lose their limit the same way. Post-optimization filter pushdown rebuilds the Parquet scan when it accepts a dynamic filter, and rebuilding the aggregates above it re-applies only the DISTINCT limit, not the TopK ones. Before #25696 both places carried the limit over with `with_limit_options(...limit_options())`. ## What changes are included in this PR? - `CombinePartialFinalAggregate`: if the final aggregate has a DISTINCT limit, re-apply it to the combined aggregate with `try_optimize_distinct_soft_limit`, which checks eligibility again. Otherwise keep the combined aggregate as built. - `AggregateExec::replace_children` (`ChildrenPropertiesMode::Recompute`): also re-apply `TopKMinMax` and `TopKDistinct` through `try_optimize_topk`, using the stored limit, direction and NULL placement. ## What is the testing strategy for this PR? New sqllogictest cases. Each one fails without the fix and passes with it: - `aggregate.slt`: both queries above at `target_partitions = 1`. An `EXPLAIN` checks the combined `mode=Single ... lim=[2]` aggregate, and a row-count check keeps the results deterministic. - `aggregates_topk.slt`: a Parquet-backed table. `EXPLAIN` checks that `lim=[10]` stays on both aggregates for `SELECT DISTINCT k ... ORDER BY k LIMIT 10`, and that `lim=[2]` stays for a MAX TopK query whose scan gets a semi-join dynamic filter. Result checks are included. ## Are there any user-facing changes? No API changes. The listed queries plan again, and TopK aggregation applies again on Parquet scans that receive dynamic filters.
1 parent e5fdc9d commit 982fca6

4 files changed

Lines changed: 173 additions & 11 deletions

File tree

‎datafusion/physical-optimizer/src/combine_partial_final_agg.rs‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,6 @@
2121
use std::sync::Arc;
2222

2323
use datafusion_common::error::Result;
24-
use datafusion_common::internal_err;
2524
use datafusion_physical_plan::ExecutionPlan;
2625
use datafusion_physical_plan::aggregates::{
2726
AggregateExec, AggregateKind, AggregateMode, PhysicalGroupBy,
@@ -103,7 +102,7 @@ impl PhysicalOptimizerRule for CombinePartialFinalAggregate {
103102
};
104103

105104
// Re-apply distinct limit optimization.
106-
//
105+
//
107106
// The optimizer related to aggregates are (in order):
108107
// - 1. Initial planning: always final/partial two stage
109108
// - 2. `LimitedDistinctAggregation`: push limit into `AggregateExec`,
@@ -126,11 +125,9 @@ impl PhysicalOptimizerRule for CombinePartialFinalAggregate {
126125
group_by: Arc::clone(group_by),
127126
limit: *limit,
128127
},
129-
_ => {
130-
return internal_err!(
131-
"The AggregateKind should stay either (a) General (b) DistinctLimit introduced with the previous optimizer pass `LimitedDistinctAggregation`, it's impossible to have other variant"
132-
);
133-
}
128+
// Step 2 may limit only the final aggregate, e.g. when its
129+
// grouping differs from the partial's. Keep both stages.
130+
_ => return Ok(Transformed::no(plan)),
134131
};
135132
combined_agg.kind = combined_aggr_kind;
136133
Some(Arc::new(combined_agg))

‎datafusion/physical-plan/src/aggregates/mod.rs‎

Lines changed: 31 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2520,11 +2520,38 @@ impl ExecutionPlan for AggregateExec {
25202520
Arc::clone(&self.input_schema),
25212521
Arc::clone(&self.schema),
25222522
)?;
2523-
// Reapply a DISTINCT limit only if the new input remains eligible.
2524-
if let AggregateKind::DistinctLimit { limit, .. } = &self.kind
2525-
&& let Some(optimized) =
2523+
// Reapply a limit only if the new input remains eligible.
2524+
let optimized = match &self.kind {
2525+
AggregateKind::General { .. } => None,
2526+
AggregateKind::DistinctLimit { limit, .. } => {
25262527
me.clone().try_optimize_distinct_soft_limit(*limit)
2527-
{
2528+
}
2529+
AggregateKind::TopKMinMax {
2530+
aggr_expr,
2531+
limit,
2532+
descending,
2533+
nulls_first,
2534+
..
2535+
} => me.clone().try_optimize_topk(
2536+
*limit,
2537+
aggr_expr.name(),
2538+
SortOptions::new(*descending, *nulls_first),
2539+
),
2540+
// TopK DISTINCT orders by its single grouping key and
2541+
// ignores `nulls_first`.
2542+
AggregateKind::TopKDistinct {
2543+
group_by,
2544+
limit,
2545+
descending,
2546+
} => group_by.expr().first().and_then(|(_, alias)| {
2547+
me.clone().try_optimize_topk(
2548+
*limit,
2549+
alias,
2550+
SortOptions::new(*descending, false),
2551+
)
2552+
}),
2553+
};
2554+
if let Some(optimized) = optimized {
25282555
me = optimized.data;
25292556
}
25302557
me.dynamic_filter.clone_from(&self.dynamic_filter);

‎datafusion/sqllogictest/test_files/aggregate.slt‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7619,6 +7619,62 @@ staging 1
76197619
statement ok
76207620
DROP TABLE distinct_limit_window_table;
76217621

7622+
# `LimitedDistinctAggregation` limits only the final aggregate when its grouping
7623+
# differs from the partial's (columns not at their output positions, or
7624+
# expression keys). `CombinePartialFinalAggregate` must leave such a pair
7625+
# uncombined instead of failing to plan.
7626+
statement ok
7627+
set datafusion.execution.target_partitions = 1;
7628+
7629+
statement ok
7630+
CREATE TABLE distinct_limit_combine AS
7631+
SELECT * FROM (VALUES (1, 10), (2, 20), (3, 30), (4, 40), (5, 50), (6, 60)) AS v(a, b);
7632+
7633+
query TT
7634+
EXPLAIN SELECT b, a FROM distinct_limit_combine GROUP BY b, a LIMIT 2;
7635+
----
7636+
logical_plan
7637+
01)Limit: skip=0, fetch=2
7638+
02)--Aggregate: groupBy=[[distinct_limit_combine.b, distinct_limit_combine.a]], aggr=[[]]
7639+
03)----TableScan: distinct_limit_combine projection=[a, b]
7640+
physical_plan
7641+
01)GlobalLimitExec: skip=0, fetch=2
7642+
02)--AggregateExec: mode=Final, gby=[b@0 as b, a@1 as a], aggr=[], lim=[2]
7643+
03)----AggregateExec: mode=Partial, gby=[b@1 as b, a@0 as a], aggr=[]
7644+
04)------DataSourceExec: partitions=1, partition_sizes=[1]
7645+
7646+
# LIMIT without ORDER BY may return any two groups
7647+
query II
7648+
SELECT count(*), count(*) FILTER (WHERE b = a * 10)
7649+
FROM (SELECT b, a FROM distinct_limit_combine GROUP BY b, a LIMIT 2);
7650+
----
7651+
2 2
7652+
7653+
query TT
7654+
EXPLAIN SELECT a % 3 FROM distinct_limit_combine GROUP BY a % 3 LIMIT 2;
7655+
----
7656+
logical_plan
7657+
01)Limit: skip=0, fetch=2
7658+
02)--Aggregate: groupBy=[[distinct_limit_combine.a % Int64(3)]], aggr=[[]]
7659+
03)----TableScan: distinct_limit_combine projection=[a]
7660+
physical_plan
7661+
01)GlobalLimitExec: skip=0, fetch=2
7662+
02)--AggregateExec: mode=Final, gby=[distinct_limit_combine.a % Int64(3)@0 as distinct_limit_combine.a % Int64(3)], aggr=[], lim=[2]
7663+
03)----AggregateExec: mode=Partial, gby=[a@0 % 3 as distinct_limit_combine.a % Int64(3)], aggr=[]
7664+
04)------DataSourceExec: partitions=1, partition_sizes=[1]
7665+
7666+
query II
7667+
SELECT count(*), count(DISTINCT m)
7668+
FROM (SELECT a % 3 AS m FROM distinct_limit_combine GROUP BY a % 3 LIMIT 2);
7669+
----
7670+
2 2
7671+
7672+
statement ok
7673+
DROP TABLE distinct_limit_combine;
7674+
7675+
statement ok
7676+
set datafusion.execution.target_partitions = 4;
7677+
76227678
#
76237679
# regr_*() tests
76247680
#

‎datafusion/sqllogictest/test_files/aggregates_topk.slt‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -789,3 +789,85 @@ drop table topk_two_groups;
789789

790790
statement ok
791791
drop table t0;
792+
793+
# Rebuilding an aggregate must keep its TopK limit. Post-optimization filter
794+
# pushdown rebuilds a Parquet scan that accepts a dynamic filter, which
795+
# recomputes the aggregates above it.
796+
statement ok
797+
COPY (SELECT (v % 100)::int AS k, v AS x FROM generate_series(1, 1000) AS s(v))
798+
TO 'test_files/scratch/aggregates_topk/topk.parquet' STORED AS PARQUET;
799+
800+
statement ok
801+
CREATE EXTERNAL TABLE topk_parquet STORED AS PARQUET
802+
LOCATION 'test_files/scratch/aggregates_topk/topk.parquet';
803+
804+
# The TopK dynamic filter on `k` is pushed through the aggregates into the scan
805+
query TT
806+
EXPLAIN SELECT DISTINCT k FROM topk_parquet ORDER BY k LIMIT 10;
807+
----
808+
logical_plan
809+
01)Sort: topk_parquet.k ASC NULLS LAST, fetch=10
810+
02)--Aggregate: groupBy=[[topk_parquet.k]], aggr=[[]]
811+
03)----TableScan: topk_parquet projection=[k]
812+
physical_plan
813+
01)SortPreservingMergeExec: [k@0 ASC NULLS LAST], fetch=10
814+
02)--SortExec: TopK(fetch=10), expr=[k@0 ASC NULLS LAST], preserve_partitioning=[true]
815+
03)----AggregateExec: mode=FinalPartitioned, gby=[k@0 as k], aggr=[], lim=[10]
816+
04)------RepartitionExec: partitioning=Hash([k@0], 4), input_partitions=1
817+
05)--------AggregateExec: mode=Partial, gby=[k@0 as k], aggr=[], lim=[10]
818+
06)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/aggregates_topk/topk.parquet]]}, projection=[k], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
819+
820+
query I
821+
SELECT DISTINCT k FROM topk_parquet ORDER BY k LIMIT 10;
822+
----
823+
0
824+
1
825+
2
826+
3
827+
4
828+
5
829+
6
830+
7
831+
8
832+
9
833+
834+
# The semi join's dynamic filter is pushed into the scan below the TopK MAX aggregates
835+
statement ok
836+
CREATE TABLE topk_parquet_keys AS VALUES (1), (2), (3);
837+
838+
query TT
839+
EXPLAIN SELECT k, max(x) FROM topk_parquet
840+
WHERE k IN (SELECT column1 FROM topk_parquet_keys)
841+
GROUP BY k ORDER BY max(x) DESC NULLS LAST LIMIT 2;
842+
----
843+
logical_plan
844+
01)Sort: max(topk_parquet.x) DESC NULLS LAST, fetch=2
845+
02)--Aggregate: groupBy=[[topk_parquet.k]], aggr=[[max(topk_parquet.x)]]
846+
03)----LeftSemi Join: CAST(topk_parquet.k AS Int64) = __correlated_sq_1.column1
847+
04)------TableScan: topk_parquet projection=[k, x]
848+
05)------SubqueryAlias: __correlated_sq_1
849+
06)--------TableScan: topk_parquet_keys projection=[column1]
850+
physical_plan
851+
01)SortPreservingMergeExec: [max(topk_parquet.x)@1 DESC NULLS LAST], fetch=2
852+
02)--SortExec: TopK(fetch=2), expr=[max(topk_parquet.x)@1 DESC NULLS LAST], preserve_partitioning=[true]
853+
03)----AggregateExec: mode=FinalPartitioned, gby=[k@0 as k], aggr=[max(topk_parquet.x)], lim=[2]
854+
04)------RepartitionExec: partitioning=Hash([k@0], 4), input_partitions=4
855+
05)--------AggregateExec: mode=Partial, gby=[k@0 as k], aggr=[max(topk_parquet.x)], lim=[2]
856+
06)----------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, maintains_sort_order=true
857+
07)------------HashJoinExec: mode=CollectLeft, join_type=RightSemi, on=[(column1@0, CAST(topk_parquet.k AS Int64)@2)], projection=[k@0, x@1]
858+
08)--------------DataSourceExec: partitions=1, partition_sizes=[1]
859+
09)--------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/aggregates_topk/topk.parquet]]}, projection=[k, x, CAST(k@0 AS Int64) as CAST(topk_parquet.k AS Int64)], output_ordering=[x@1 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
860+
861+
query II
862+
SELECT k, max(x) FROM topk_parquet
863+
WHERE k IN (SELECT column1 FROM topk_parquet_keys)
864+
GROUP BY k ORDER BY max(x) DESC NULLS LAST LIMIT 2;
865+
----
866+
3 903
867+
2 902
868+
869+
statement ok
870+
drop table topk_parquet_keys;
871+
872+
statement ok
873+
drop table topk_parquet;

0 commit comments

Comments
 (0)