feat: AQE DPP for native Parquet scans with broadcast reuse

[mbutrovich]

Which issue does this PR close?

Partially addresses #3510. Closes #4045 (V1 Parquet AQE DPP). Related PRs: #4011 (non-AQE DPP), #4053 (scalar subquery pushdown + CometReuseSubquery), #4037 (non-AQE DPP edge case tests), #4033 (AQE DPP for Iceberg, draft).

Rationale for this change

Under AQE (the default), Spark creates SubqueryAdaptiveBroadcastExec (SAB) for DPP. Spark's PlanAdaptiveDynamicPruningFilters converts these by finding BroadcastHashJoinExec in the plan. After Comet replaces it with CometBroadcastHashJoinExec, Spark's rule can't find a match and replaces DPP with Literal.TrueLiteral, disabling partition pruning. Previously, the isAqeDynamicPruningFilter rejection caused the scan to fall back to Spark entirely, losing native acceleration for all DPP queries under AQE.

What changes are included in this PR?

Two-phase SAB conversion

Spark's PlanAdaptiveDynamicPruningFilters runs before custom queryStageOptimizerRules and converts SABs to TrueLiteral. We work around this in two phases:

1. CometExecRule (queryStagePreparationRules, before Spark's rule): Wraps SABs in CometSubqueryAdaptiveBroadcastExec so Spark's pattern match doesn't recognize them. Only wraps in CometNativeScanExec nodes (non-Comet scans keep the original SAB for Spark to handle).
2. CometPlanAdaptiveDynamicPruningFilters (queryStageOptimizerRule, after Spark's rule): Converts following Spark's PlanAdaptiveDynamicPruningFilters decision tree:
   • exchangeReuseEnabled + matching broadcast join: CometSubqueryBroadcastExec wired to BroadcastQueryStageExec for broadcast reuse
   • No reusable broadcast + onlyInBroadcast=true: Literal.TrueLiteral (DPP disabled)
   • No reusable broadcast + onlyInBroadcast=false: aggregate SubqueryExec (DPP via separate execution, matching Spark's PlanAdaptiveDynamicPruningFilters lines 68-79)

Cross-stage broadcast search

Spark's PlanAdaptiveDynamicPruningFilters is constructed with rootPlan = this (the current ASPE), giving each ASPE its own rule instance. Custom queryStageOptimizerRules via injectQueryStageOptimizerRule are shared across all ASPEs without a per-ASPE rootPlan.

We approximate this with two searches:

1. stagePlan (the plan arg to apply()): same-stage joins and scalar subqueries where scan and join are under one exchange
2. context.qe.executedPlan (the main query's ASPE): cross-stage joins where a shuffle separates scan from broadcast join

When the broadcast is not yet materialized (cross-stage case), we follow Spark's PlanAdaptiveDynamicPruningFilters pattern (lines 44-64): construct a new broadcast exchange, wrap in a new ASPE, and let AQE's stageCache canonicalization ensure the broadcast runs once.

Subquery deduplication via shared cache

Our rule runs after Spark's ReuseAdaptiveSubquery (which can't see our subqueries because they don't exist yet). We register DPP subqueries directly in AdaptiveExecutionContext.subqueryCache, matching ReuseAdaptiveSubquery's behavior for cross-plan reuse (e.g., main query and scalar subquery with identical DPP).

Dual-filter resolution

CometNativeScanExec.partitionFilters and CometScanExec.partitionFilters contain separate InSubqueryExec instances. CometExecRule only wraps the outer filters (the inner CometScanExec is @transient). CometPlanAdaptiveDynamicPruningFilters converts both: CometSubqueryAdaptiveBroadcastExec (wrapped, outer) and SubqueryAdaptiveBroadcastExec (unwrapped, inner).

Spark 3.4 fallback

injectQueryStageOptimizerRule is unavailable on 3.4. SAB wrapping is gated on isSpark35Plus. On 3.4, AQE DPP scans fall back to Spark so Spark's rule handles them natively.

Broadcast fallback cases

• Spark BHJ (Comet BHJ disabled): finds BroadcastHashJoinExec, creates SubqueryBroadcastExec via shim.
• SMJ (no broadcast): falls back to Literal.TrueLiteral or aggregate SubqueryExec depending on onlyInBroadcast.
• ReusedExchangeExec: BroadcastQueryStageExec.plan may be ReusedExchangeExec when AQE reuses exchanges across plans. The rule unwraps it to verify the underlying exchange type.

Other changes

• CometBroadcastExchangeExec: handles non-Comet children (e.g., LocalTableScan after AQE re-optimization of empty broadcasts) by wrapping in CometSparkToColumnarExec.
• CometNativeScanExec.doCanonicalize: strips DPP filters from originalPlan to prevent stale SABs from blocking exchange reuse.
• CometShuffleExchangeExec.doCanonicalize: excludes originalPlan from canonical form (matches CometBroadcastExchangeExec).
• CometScanUtils.filterUnusedDynamicPruningExpressions: strips unconverted SABs in addition to TrueLiteral, matching Spark's FileSourceScanExec.filterUnusedDynamicPruningExpressions.
• ShimPrepareExecutedPlan: new shim for QueryExecution.prepareExecutedPlan (3-arg on 3.x/4.0, 2-arg on 4.1+).
• Existing DPP tests (CometDppFallbackRepro3949Suite, CometShuffleFallbackStickinessSuite) updated to disable native scan to preserve the stageContainsDPPScan stickiness code path.
• Removed two IgnoreComet(#4045) tags from Spark's DynamicPartitionPruningSuite diffs (SPARK-32509, SPARK-34637). Tests ported to CometExecSuite.

How are these changes tested?

16 new AQE DPP tests in CometExecSuite covering BHJ/SMJ/empty broadcast/dual filters/exchange reuse/non-atomic types/cross-stage search/scalar subquery deduplication/SPARK-32509/SPARK-34637/SPARK-39447. All tests have version-specific assertions (3.5+ native path vs 3.4 fallback). Existing non-AQE DPP tests renamed to consistent "[non-AQE|AQE] DPP: <scenario>" format.