Skip to content

fix: handle nested RANGE CURRENT ROW peers - #26067

Open
sunchao wants to merge 1 commit into
apache:mainfrom
sunchao:fix/nested-range-peer-boundaries
Open

sunchao wants to merge 1 commit into
apache:mainfrom
sunchao:fix/nested-range-peer-boundaries

Conversation

@sunchao

@sunchao sunchao commented Oct 6, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Related to #24937. This fixes the RANGE-window case; general MIN/MAX scalar comparisons remain separate.

Rationale for this change

A RANGE window ordered by a list containing lists or structs can plan successfully and then fail with Uncomparable values. For example:

SELECT count(*) OVER (ORDER BY l)
FROM (VALUES ([[1]]), ([[2]])) t(l);

CURRENT ROW boundaries only need to find equal ordering keys in the sorted input.

What changes are included in this PR?

Use Arrow's comparator to find nested CURRENT ROW peer boundaries. Reuse the comparator while the ordering arrays are unchanged and release it when buffered rows are pruned.

SQL type admission, scalar-key handling, and finite-offset calculations are unchanged.

What is the testing strategy for this PR?

Update the two existing SQL regressions from expected errors to successful counts. Physical-operator tests cover nested lists/structs, nulls, tied ordering keys across batches, frame boundaries, and completed-peer emission before EOF in sorted mode. A small state test covers comparator reuse and pruning.

Local validation on this patch:

  • Full extended workspace suite: 12,421 Rust tests passed, eight ignored, and all 527 SQL-logic files completed.
  • All-target/all-feature Clippy with warnings denied passed.
  • Formatting, license/typo checks, generated documentation checks, local link checks, and the security audit passed. API and website documentation builds also passed with warnings treated as errors.
  • Full lint could not complete locally: the package mirror does not offer cargo 0.100.0, pinned by the existing dev/depcheck tool. This blocks the circular-dependency check and that tool's metadata inspection by cargo-machete; cargo-machete otherwise reported no unused dependencies. This patch changes no dependency manifests or lockfiles.
  • The physical regression and both SQL regressions fail on the unpatched target base with the original comparison error.
  • All six existing aggregate-window benchmark smoke cases pass on both base and candidate. No throughput measurements are claimed.

Are there any user-facing changes?

Already-admitted SQL RANGE windows over nested list keys return results instead of an internal comparison error. The same boundary fix applies to windows constructed through the physical API. No new public API or SQL-admitted types are introduced.

@github-actions github-actions Bot added logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion v55.1.0 (current)
       Built [  48.035s] (current)
     Parsing datafusion v55.1.0 (current)
      Parsed [   0.029s] (current)
    Building datafusion v55.1.0 (baseline)
       Built [  48.513s] (baseline)
     Parsing datafusion v55.1.0 (baseline)
      Parsed [   0.029s] (baseline)
    Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.749s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  99.532s] datafusion
    Building datafusion-expr v55.1.0 (current)
       Built [  25.093s] (current)
     Parsing datafusion-expr v55.1.0 (current)
      Parsed [   0.072s] (current)
    Building datafusion-expr v55.1.0 (baseline)
       Built [  25.466s] (baseline)
     Parsing datafusion-expr v55.1.0 (baseline)
      Parsed [   0.075s] (baseline)
    Checking datafusion-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   1.545s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure auto_trait_impl_removed: auto trait no longer implemented ---

Description:
A public type has stopped implementing one or more auto traits. This can break downstream code that depends on the traits being implemented.
        ref: https://doc.rust-lang.org/reference/special-types-and-traits.html#auto-traits
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/auto_trait_impl_removed.ron

Failed in:
  type WindowFrameStateRange is no longer UnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/window_state.rs:348
  type WindowFrameStateRange is no longer RefUnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/window_state.rs:348

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  53.246s] datafusion-expr
    Building datafusion-optimizer v55.1.0 (current)
       Built [  22.176s] (current)
     Parsing datafusion-optimizer v55.1.0 (current)
      Parsed [   0.026s] (current)
    Building datafusion-optimizer v55.1.0 (baseline)
       Built [  22.177s] (baseline)
     Parsing datafusion-optimizer v55.1.0 (baseline)
      Parsed [   0.028s] (baseline)
    Checking datafusion-optimizer v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.182s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  45.315s] datafusion-optimizer
    Building datafusion-sqllogictest v55.1.0 (current)
       Built [  77.337s] (current)
     Parsing datafusion-sqllogictest v55.1.0 (current)
      Parsed [   0.013s] (current)
    Building datafusion-sqllogictest v55.1.0 (baseline)
       Built [  74.427s] (baseline)
     Parsing datafusion-sqllogictest v55.1.0 (baseline)
      Parsed [   0.014s] (baseline)
    Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.102s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 154.193s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Oct 6, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.00000% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.69%. Comparing base (c0e872f) to head (4a19543).

Files with missing lines Patch % Lines
datafusion/expr/src/window_state.rs 87.00% 6 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26067      +/-   ##
==========================================
- Coverage   82.69%   82.69%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      447348   447446      +98     
  Branches   447348   447446      +98     
==========================================
+ Hits       369936   370002      +66     
- Misses      54999    55017      +18     
- Partials    22413    22427      +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants