Skip to content

chore(deps): bump substrait from 0.63.0 to 0.65.0 - #24861

Open
alexandrefimov wants to merge 9 commits into
apache:mainfrom
alexandrefimov:substrait-0.65
Open

alexandrefimov wants to merge 9 commits into
apache:mainfrom
alexandrefimov:substrait-0.65

Conversation

@alexandrefimov

@alexandrefimov alexandrefimov commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

None. Supersedes #24255 and #24842.

Rationale for this change

Upgrade substrait to 0.65.0 (spec 0.102.0), adapting to removed protobuf fields.

What changes are included in this PR?

Migrate Fetch, window bounds, grouping references and stored plans. Update literal/type conversions and outer references, set execution behavior, bump pbjson-types, and add protox support.

What is the testing strategy for this PR?

Passed: 73 unit tests, 226 integration tests, 3 doctests, formatting, workspace clippy and the extended suite with 532 SQLLogic files. Six integration tests are ignored.

New cases cover window offsets and unsupported outer-reference forms.

Are there any user-facing changes?

Removed fields are no longer supported. Old protobuf plans may lose limits or grouping keys, changing query results.

Downstream code must adapt to removed consume_enum and changed protobuf types.

@github-actions github-actions Bot added the substrait Changes to the substrait crate label Sep 1, 2026
The 0.65.0 release of the substrait crate moves its embedded spec from
0.87.0 to 0.102.0, so fifteen spec releases of removals arrive at once.
API changes adapted:

- FetchRel: the offset/count oneofs are gone, the expression fields are
  the only form left
- window function: Bound and its Preceding/Following are boxed, and a
  bound carries offset_expr next to the deprecated offset
- interval literals: the precision_mode oneof became a plain precision
  field, and the deprecated microseconds form is gone
- aggregates: a grouping set references relation-level expressions by
  index rather than embedding them
- virtual tables: rows are expressions, values was removed
- function calls: the deprecated args field was removed in favour of
  arguments
- types and literals: Timestamp, TimestampTz and Time gave way to the
  precision-carrying variants, and UserDefinedTypeReference to UserDefined
- OuterReference: steps_out moved into a oneof alongside a relation
  reference
- Plan requires execution_behavior, RelCommon requires rel_anchor, and
  RelType gained LateralJoin and TopN
- the enum expression form was removed

The consumer paths that read the removed fields are deleted rather than
kept: those fields are reserved in the spec, so they cannot appear on a
decoded plan. The checked-in JSON fixtures that used the old grouping and
fetch shapes are rewritten accordingly.

The pbjson-types pin moves to 0.9.0 to match the version substrait uses.
…expr as CurrentRow, and read RelCommon from UpdateRel
Comment thread datafusion/substrait/Cargo.toml
Forward the protox feature so consumers can build without an external
protoc binary.

Merge main and retain its grouping-set output order and type metadata
support with the updated Substrait API.

Assisted-by: gpt-6-astra (OpenAI Codex)
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-substrait v55.1.0 (current)
       Built [ 319.032s] (current)
     Parsing datafusion-substrait v55.1.0 (current)
      Parsed [   0.012s] (current)
    Building datafusion-substrait v55.1.0 (baseline)
       Built [ 304.694s] (baseline)
     Parsing datafusion-substrait v55.1.0 (baseline)
      Parsed [   0.012s] (baseline)
    Checking datafusion-substrait v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.227s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure trait_method_missing: pub trait method removed or renamed ---

Description:
A trait method is no longer callable, and may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-item-signature
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/trait_method_missing.ron

Failed in:
  method consume_enum of trait SubstraitConsumer, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/ad995f4dd276695f3b49af03f875b86fea22de77/datafusion/substrait/src/logical_plan/consumer/substrait_consumer.rs:391

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 626.165s] datafusion-substrait

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Sep 22, 2026
@codecov-commenter

codecov-commenter commented Sep 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.59829% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.78%. Comparing base (06aa131) to head (aead3be).

Files with missing lines Patch % Lines
...ait/src/logical_plan/consumer/rel/aggregate_rel.rs 64.70% 4 Missing and 2 partials ⚠️
...ubstrait/src/logical_plan/consumer/expr/literal.rs 66.66% 3 Missing ⚠️
.../src/logical_plan/consumer/expr/window_function.rs 96.47% 0 Missing and 3 partials ⚠️
...ion/substrait/src/logical_plan/consumer/rel/mod.rs 0.00% 3 Missing ⚠️
...ait/src/logical_plan/producer/rel/aggregate_rel.rs 25.00% 3 Missing ⚠️
...bstrait/src/logical_plan/consumer/rel/fetch_rel.rs 50.00% 0 Missing and 2 partials ⚠️
...on/substrait/src/logical_plan/consumer/expr/mod.rs 85.71% 1 Missing ⚠️
...usion/substrait/src/logical_plan/consumer/types.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #24861    +/-   ##
========================================
  Coverage   82.77%   82.78%            
========================================
  Files        1147     1147            
  Lines      450909   451000    +91     
  Branches   450909   451000    +91     
========================================
+ Hits       373252   373356   +104     
+ Misses      54934    54919    -15     
- Partials    22723    22725     +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants