fix: Set Substrait output_type on window functions and LIKE by namanjain24-sudo · Pull Request #25367 · apache/datafusion

namanjain24-sudo · 2026-09-16T09:18:13Z

Which issue does this PR close?

Closes Substrait producer omits output_type on window functions and LIKE expressions #25366.

Rationale for this change

The producer left output_type unset on window function calls and on LIKE / ILIKE, including the not that wraps a negated one. Substrait documents both Expression.WindowFunction.output_type and Expression.ScalarFunction.output_type as:

Must be set to the return type of the function, exactly as derived using the declaration in the extension.

A consumer that reads the field rejects such a call. substrait-java 0.103.0 refuses both plans, from ProtoTypeConverter.from:125:

plan produced for	error before this PR
`SELECT sum(i) OVER (ORDER BY i) FROM t`	`UnsupportedOperationException: Type is not set` at `ProtoExpressionConverter.fromWindowFunction:406`
`SELECT i FROM t WHERE CAST(i AS VARCHAR) LIKE '1%'`	`UnsupportedOperationException: Type is not set` at `ProtoExpressionConverter:201`, via `ProtoRelConverter.newFilter`

A DataFusion round trip cannot catch this: the consumer reads output_type only in consumer/expr/cast.rs, so producer and consumer agree on the omission.

#15831 and #20597 set this field for binary and unary expressions, from_function and higher order functions, and #25049 and #25090 cover AggregateFunction. These were the remaining expression kinds.

What changes are included in this PR?

Both types come from the expression itself, so they match what DataFusion derives rather than being restated in the producer:

from_window_function derives the type from Expr::WindowFunction(..).to_field(schema) and writes it on the call it builds. make_substrait_window_function had one caller and eight parameters once the type was added, so its body now sits in from_window_function and the helper is gone.
from_like derives the type from Expr::Like(..).to_field(schema) and sets it on the like call and on the not wrapper of a negated one, which yields that same type. make_substrait_like_expr had from_like as its only caller, so, as with the window helper, its body now sits in from_like.

What is the testing strategy for this PR?

Two unit tests next to the existing binary_expr_output_type: window_function_output_type asserts sum(i) over a nullable i64 carries a nullable i64, and like_output_type asserts a LIKE over a nullable input carries a nullable boolean, on the like call, on the not of a negated one, and on the like nested inside that not.
Each half was reverted on its own to check the tests pin it: without the window change only window_function_output_type fails, without the LIKE change only like_output_type fails.
cargo test -p datafusion-substrait passes (60 unit, 210 integration with the 6 that were already ignored, 3 doc tests), and ./ci/scripts/rust_clippy.sh, the workspace clippy CI runs, is clean.
Emitted protobuf before and after, read directly rather than through a consumer:

plan	before	after
window `sum`	`output_type` unset	set
`like`	unset	set
`not` around a negated `like`	unset	set

With the same plans, substrait-java 0.103.0 no longer raises Type is not set. Both now get through ProtoPlanConverter, while the plans built from main still fail there. Each then stops further along for reasons that have nothing to do with this field: substrait-spark has no visitor for a window function that appears in a project expression, and Spark's like takes two arguments while the call we emit passes three, the third being the escape character. That second one is fix: Emit LIKE the way the Substrait extension defines it #25442, which emits LIKE in the two-argument form the extension defines.

Are there any user-facing changes?

No API changes. Plans produced for window functions and LIKE now carry the expression's type, which consumers that require it will accept.

Substrait documents output_type on both ScalarFunction and WindowFunction as the return type of the function, and a consumer that reads it rejects a call that leaves it unset: substrait-java refuses a window function plan and a LIKE plan with "Type is not set". Both types are derived from the expression itself, so they match what DataFusion derives: from_window_function from Expr::WindowFunction, and make_substrait_like_expr from Expr::Like, which also covers the not() wrapper a negated LIKE emits. make_substrait_window_function had a single caller and its body now sits in from_window_function, and make_substrait_like_expr takes the Like rather than its five fields, so neither grows an extra parameter. A DataFusion round trip cannot catch this, because the consumer reads output_type only when it converts a cast.

codecov-commenter · 2026-09-18T03:18:53Z

Codecov Report

❌ Patch coverage is 73.61111% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.61%. Comparing base (edc936f) to head (7994274).
⚠️ Report is 213 commits behind head on main.

Files with missing lines	Patch %	Lines
.../src/logical_plan/producer/expr/window_function.rs	75.00%	4 Missing and 6 partials ⚠️
.../src/logical_plan/producer/expr/scalar_function.rs	71.87%	4 Missing and 5 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25367      +/-   ##
==========================================
+ Coverage   82.33%   82.61%   +0.27%     
==========================================
  Files        1137     1147      +10     
  Lines      432498   444837   +12339     
  Branches   432498   444837   +12339     
==========================================
+ Hits       356116   367504   +11388     
- Misses      54843    55001     +158     
- Partials    21539    22332     +793

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

make_substrait_like_expr had one caller once it took the Like itself, so its body now sits in from_like, the same as the window function helper. Also restore the doc comment on to_substrait_unary_scalar_fn, which the previous commit dropped by accident.

namanjain24-sudo · 2026-09-28T08:34:57Z

Bumping this — still open for review whenever someone has bandwidth.

kosiew

@namanjain24-sudo,

Thanks for working on this. The changes look good overall. I left one non-blocking suggestion to strengthen the regression coverage.

kosiew · 2026-10-02T10:14:08Z

+    /// Substrait requires the return type on a window function call, and a
+    /// consumer that reads it rejects the call when it is unset.
+    #[test]
+    fn window_function_output_type() -> datafusion::common::Result<()> {


Could you also add a COUNT(*) OVER () case, for example using count_all_window(), and assert that its output is required i64? The current test only covers nullable output, so this would catch a regression where output_type is always emitted as nullable.

Guards against a regression where output_type is always emitted as nullable, which the existing window_function_output_type test (over a nullable column) would not catch.

namanjain24-sudo · 2026-10-02T16:58:42Z

@kosiew added count_window_function_output_type_is_non_nullable, asserting COUNT(*) OVER () emits a non-nullable i64 output_type.

kosiew

@namanjain24-sudo,

Thanks for working on this. The changes correctly populate Substrait output types for window functions and LIKE expressions while preserving DataFusion's derived type and nullability. The added COUNT(*) window test also covers the important non-nullable case.

kosiew · 2026-10-05T04:19:24Z

🚀
@namanjain24-sudo
Thank you for your contribution.

## Which issue does this PR close? - Closes apache#25368. ## Rationale for this change `functions_string.yaml` defines one function for this, with two arguments and an option: ```yaml name: like impls: - args: - value: "varchar<L1>" name: "input" - value: "varchar<L2>" name: "match" options: case_sensitivity: values: [ CASE_SENSITIVE, CASE_INSENSITIVE, CASE_INSENSITIVE_ASCII ] return: "boolean" ``` The producer emitted three arguments, registered `ilike` for the case insensitive form, and never set the option. Neither `ilike` nor an escape argument appears in any extension file, in the pinned crate or upstream. ## What changes are included in this PR? - `make_substrait_like_expr` registers `like`, emits the two arguments the definition takes, and sets `case_sensitivity` to `CASE_INSENSITIVE` for `ILIKE`. A plain `LIKE` sets no option, which leaves the default. - `LIKE ... ESCAPE` is now `not_impl_err`. There is no escape in the definition, so the third argument only produced a call a consumer cannot bind. - The consumer reads `case_sensitivity`, taking the first value it supports and refusing the ones it does not, as Substrait requires. It still accepts the `ilike` name and the three argument form, so plans written by an older DataFusion still load. ## What is the testing strategy for this PR? - New producer tests: `LIKE` emits `like` with two arguments and no option, `ILIKE` emits the `case_sensitivity` option, and an escape character is rejected. - `roundtrip_like` and `roundtrip_ilike` still pass, which is what covers the consumer: with the option reading removed, `roundtrip_ilike` fails because `ILIKE` comes back as `LIKE`. Removing the producer half likewise fails the new producer test. - `cargo test -p datafusion-substrait` passes and `./ci/scripts/rust_clippy.sh` is clean. Measured against substrait-java 0.103.0 and Spark 3.5.4, over `t(s)` holding `abc` and `ABC`, with the apache#11545 URN patched and `output_type` filled (apache#25049): | plan | before | after | | --- | --- | --- | | `s LIKE 'a%'` | `AnalysisException: WRONG_NUM_ARGS ... requires 2 parameters but the actual number is 3` | runs, returns `abc` | | `s ILIKE 'a%'` | `IllegalArgumentException: Unexpected scalar function with key ilike:str_str` | runs, returns `abc` | **The second row is a trade-off worth stating.** `ILIKE` used to fail loudly there, and now it is accepted but answered case sensitively, because substrait-spark does not read the option: `case_sensitivity` appears nowhere in its sources. Substrait says a consumer that does not recognise an option must reject the call, so that looks like a gap on that side, and I am happy to report it there. If you would rather not emit something a known consumer mishandles, the alternative is to reject `ILIKE` as well until a consumer honours the option, and I will change it. ## Are there any user-facing changes? `LIKE` and `ILIKE` are emitted in the form the extension defines, so consumers that bind arguments by the declaration can now read them. `LIKE ... ESCAPE` is rejected instead of being emitted in a shape no consumer can bind; DataFusion round trips it today only because its own consumer reads the same private convention. This touches `make_substrait_like_expr`, which apache#25367 folds into `from_like`, so whichever lands second needs a small rebase. --------- Co-authored-by: namanjain24-sudo <180642416+namanjain24-sudo@users.noreply.github.com>

github-actions Bot added the substrait Changes to the substrait crate label Sep 16, 2026

This was referenced Sep 16, 2026

Substrait producer emits LIKE with three arguments and an undefined ilike function #25368

Closed

fix: Emit LIKE the way the Substrait extension defines it #25442

Merged

namanjain24-sudo force-pushed the fix-substrait-window-like-output-type branch from dd81646 to be39554 Compare September 18, 2026 02:58

namanjain24-sudo mentioned this pull request Sep 17, 2026

fix: Set Substrait output_type on aggregate functions #25090

Merged

kosiew approved these changes Oct 2, 2026

View reviewed changes

test: assert COUNT(*) OVER () emits a non-nullable Substrait output_type

7994274

Guards against a regression where output_type is always emitted as nullable, which the existing window_function_output_type test (over a nullable column) would not catch.

kosiew approved these changes Oct 5, 2026

View reviewed changes

kosiew added this pull request to the merge queue Oct 5, 2026

Merged via the queue into apache:main with commit 62f2943 Oct 5, 2026
42 checks passed

github-actions Bot added the v56.0.0 label Oct 5, 2026

Navigation Menu

Repository navigation

fix: Set Substrait output_type on window functions and LIKE - #25367

fix: Set Substrait output_type on window functions and LIKE#25367
kosiew merged 3 commits into
apache:mainfrom
namanjain24-sudo:fix-substrait-window-like-output-type

namanjain24-sudo commented Sep 16, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Sep 18, 2026 •

edited

Loading

Uh oh!

namanjain24-sudo commented Sep 28, 2026

Uh oh!

kosiew left a comment

Uh oh!

kosiew Oct 2, 2026

Uh oh!

namanjain24-sudo commented Oct 2, 2026

Uh oh!

kosiew left a comment

Uh oh!

kosiew commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

namanjain24-sudo commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

namanjain24-sudo commented Sep 28, 2026

Uh oh!

kosiew left a comment

Choose a reason for hiding this comment

Uh oh!

kosiew Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

namanjain24-sudo commented Oct 2, 2026

Uh oh!

kosiew left a comment

Choose a reason for hiding this comment

Uh oh!

kosiew commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

namanjain24-sudo commented Sep 16, 2026 •

edited

Loading

codecov-commenter commented Sep 18, 2026 •

edited

Loading