Repository navigation
Commit 62f2943
authored
fix: Set Substrait output_type on window functions and LIKE (#25367)
## Which issue does this PR close?
- Closes #25366.
## Rationale for this change
The producer left `output_type` unset on window function calls and on
`LIKE` / `ILIKE`, including the `not` that wraps a negated one.
Substrait documents both `Expression.WindowFunction.output_type` and
`Expression.ScalarFunction.output_type` as:
> Must be set to the return type of the function, exactly as derived
using the declaration in the extension.
A consumer that reads the field rejects such a call. substrait-java
0.103.0 refuses both plans, from `ProtoTypeConverter.from:125`:
| plan produced for | error before this PR |
| --- | --- |
| `SELECT sum(i) OVER (ORDER BY i) FROM t` |
`UnsupportedOperationException: Type is not set` at
`ProtoExpressionConverter.fromWindowFunction:406` |
| `SELECT i FROM t WHERE CAST(i AS VARCHAR) LIKE '1%'` |
`UnsupportedOperationException: Type is not set` at
`ProtoExpressionConverter:201`, via `ProtoRelConverter.newFilter` |
A DataFusion round trip cannot catch this: the consumer reads
`output_type` only in `consumer/expr/cast.rs`, so producer and consumer
agree on the omission.
#15831 and #20597 set this field for binary and unary expressions,
`from_function` and higher order functions, and #25049 and #25090 cover
`AggregateFunction`. These were the remaining expression kinds.
## What changes are included in this PR?
Both types come from the expression itself, so they match what
DataFusion derives rather than being restated in the producer:
- `from_window_function` derives the type from
`Expr::WindowFunction(..).to_field(schema)` and writes it on the call it
builds. `make_substrait_window_function` had one caller and eight
parameters once the type was added, so its body now sits in
`from_window_function` and the helper is gone.
- `from_like` derives the type from `Expr::Like(..).to_field(schema)`
and sets it on the `like` call and on the `not` wrapper of a negated
one, which yields that same type. `make_substrait_like_expr` had
`from_like` as its only caller, so, as with the window helper, its body
now sits in `from_like`.
## What is the testing strategy for this PR?
- Two unit tests next to the existing `binary_expr_output_type`:
`window_function_output_type` asserts `sum(i)` over a nullable `i64`
carries a nullable `i64`, and `like_output_type` asserts a `LIKE` over a
nullable input carries a nullable boolean, on the `like` call, on the
`not` of a negated one, and on the `like` nested inside that `not`.
- Each half was reverted on its own to check the tests pin it: without
the window change only `window_function_output_type` fails, without the
`LIKE` change only `like_output_type` fails.
- `cargo test -p datafusion-substrait` passes (60 unit, 210 integration
with the 6 that were already ignored, 3 doc tests), and
`./ci/scripts/rust_clippy.sh`, the workspace clippy CI runs, is clean.
- Emitted protobuf before and after, read directly rather than through a
consumer:
| plan | before | after |
| --- | --- | --- |
| window `sum` | `output_type` unset | set |
| `like` | unset | set |
| `not` around a negated `like` | unset | set |
- With the same plans, substrait-java 0.103.0 no longer raises `Type is
not set`. Both now get through `ProtoPlanConverter`, while the plans
built from `main` still fail there. Each then stops further along for
reasons that have nothing to do with this field: substrait-spark has no
visitor for a window function that appears in a project expression, and
Spark's `like` takes two arguments while the call we emit passes three,
the third being the escape character. That second one is #25442, which
emits `LIKE` in the two-argument form the extension defines.
## Are there any user-facing changes?
No API changes. Plans produced for window functions and `LIKE` now carry
the expression's type, which consumers that require it will accept.1 parent 6e319bd commit 62f2943
2 files changed
Lines changed: 138 additions & 55 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
245 | 246 | | |
246 | 247 | | |
247 | 248 | | |
248 | | - | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
255 | | - | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
264 | | - | |
265 | | - | |
266 | | - | |
267 | | - | |
268 | | - | |
| 249 | + | |
269 | 250 | | |
270 | 251 | | |
271 | 252 | | |
272 | 253 | | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
273 | 261 | | |
274 | 262 | | |
275 | 263 | | |
| |||
293 | 281 | | |
294 | 282 | | |
295 | 283 | | |
296 | | - | |
| 284 | + | |
297 | 285 | | |
298 | 286 | | |
299 | 287 | | |
300 | 288 | | |
301 | 289 | | |
302 | | - | |
| 290 | + | |
303 | 291 | | |
304 | 292 | | |
305 | 293 | | |
| |||
309 | 297 | | |
310 | 298 | | |
311 | 299 | | |
312 | | - | |
| 300 | + | |
| 301 | + | |
313 | 302 | | |
314 | 303 | | |
315 | 304 | | |
| |||
452 | 441 | | |
453 | 442 | | |
454 | 443 | | |
455 | | - | |
| 444 | + | |
456 | 445 | | |
457 | 446 | | |
458 | | - | |
| 447 | + | |
| 448 | + | |
459 | 449 | | |
460 | 450 | | |
| 451 | + | |
461 | 452 | | |
462 | 453 | | |
463 | 454 | | |
| |||
479 | 470 | | |
480 | 471 | | |
481 | 472 | | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
482 | 514 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| |||
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
30 | | - | |
| 32 | + | |
31 | 33 | | |
32 | 34 | | |
33 | 35 | | |
| |||
79 | 81 | | |
80 | 82 | | |
81 | 83 | | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
92 | 92 | | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | 93 | | |
103 | | - | |
| 94 | + | |
104 | 95 | | |
105 | | - | |
| 96 | + | |
106 | 97 | | |
107 | | - | |
108 | | - | |
| 98 | + | |
| 99 | + | |
109 | 100 | | |
110 | | - | |
| 101 | + | |
111 | 102 | | |
112 | | - | |
| 103 | + | |
113 | 104 | | |
114 | 105 | | |
115 | 106 | | |
116 | 107 | | |
117 | 108 | | |
118 | 109 | | |
119 | 110 | | |
120 | | - | |
| 111 | + | |
121 | 112 | | |
122 | | - | |
| 113 | + | |
123 | 114 | | |
124 | 115 | | |
125 | 116 | | |
| |||
185 | 176 | | |
186 | 177 | | |
187 | 178 | | |
188 | | - | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
189 | 240 | | |
190 | 241 | | |
191 | 242 | | |
| |||
0 commit comments