Repository navigation
Substrait producer omits the required output_type on aggregate functions #25049
Description
Describe the bug
The Substrait producer emits aggregate calls with output_type: None. The logical plan already has a known result type, but the exported call does not carry it. This affects count, sum, avg and min in the reproducer below.
Reproduced on main at 8a9228164fddeaa03c9ee510b641990e403f2eb5.
To reproduce
From a DataFusion checkout, using its pinned Rust toolchain:
git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout c4d3ffd74842aa38079b3d523913199dc615a352
mkdir -p datafusion/substrait/examples
cp conformance-cases/probe/datafusion_producer_probe.rs datafusion/substrait/examples/corpus_producer.rs
cargo run --locked -p datafusion-substrait --example corpus_producer -- --aggregate-output-typesUse an unused example filename if corpus_producer.rs already exists. The probe registers an empty named table with a required Int64 column and calls to_substrait_plan for each aggregate. It inspects the produced protobuf directly, without converting it back through a consumer.
| Query | Logical output type | Exported AggregateFunction.output_type |
|---|---|---|
SELECT count(i) FROM t |
Int64 | None |
SELECT sum(i) FROM t |
Int64 | None |
SELECT avg(i) FROM t |
Float64 | None |
SELECT min(i) FROM t |
Int64 | None |
The final diagnostic is {"cases":4,"missing_output_types":4}.
Expected behavior
AggregateFunction.output_type must be set to the return type derived from the referenced function declaration. The producer should write a conforming type, or report that it cannot represent that function contract.
from_aggregate_function currently sets output_type: None explicitly. A consumer that validates required fields can reject these plans; one that uses the declaration for schema inference has no type to use.
This report is about the missing declaration. It does not prescribe an integer AVG return type or address aggregate phases. #24967 and #25045 concern phase handling in the consumer; #15831 fixed the corresponding missing-output-type issue for BinaryExpr.
Activity
namanjain24-sudo commentedon Sep 8, 2026
take
fix: Set Substrait output_type on aggregate functions
fix: Set Substrait output_type on aggregate functions
fix: Set Substrait output_type on aggregate functions
fix: Set Substrait output_type on aggregate functions
alamb commentedon Sep 11, 2026
Same question: #25190 (comment)
I don't understand what you are doing, or why this is a bug. Can you please fix clarify on this PR what the actual problem is you are solving?
alexandrefimov commentedon Sep 11, 2026
Yes, this is a compatibility problem. I should have included a concrete example.
DataFusion produces aggregate calls without
output_type. Current substrait-java tries to read this field and rejects a minimalsum:i64plan with:UnsupportedOperationException: Type is not setThe same plan works when
output_typeis present and produces the expected nullablei64field. DataFusion-to-DataFusion round trips do not catch this because the problem only shows up when another implementation reads the plan.So #25090 fixes plans exported by DataFusion for consumers such as substrait-java.
namanjain24-sudo commentedon Sep 11, 2026
@alamb answered this in full on #25090, since you asked for it there:
#25090 (comment) — the engine, the two source
lines it fails on, and a before/after run.
One thing worth adding to @alexandrefimov's reply, because it matters if you try to reproduce it. The
error is exactly right for a minimal hand-built
sum:i64plan, but a plan that DataFusion actuallyexports never reaches that check. Handing the output of
to_substrait_planstraight tosubstrait-java 0.103.0 fails earlier, at extension resolution:
We write
extension_urn_reference: u32::MAXfor every function declaration (extensions.rs:120),which is #11545. Only once a URN is registered on the plan and the compound key
sum:i64is useddoes it get as far as
output_typeand throwUnsupportedOperationException: Type is not set. Withthis change and that same workaround the plan is accepted and comes back as nullable
i64, which iswhat our own optimized plan says
sum(t.i)is.So it is a real compatibility bug and it is the second of two gates rather than the only one. #11545
is the first. Fixing either alone still leaves our aggregate plans unreadable by substrait-java.
fix: Set Substrait output_type on aggregate functions
test: run a DataFusion plan through substrait-java and Spark
fix: Set Substrait output_type on aggregate functions
fix: Set Substrait output_type on aggregate functions (apache#25090)
test: run a DataFusion plan through substrait-java and Spark
fix: Set Substrait output_type on window functions and LIKE (apache#2…
test: run a DataFusion plan through substrait-java and Spark
fix: Emit LIKE the way the Substrait extension defines it (apache#25442)
fix: Set Substrait output_type on aggregate functions (apache#25090)
fix: Set Substrait output_type on window functions and LIKE (apache#2…
fix: Emit LIKE the way the Substrait extension defines it (apache#25442)