Skip to content

fix: preserve schema consistency across union and cast - #26201

Open
Amogh-2404 wants to merge 1 commit into
apache:mainfrom
Amogh-2404:fix/union-recompute-schema-field-metadata
Open

Amogh-2404 wants to merge 1 commit into
apache:mainfrom
Amogh-2404:fix/union-recompute-schema-field-metadata

Conversation

@Amogh-2404

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

CAST(coalesce(non_null_col, lit) AS T) under UNION ALL + aggregate fails planning when the column carries Arrow field metadata (PARQUET:field_id):

Internal error: Physical input schema should be the same as the one converted
from logical input schema. Differences:
  - field metadata at index 0 [x]: (physical) {"PARQUET:field_id": "1"} vs (logical) {}.

coalesce on a NOT NULL column simplifies to the column. Child projections then advertise that metadata; the Union's stored schema still has {} from the pre-simplify coalesce field. Physical planning derives from the simplified exprs, so the aggregate schema check fails.

What changes are included in this PR?

LogicalPlan::recompute_schema for Union no longer keeps the old schema whenever field count is unchanged. It re-derives via Union::try_new so metadata/nullability track the current children. If width is unchanged but try_new fails (loose-typed children before coercion), the previous schema is kept; width mismatches still return the try_new error.

What is the testing strategy for this PR?

  • Integration: datafusion/core/tests/sql/union_field_metadata.rs — issue query (m=3, c=6), controls (no metadata / CAST(b) / direct b AS x / no union), mixed-metadata legs.
  • Unit: recompute_schema_refreshes_union_field_metadata, loose-union preserve, width-mismatch error.

Verified fail-before / pass-after on coalesce_cast_union_aggregate_preserves_field_metadata by temporarily reverting the plan.rs change.

Commands run:

cargo test -p datafusion-expr --lib logical_plan::plan::tests::recompute_schema
cargo test -p datafusion --test core_integration coalesce_cast_union_aggregate
cargo test -p datafusion --test core_integration union_
cargo test -p datafusion-optimizer --lib
cargo test -p datafusion --lib physical_planner
cargo test -p datafusion-sqllogictest --test sqllogictests metadata.slt
cargo clippy -p datafusion-expr -p datafusion --tests -- -D warnings
cargo fmt --all -- --check

Not run: full ./dev/rust_lint.sh, full extended CI suite, full sqllogictest corpus.

Are there any user-facing changes?

No public API change. Queries that previously hit this internal error now plan. Union output may correctly carry field metadata that a stale schema had dropped after expression rewrites. CAST metadata inheritance is unchanged.

@github-actions github-actions Bot added logical-expr Logical plan and expressions core Core DataFusion crate labels Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate logical-expr Logical plan and expressions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Internal error: CAST over an expression simplified to a metadata-carrying column, under UNION ALL + aggregate (physical vs logical field metadata)

1 participant