fix: preserve schema consistency across union and cast

[Amogh-2404]

Which issue does this PR close?

• Closes Internal error: CAST over an expression simplified to a metadata-carrying column, under UNION ALL + aggregate (physical vs logical field metadata) #26178.

Rationale for this change

CAST(coalesce(non_null_col, lit) AS T) under UNION ALL + aggregate fails planning when the column carries Arrow field metadata (PARQUET:field_id):

Internal error: Physical input schema should be the same as the one converted
from logical input schema. Differences:
  - field metadata at index 0 [x]: (physical) {"PARQUET:field_id": "1"} vs (logical) {}.

coalesce on a NOT NULL column simplifies to the column. Child projections then advertise that metadata; the Union's stored schema still has {} from the pre-simplify coalesce field. Physical planning derives from the simplified exprs, so the aggregate schema check fails.

What changes are included in this PR?

LogicalPlan::recompute_schema for Union no longer keeps the old schema whenever field count is unchanged. It re-derives via Union::try_new so metadata/nullability track the current children. If width is unchanged but try_new fails (loose-typed children before coercion), the previous schema is kept; width mismatches still return the try_new error.

What is the testing strategy for this PR?

• Integration: datafusion/core/tests/sql/union_field_metadata.rs — issue query (m=3, c=6), controls (no metadata / CAST(b) / direct b AS x / no union), mixed-metadata legs.
• Unit: recompute_schema_refreshes_union_field_metadata, loose-union preserve, width-mismatch error.

Verified fail-before / pass-after on coalesce_cast_union_aggregate_preserves_field_metadata by temporarily reverting the plan.rs change.

Commands run:

cargo test -p datafusion-expr --lib logical_plan::plan::tests::recompute_schema
cargo test -p datafusion --test core_integration coalesce_cast_union_aggregate
cargo test -p datafusion --test core_integration union_
cargo test -p datafusion-optimizer --lib
cargo test -p datafusion --lib physical_planner
cargo test -p datafusion-sqllogictest --test sqllogictests metadata.slt
cargo clippy -p datafusion-expr -p datafusion --tests -- -D warnings
cargo fmt --all -- --check

Not run: full ./dev/rust_lint.sh, full extended CI suite, full sqllogictest corpus.

Are there any user-facing changes?

No public API change. Queries that previously hit this internal error now plan. Union output may correctly carry field metadata that a stale schema had dropped after expression rewrites. CAST metadata inheritance is unchanged.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 72.63158% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.80%. Comparing base (a4b8198) to head (ced67e8).

Files with missing lines	Patch %	Lines
datafusion/expr/src/logical_plan/plan.rs	72.63%	7 Missing and 19 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26200      +/-   ##
==========================================
- Coverage   82.80%   82.80%   -0.01%     
==========================================
  Files        1149     1149              
  Lines      452205   452296      +91     
  Branches   452205   452296      +91     
==========================================
+ Hits       374468   374540      +72     
- Misses      54942    54944       +2     
- Partials    22795    22812      +17     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.