Skip to content

fix: preserve field metadata for simple CASE branches - #26145

Draft
osipovartem wants to merge 13 commits into
apache:mainfrom
Embucket:upstream-case-field-metadata
Draft

osipovartem wants to merge 13 commits into
apache:mainfrom
Embucket:upstream-case-field-metadata

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Part of #24860. This is logical schema propagation for CASE, complementary to #24865.

What changes?

  • Preserve Arrow field metadata when supported CASE result branches agree on type and metadata. Untyped NULL does not erase it; typed NULL carrying matching metadata remains eligible.
  • Handle columns, literals, casts, aliases, and nested CASE iteratively. The ordinary unmarked CASE fast path is unchanged.
  • Preserve metadata returned by shallow scalar/higher-order functions (including a shallow nested CASE argument) and explicit-metadata Cast, TryCast, and Alias wrappers using their existing to_field contract. Exact delegation is limited to expression depth 8 to bound repeated field inference. Deeper function branches and unsupported branch forms conservatively leave CASE metadata empty.
  • Physical CaseExpr metadata and constant-false simplification remain separate follow-ups to CASE expressions drop field metadata, silently stripping Arrow extension types and breaking metadata-aware UDFs #24860.

Validation

  • Local datafusion-expr and datafusion-optimizer suites: 283 + 934 tests passed. Focused CASE tests, formatting, and datafusion-expr test-target Clippy also pass.
  • SQL integration test passes in both Apache and Embucket builds. It checks planned and optimized logical field metadata for searched and simple CASE, matching, NULL, and conflicting result branches, plus executes a result-row smoke test. The latest focused core_integration run passed and cargo fmt --all --check passed.
  • Focused regression covers matching/conflicting metadata, absent ELSE, multiple WHEN branches, typed/untyped NULL, marked and unmarked literals, negative and unsupported-NOT expressions, nested CASE/cast, shallow nested CASE within a metadata-bearing UDF, explicit Cast/TryCast/Alias wrappers, deep fallback, outer references, scalar variables, placeholders, lambda variables, first-branch NULL, and type mismatch.
  • Independent read-only reviews approved correctness, bounded planning cost, API compatibility, and expression-level and SQL-level test scope.
  • The last posted Codecov comment measured 74ac5de5b at 74.06% patch coverage. Additional branch-focused tests were added in cc221b5fd, d03385576, and 3a8631b7f; a report for the current head is still pending. Some old partials are defensive errors and Rust ? branches in tests. No coverage percentage is claimed for the current head yet.

@codecov-commenter

codecov-commenter commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.00000% with 110 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.76%. Comparing base (bc693cc) to head (3a8631b).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/expr/src/expr_schema.rs 74.51% 24 Missing and 82 partials ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs 83.33% 1 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26145      +/-   ##
==========================================
- Coverage   82.77%   82.76%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      450580   451349     +769     
  Branches   450580   451349     +769     
==========================================
+ Hits       372946   373574     +628     
- Misses      54931    54963      +32     
- Partials    22703    22812     +109     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@osipovartem
osipovartem marked this pull request as draft October 8, 2026 23:00
@osipovartem

Copy link
Copy Markdown
Contributor Author

Independent review found that the iterative CASE metadata walker still needs contract-equivalent UDF argument fields (including coerced scalar literals and CASE nullability) and a bounded higher-order-function path. I marked this PR draft until those correctness and planning-cost issues are resolved. Please do not merge the current version.

@osipovartem osipovartem changed the title fix: preserve common field metadata through CASE expressions fix: preserve field metadata for simple CASE branches Oct 8, 2026
@osipovartem
osipovartem marked this pull request as ready for review October 9, 2026 08:58
@osipovartem
osipovartem marked this pull request as draft October 9, 2026 12:37
@osipovartem

Copy link
Copy Markdown
Contributor Author

Independent review found that nested alias field inference can amplify CASE metadata planning cost. The standalone fix is #26156 (12 nested aliases: 4,096 source-field resolutions before, one after). I marked this PR draft until that fix lands, this branch is rebased, and its CASE-specific behavior is re-reviewed. Physical CaseExpr propagation remains the separate follow-up described above.

@osipovartem

Copy link
Copy Markdown
Contributor Author

I checked the latest Codecov report (74.06% patch coverage, 104 lines not fully covered) against its line-level data. The four fully missed lines in the new CASE production walker are defensive internal_err! paths for work-stack invariants; the supported execution paths are exercised. Most remaining misses/partials are in test-only helpers and ? error branches in tests. I will not add artificial failure-path tests solely to raise the percentage. The PR remains draft pending #26156 and a fresh correctness/performance review after rebase.

(cherry picked from commit 816119e839317cca1b31ef02ad0e8e302e0c60d0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants