Repository navigation
fix: avoid sort-based joins for nested float join keys with negative zeros - #26146
Open
entity-0-0-2 wants to merge 1 commit into
Open
entity-0-0-2 wants to merge 1 commit into
entity-0-0-2 wants to merge 1 commit into
Conversation
…zeros SortMergeJoinExec and PiecewiseMergeJoinExec normalize -0.0 to +0.0 in their merge comparators, but the sorts feeding them use Arrow's IEEE total order where -0.0 < +0.0. For flat keys the two orders agree. For nested keys with float leaves the comparison continues past the zero element and the orders diverge, so the merge stream sees mis-sorted input and silently drops matched rows. The divergence opened when normalize_float_zero learned to recurse into nested children in apache#24827. The planner now routes joins whose keys are nested with float leaves to HashJoinExec instead of SortMergeJoinExec, and to NestedLoopJoinExec instead of PiecewiseMergeJoinExec. Key types where both orders provably agree keep the sort-based fast paths. Fixes apache#26115
github-actions
Bot
added
core
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sort merge join silently drops rows when the join keys contain nested float columns with negative zeros: the issue's query loses row (2, 11) with no error.
The comparator built in datafusion/physical-plan/src/joins/utils.rs (JoinKeyComparator::build) normalizes negative zeros on both sides of the key, but since d42cd85 (#24827) that normalization recurses into nested children while the sort feeding the merge keeps Arrow's IEEE total order, where -0.0 sorts before +0.0. For nested keys the two orders disagree and the merge stream skips matches.
The physical planner now routes joins with nested float keys to hash join (or nested loop join for range conditions), so sort-based joins only see keys where normalization and sort order agree. A sqllogictest covering the issue's query fails on main with the missing row and passes with the change.
Fixes #26115