Skip to content

box/aabb2d: fix wrong face selected below AABB min bound in sphere/circle test - #496

Merged
recp merged 1 commit into
recp:masterfrom
94xhn:fix/aabb-sphere-below-min-face-selection
Jul 18, 2026
Merged

box/aabb2d: fix wrong face selected below AABB min bound in sphere/circle test#496
recp merged 1 commit into
recp:masterfrom
94xhn:fix/aabb-sphere-below-min-face-selection

Conversation

@94xhn

@94xhn 94xhn commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Problem

glm_aabb_sphere() (include/cglm/box.h) and glm_aabb2d_circle() (include/cglm/aabb2d.h) compute the closest point on an AABB to a sphere/circle center per axis using:

a = (s[i] < box[0][i]) + (s[i] > box[1][i]);
...
box[!(a - 1)][i]

a is 1 for either out-of-range direction (below box[0] or above box[1]), and !(a - 1) collapses to index 1 (the max face) whenever a == 1, regardless of which side the point is actually on. So when the query point is below the box's min bound on an axis, the code clamps to the max face instead of the min face, producing an incorrect (too-large) distance-squared contribution and false-negative intersection results.

Fix

Use the already-computed "above max" boolean directly as the face index instead of the !(a - 1) derivation:

box[s[i] > box[1][i]][i]

This selects box[0] (min face) when the point is below the min bound and box[1] (max face) when above the max bound. The in-range case is unaffected — it's already zeroed out by the existing (a != 0) multiplier, so the index value there doesn't matter.

Same one-line-per-axis change applied identically to both glm_aabb_sphere (3 axes) and glm_aabb2d_circle (2 axes), since they share the exact same root cause.

Verification

  • Manual cases: sphere/circle center below box[0] on one axis, below on two axes simultaneously (3D), etc. — all mismatched (false negative) before the fix, correct after.
  • Randomized fuzz test, 200,000 cases with random box bounds / sphere center / radius, checked against a hand-written point-to-AABB squared-distance reference: 3418/200000 (1.7%) false negatives before the fix, 0/200000 after.
  • Full cglm.h still compiles clean with -Wall -Wextra.

Diff is minimal: 2 files, 5 insertions / 5 deletions, no behavior change to the in-range or above-max cases.

…rcle test

glm_aabb_sphere() and glm_aabb2d_circle() pick which box face to
measure axis distance to via `!(a - 1)`, where
`a = (s[i] < box[0][i]) + (s[i] > box[1][i])` is 1 for either
out-of-range direction. `!(a - 1)` always resolves to index 1
(the max face) whenever a == 1, so when the query point is below
the box's min bound on an axis, distance is measured to the wrong
(max) face instead of the min face, inflating dmin and producing
false-negative intersection results.

The above-max direction happens to pick the correct face already,
which is why this went unnoticed since the index-selection logic
was introduced in recp#180 (fix for recp#179).

Fix: index directly with the "above max" boolean
(s[i] > box[1][i]) instead of the derived !(a - 1) expression --
it's 0 for below-min (selects box[0], correct) and 1 for
above-max (selects box[1], correct), and is already computed as
part of `a`/`b`/`c` so no extra branching is introduced. Same fix
applied to both the 3D (box.h) and 2D (aabb2d.h) variants since
they share the identical pattern.

Verified against a hand-written Ericson/Graphics-Gems reference
point-AABB distance test: 200k randomized box/sphere fuzz cases
show 3418/200000 false negatives before this fix and 0/200000
after, with no change to the already-correct above-max and
fully-enclosed cases.
@recp
recp merged commit b7813d0 into recp:master Jul 18, 2026
130 of 188 checks passed
@recp

recp commented Jul 18, 2026

Copy link
Copy Markdown
Owner

@94xhn the PR is merged, thanks for your contributions 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants