Fixes #25978: correct nullability of AND/OR expressions to resolve plan mismatch in CSE

[siddubakka]

Fixes #25978

This PR fixes an internal error where the physical input schema mismatches the logical input schema due to incorrect nullability propagation for AND/OR expressions in CommonSubexprEliminate.

It adds robust contains_is_not_null and contains_is_null helper functions to accurately determine if binary expressions are nullable.

cc @alamb @jayzhan211

[alamb]

thanks @siddubakka

[alamb]

please write this as a slt test

As it is, this test only verifies the query doesn't error -- using slt you can aso verify its correctness

[siddubakka]

makes sense, switched it to slt!

[alamb]

this feels like treating the symptom rather than the root cause -- why would we need to special case IS NOT NULL as part of AND / OR ? Wouldn't it make more sense to perhaps set nullability for IsNotNull itself?

[siddubakka]

yeah fair point -- so IsNotNull already returns Ok(false) for nullable since it's always a bool. the thing is when the simplifier turns CASE WHEN x IS NOT NULL THEN x ELSE false END into x IS NOT NULL AND x, the AND sees that x is nullable and reports the whole thing as nullable. but semantically it can't be null -- if x is null then IS NOT NULL(x) is false and alse AND NULL = false. so the special-casing here is really about AND/OR knowing that an IS NOT NULL guard makes the other operand safe. happy to explore a different approach if you have something in mind though!

[alamb]

Maybe we can add some more comments explaining why we need to explicitly check IS NOT NULL

ALso, the example you gave above is only valid when the argument (x) is on both sides

x IS NOT NULL AND x

However it doesn't hold for:

y IS NOT NULL and x (different arguments).

This seems very similar to something @pepijnve hit a while ago with Case nullability -- specifically that the after simplification the nullability can sometimes be smarter than before the simplification.

I am still missing why we need to make the nullability check smarter (why isn't leaving it as nullable ok, even if overly conservative)?

[alamb]

is this tested anywhere? How is it related to the other fix?

[siddubakka]

yeah sorry about that, it got mixed in from another branch. removed it now.

[siddubakka]

hey @alamb, pushed the updates -- switched to an slt test, dropped the unrelated join_selection changes, and rebased on main. should be a clean single commit now. let me know if anything else needs changing!

[Ishaan400]

Thanks for working on this, @siddubakka! I can see the motivation: once the simplifier turns CASE WHEN e IS NOT NULL THEN e ELSE false END into e IS NOT NULL AND e, the AND is reported as nullable even though, for a deterministic e, it can't be NULL.

I found two problems with the current approach, though.

1. The recursive search through AND/OR chains produces wrong results. A guard deeper in the chain doesn't make the whole expression non-nullable when other operands are nullable. y AND x IS NOT NULL AND x with y = NULL, x = true evaluates to (NULL AND true) AND true = NULL, but nullable() now returns false, so the simplifier rewrites (...) IS NULL to false:

CREATE TABLE t(id INT, y BOOLEAN, x BOOLEAN) AS VALUES
  (1, NULL, true), (2, true, true), (3, NULL, NULL), (4, false, false), (5, NULL, false);

SELECT id, y AND x IS NOT NULL AND x, (y AND x IS NOT NULL AND x) IS NULL FROM t ORDER BY id;
-- expected: 1 NULL true | 2 true false | 3 false false | 4 false false | 5 false false

SELECT count(*) FROM t WHERE (y AND x IS NOT NULL AND x) IS NULL;
-- expected: 1

SELECT id, (y OR x IS NULL OR x) IS NULL FROM t ORDER BY id;
-- expected: only row 5 is true

With this PR, row 1 returns false in the first query, the count is 0, and row 5 is false in the OR version. All three are correct on main.

A smaller point: even the direct pattern only holds for deterministic operands. With v = CASE WHEN random() < 0.5 THEN NULL ELSE true END, the two copies are evaluated independently, and WHERE (v IS NOT NULL AND v) IS NULL returns no rows with this PR, versus some rows on main.

2. I don't think this change fixes #25978. The error is field nullability at index 1 [f]: (physical) true vs (logical) false, and the physical side comes from the physical BinaryExpr::nullable, which this PR doesn't change. Locally:

• On 16ecdc9bd, where the issue's query still fails, applying this PR's expr_schema.rs change gives the same error.
• On current main, the issue's query passes without this change. Bisecting points to fix: preserve short-circuit evaluation in Boolean CASE simplification #25143 (587e74c84), which stopped rewriting this CASE into AND/OR. Reverting fix: preserve short-circuit evaluation in Boolean CASE simplification #25143 on main brings the error back, and this PR's change doesn't remove it.
• The new issue_25978.slt passes on both of those failing commits even without the fix. There, the coalesce is above the UNION rather than inside a branch under the GROUP BY.

On @alamb's question, I think leaving AND/OR conservatively nullable is fine. This looks like the case he mentioned, where simplification changes an expression's nullability after the plan schema is fixed: the CASE is non-nullable, and the AND it was rewritten into is nullable.

One possible path forward would be to add a regression test using the issue's original query, since it fails before #25143 and passes on current main. That would preserve coverage for the original bug without relying on this nullability change.

Happy to share the reproduction files.

[siddubakka]

thanks @alamb and @Ishaan400! you're totally right, the expr_schema changes were just fixing the symptom and causing those correctness issues with AND/OR chains. i've reverted the expr_schema.rs changes completely.

instead i just added the original issue's query as an slt test so we keep coverage for the fix that was already merged in #25143.