Skip to content

Commit 454ca08

Browse files
authored
fix(spark): derive concat_ws nullability from separator (#26205)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> Part of the work towards #15914 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> `concat_ws` return value is always nullable in DF, which is incorrect. Per upstream Spark implementation, this is only true when separator itself is null. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> Derive the nullability based on the separator nullability. ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> Add SLT tests for schema. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> I believe this doesn't count as an API change but would like guidance from a committer.
1 parent 75ddead commit 454ca08

2 files changed

Lines changed: 31 additions & 1 deletion

File tree

‎datafusion/spark/src/function/string/concat_ws.rs‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,11 @@ use arrow::array::{
3434
StringArray, StringBuilder, StringViewArray,
3535
};
3636
use arrow::datatypes::{DataType, Field};
37+
use datafusion_common::arrow::datatypes::FieldRef;
3738
use datafusion_common::{Result, ScalarValue};
3839
use datafusion_expr::{
39-
ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
40+
ColumnarValue, ReturnFieldArgs, ScalarFunctionArgs, ScalarUDFImpl, Signature,
41+
Volatility,
4042
};
4143

4244
use crate::function::error_utils::{
@@ -75,6 +77,17 @@ impl ScalarUDFImpl for SparkConcatWs {
7577
Ok(DataType::Utf8)
7678
}
7779

80+
fn return_field_from_args(&self, args: ReturnFieldArgs<'_>) -> Result<FieldRef> {
81+
// Spark skips null values after the separator, so only the separator controls nullability:
82+
// https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L94-L96
83+
let separator = &args.arg_fields[0];
84+
Ok(Arc::new(Field::new(
85+
self.name(),
86+
DataType::Utf8,
87+
separator.is_nullable(),
88+
)))
89+
}
90+
7891
fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
7992
if arg_types.is_empty() {
8093
return Err(invalid_arg_count_exec_err("concat_ws", (1, i32::MAX), 0));

‎datafusion/sqllogictest/test_files/spark/string/concat_ws.slt‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,11 @@ SELECT concat_ws(',');
5353
----
5454
(empty)
5555

56+
query TTT
57+
DESCRIBE SELECT concat_ws(',') AS result;
58+
----
59+
result Utf8 NO
60+
5661
## Null separator returns null
5762
query T
5863
SELECT concat_ws(NULL, 'a', 'b', 'c');
@@ -113,6 +118,12 @@ SELECT concat_ws(sep) AS result FROM VALUES (CAST(NULL AS STRING)), (',') AS t(s
113118
NULL
114119
(empty)
115120

121+
query TTT
122+
DESCRIBE SELECT concat_ws(sep) AS result
123+
FROM VALUES (CAST(NULL AS STRING)), (',') AS t(sep);
124+
----
125+
result Utf8 YES
126+
116127
## ── Column expressions ──────────────────────────────────────
117128

118129
## concat_ws on columns
@@ -130,6 +141,12 @@ a,b
130141
c
131142
d
132143

144+
query TTT
145+
DESCRIBE SELECT concat_ws(',', a, b) AS result
146+
FROM VALUES ('a', 'b'), ('c', CAST(NULL AS STRING)), (CAST(NULL AS STRING), 'd') AS t(a, b);
147+
----
148+
result Utf8 NO
149+
133150
## Scalar-only arguments over multiple rows (broadcast test)
134151
query T
135152
SELECT concat_ws(',', 'a', 'b') AS result FROM VALUES (1), (2), (3) AS t(x);

0 commit comments

Comments
 (0)