Proposal: Standardize case-sensitive column resolution in Gluten

[sabbasani]

Proposal: Standardize case-sensitive column resolution in Gluten

Hi all,

While working on PR #12726, I found that some Gluten components use toLowerCase() when comparing or normalizing column names.

This can cause incorrect behavior when:

spark.sql.caseSensitive=true

For example, Spark should treat these as different columns:

Input_File_Name
input_file_name

Proposed approach

To keep Gluten consistent with Spark:

• Use ConverterUtils.normalizeColName for column-name normalization.
• Use SQLConf.get.resolver for column-name comparison.
• Avoid unconditional toLowerCase() when resolving Spark columns.

PR #12726 applies this approach to Iceberg scans and input_file_name() handling.

The change mainly affects scan, metadata-column, and input-file-expression handling, with related updates for Velox, ClickHouse, Iceberg, and Delta paths.

Would it make sense to use this as the standard approach across Gluten when handling Spark column names?

Feedback from Velox, ClickHouse, and Iceberg maintainers would be appreciated.

Related PR: #12726

[philo-he]

Thanks for the proposal.

Curious whether this is a real requirement on your side. My understanding is that most Spark users set caseSensitive to false.
The proposed PR makes column resolution respect the case-sensitivity config. As long as there's no side effect on the regular paths and it doesn't complicate the regular paths, I think it's fine to do this.

[sabbasani]

Regarding the impact on regular paths:

• No behavior change for the default caseSensitive=false: ConverterUtils.normalizeColName and SQLConf.get.resolver preserve Spark’s existing case-insensitive behavior, so the regular path remains unchanged.
• Cleaner resolution logic: The PR replaces hardcoded toLowerCase / equalsIgnoreCase checks with Spark’s configuration-aware resolution utilities, keeping the implementation consistent with Spark’s identifier resolution model.
• No additional complexity: There is no separate case-sensitive path added to the regular flow; the existing resolver determines the behavior based on Spark’s configuration.
• Testing: The test suites pass for both caseSensitive=false and caseSensitive=true, with style/format checks also passing.

This should allow the existing default behavior to remain intact while supporting environments that require exact case-sensitive column resolution.

[philo-he]

Sounds good. Thanks.