[spark] Fix empty projection causing Invalid metadata length for COUNT(*)/COUNT(1)#3227
Open
Kaixuan-Duan wants to merge 1 commit intoapache:mainfrom
Open
[spark] Fix empty projection causing Invalid metadata length for COUNT(*)/COUNT(1)#3227Kaixuan-Duan wants to merge 1 commit intoapache:mainfrom
Kaixuan-Duan wants to merge 1 commit intoapache:mainfrom
Conversation
Yohahaha
reviewed
Apr 30, 2026
| SchemaGetter schemaGetter, | ||
| ArrowCompressionInfo compressionInfo, | ||
| int[] selectedFieldPositions) { | ||
| // Empty projection (selectedFieldPositions.length == 0) is currently not supported on the |
Contributor
There was a problem hiding this comment.
It would be good to also verify this behavior and the fix in the Flink connector if needed.
Contributor
|
@Kaixuan-Duan Hi, seems is it same with #2725. cc @beryllw |
Yohahaha
reviewed
Apr 30, 2026
| } | ||
| } | ||
|
|
||
| test("Spark Read: COUNT(*) without filter") { |
Contributor
There was a problem hiding this comment.
What happens when there is a filter?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #2724
When Spark pushes down an empty column projection for
COUNT(*)/COUNT(1)queries, the Fluss server fails withIllegalStateException("Invalid metadata length")inFileLogProjection.project(), causing the client to retry indefinitely and the query to hang.This PR fixes the issue from two sides:
Server side: reject empty projection early with a clear
InvalidColumnProjectionExceptioninstead of crashing with an internal error.Spark connector side: fall back to projecting the first column when Spark pushes down an empty projection, so the row count is preserved without fetching unnecessary data.
Brief change log
FileLogProjection#setCurrentProjection: add a guard that throws InvalidColumnProjectionException when selectedFieldPositions is empty.FileLogProjectionTest: add testEmptyProjectionRejectsWithClearError to verify the server-side guard.FlussBatch#projection/FlussMicroBatchStream#projection: when readSchema yields an empty projection, fall back to Array(0) (first column).SparkLogTableReadTest: addCOUNT(*)andCOUNT(1)end-to-end tests for log tables.SparkPrimaryKeyTableReadTest: addCOUNT(*)end-to-end test for primary key tables.Tests
./mvnw -pl fluss-common -DskipTests=false -Dtest=FileLogProjectionTest#testEmptyProjectionRejectsWithClearError test
./mvnw -pl fluss-spark/fluss-spark-ut -am install -DskipTests
./mvnw -pl fluss-spark/fluss-spark-ut -Dsuites='org.apache.fluss.spark.SparkLogTableReadTest' test
./mvnw -pl fluss-spark/fluss-spark-ut -Dsuites='org.apache.fluss.spark.SparkPrimaryKeyTableReadTest' test
API and Format
Documentation