Skip to content

Commit e3163d4

Browse files
authored
feat: Support lazy per-file Parquet Arrow schema derivation (#25343)
## Which issue does this PR close? Closes #25251. ## Rationale for this change Readers such as Comet need each file's physical Parquet schema to choose Arrow types for nested fields, but that schema is only available after loading the footer. ## What changes are included in this PR? Add an optional `ParquetFileSchemaProvider` to `ParquetSource`, invoked after footer loading and before Arrow schema inference. It supplies a complete file schema through Arrow's existing `with_schema` API while retaining the original metadata. Explicit `PartitionedFile.arrow_schema` takes precedence. ## What is the testing strategy for this PR? Regression tests cover differing nested layouts, ENUM/BINARY handling, cached and encrypted scans, schema precedence and validation, and serialization. Extended workspace tests and lint checks pass. ## Are there any user-facing changes? Adds `ParquetSource::with_schema_provider`. Sources using a provider require a custom codec for plan serialization. Default scans are unchanged.
1 parent 140c7c5 commit e3163d4

4 files changed

Lines changed: 473 additions & 11 deletions

File tree

0 commit comments

Comments
 (0)