Repository navigation
Commit e3163d4
authored
feat: Support lazy per-file Parquet Arrow schema derivation (#25343)
## Which issue does this PR close?
Closes #25251.
## Rationale for this change
Readers such as Comet need each file's physical Parquet schema to choose
Arrow types for nested fields, but that schema is only available after
loading the footer.
## What changes are included in this PR?
Add an optional `ParquetFileSchemaProvider` to `ParquetSource`, invoked
after footer loading and before Arrow schema inference. It supplies a
complete file schema through Arrow's existing `with_schema` API while
retaining the original metadata. Explicit `PartitionedFile.arrow_schema`
takes precedence.
## What is the testing strategy for this PR?
Regression tests cover differing nested layouts, ENUM/BINARY handling,
cached and encrypted scans, schema precedence and validation, and
serialization. Extended workspace tests and lint checks pass.
## Are there any user-facing changes?
Adds `ParquetSource::with_schema_provider`. Sources using a provider
require a custom codec for plan serialization. Default scans are
unchanged.1 parent 140c7c5 commit e3163d4
4 files changed
Lines changed: 473 additions & 11 deletions
File tree
- datafusion
- datasource-parquet/src
- opener
- proto/tests/cases/plans
0 commit comments