While we allow file formats and table providers to be plugged in via Python, we don't have a layer for catalogs. We should add the ability for this!
class MyCatalogProvider(CatalogProvider):
def list_catalogs(self):
...This stack would mirror the CatalogProviderList / CatalogProvider / SchemaProvider stack from DataFusion, although we should arguably make it arbitrarily nestable to support catalogs that allow that.
This is in preparation for supporting providers like Iceberg (REST catalog) that have a catalog-like hierarchy built in. It will also help with things like Overture that benefit from a metadata cache with very specific parameters / schemas that are expensive to compute but are mostly known ahead of time.
While we allow file formats and table providers to be plugged in via Python, we don't have a layer for catalogs. We should add the ability for this!
This stack would mirror the CatalogProviderList / CatalogProvider / SchemaProvider stack from DataFusion, although we should arguably make it arbitrarily nestable to support catalogs that allow that.
This is in preparation for supporting providers like Iceberg (REST catalog) that have a catalog-like hierarchy built in. It will also help with things like Overture that benefit from a metadata cache with very specific parameters / schemas that are expensive to compute but are mostly known ahead of time.