Support harvesting DCAT-US3 Catalog objects #6180

User Story

In order to capture catalog-level DCAT-US3.0 metadata, OMB wants Catalog objects harvested and retained for later use.

Acceptance Criteria

• GIVEN a valid DCAT-US3 Catalog example is present in the source metadata, WHEN a harvest runs, THEN Catalog top level fields are detected and persisted in the harvest database on the job table.
• GIVEN a harvested Catalog object contains the catalog field, WHEN a harvest runs, THEN the datasets within this sub Catalog (or sub-sub Catalog, nested however deep).

Background

This story splits out the Catalog portion of #6000. The parent issue suggests catalog-level metadata may be stored as JSON on the harvest job. There is no intention of this metadata being searchable or discoverable in catalog or openSearch at this time, but it may at some point be useful for data providers in the harvest application.

Reference material:

• Develop harvest for DCAT-US3.0 classes/objects (other than dataset) #6000
• https://github.com/GSA/dcat-us/blob/main/jsonschema/examples/Catalog/good/complete_example.json

Security Considerations (required)

None.

Sketch

• Clean the catalog metadata by removing any fields that we harvest directly (catalogRecord, Dataset, DataService). If catalog has other catalogs, filter the same field out of those as well. See schema for reference of what this looks like.
• Update harvest extraction to keep catalog-level metadata even when dataset lists are absent. This will require a schema change on the job to allow to store this info. See Develop harvest for DCAT-US3.0 classes/objects (other than dataset) #6000 (comment), expect this to be dcatus_catalog field name with JSONB type.
• If catalog listing is present, recursively process data fields (dataset, etc). Prefer to not start a new "job", as the extract will have already occurred at this point. Suggest reviewing the code here as the starting point.