Support harvesting DCAT-US3 DatasetSeries objects #6179

User Story

In order to harvest DCAT-US3.0 metadata from dataset sub-structures, OMB wants DatasetSeries objects supported and the relationship with the sub Datasets saved.

Acceptance Criteria

• GIVEN a valid DCAT-US3 DatasetSeries example is present inside a Catalog payload, WHEN a harvest runs, THEN DatasetSeries objects are detected and persisted in the harvest database.
• GIVEN a harvested DatasetSeries object contains supported fields, WHEN the harvested record is stored, THEN the selected fields are available for downstream processing without being silently dropped.
• GIVEN a source contains Dataset objects within DatasetSeries entries, WHEN the harvest runs, THEN DatasetSeries objects are handled without breaking Dataset harvesting.

Background

The DatasetSeries replaces the isPartOf functionality of DCAT-US1.1. That just allowed a string, and that string defined the parent by referencing the ID.
In DCAT-US3, a DatasetSeries is defined at the catalog level, and has a list of Datasets. Also DatasetSeries doesn't have an identifier field, but does have an @id field that is recommended. DatasetSeries has much less information than a Dataset.
We need to be able to save and/or replicate the relationship information in the same way that 1.1 does, and make DatasetSeries objects displayable and searchable.

Security Considerations (required)

None.

Sketch

Assuming this guidance holds, we will need to do the following:

• Ingest any DatasetSeries objects defined in a catalog under the datasetSeries field. If there are any datasets within the datasetSeries, harvest those objects as well as normal datasets.
• Maintain the link between the datasets in the harvest system such that the relationship isn't lost. Some code references for how we do this in the 1.1 world:
  • adding a parent
  • tracking the parent in DB
  • manually adding/tracking for ISO in DB
  • parent utilization in catalog

Please note that while the link in DCAT-US1 is simple and contained within the metadata, it's more complex in the ISO world, and is defined by the harvest source and manually propagated in the harvest process. The DCAT-US3 will be a marriage between the 2; the metadata itself will drive the relationships but since we want to maintain dataset and datasetSeries as different objects and the metadata doesn't support simple ID linking like DCAT-US1, we need to store the relationship in the DB and/or openSearch outside of the DCAT object itself.