Develop harvest for DCAT-US3.0 classes/objects (other than dataset)

User Story

In order to harvest DCAT-US3.0 metadata, OMB wants all new types of metadata supported.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

GIVEN a valid complete example (https://github.com/GSA/dcat-us/blob/main/jsonschema/examples/Catalog/good/complete_example.json) is available
WHEN a harvest of the source occurs
THEN all relevant metadata is saved and stored in the harvest database

Background

Ideally #5999 is already complete. Validation of datasets is already setup.
This has been broken down into sub-tickets to apply the necessary harvest of the various metadata components into the right areas.

service (DataService class): similar to Dataset but somewhat unique, providing information about a data service.
series (DataSeries class): similar to Dataset but somewhat unique, providing information about a "collection" of datasets that are related in a series (possibly geographic, or time based such as annual releases, etc).
record (CatalogRecord class): list of records related to the catalog.
catalog (Catalog class): list of other catalogs embedded in this catalog. Yes it's weird. Don't expect it to be highly utilized, but it does exist.

Security Considerations (required)

None

Sketch

This parent ticket is only concerned with handling data ingestion, ie into the database. Future integration into OpenSearch for searching/filtering and data display will be completed later on an as-needed basis for each metadata class.
Each ticket will consider the following:

How can I capture this list or object? How is this similar to dataset? Different? Documentation is at https://resources.data.gov/standards/catalog/dcat-us-3/dataset/
Provide test cases for the objects, both "typical" and "complete". These can be seen at https://github.com/GSA/dcat-us/tree/main/jsonschema/examples
Utilize new record_type field, or new catalog_json field where necessary (see Develop harvest for DCAT-US3.0 classes/objects (other than dataset) #6000 (comment) for sketch if not yet implemented)

the minimum db change we should need in order to persist all of these objects should be as follows.

catalog_json would store the entire catalog JSON file fetched from the source (not to be confused with a catalog which can exist within that. more to be discussed on that later)
record_type indicates what type of dcatus "object" is being stored. until now, we've only ever done dataset. but with dcatus 3.0 we need to process 4 different ones. our harvest record table is a change log and has, fortunately, been designed in a way which accommodates these new "objects" but we need to label what type of "object" they are in anticipation of further processing (instead of guessing what they are)
ix_harvest_record_source_type_identifier_created_success index would need to be updated to accept record_type which means we need to update how we calculate latest_set_of_successful_records to handle record_type. why should a Dataset with the same identifier conflict with a DataService?

# in harvest job
dcatus_catalog = Column(JSONB)

# in harvest record
record_type = Column(
        Enum(
            "catalog_record",
            "dataset",
            "data_series",
            "data_service",
            name="type",
        ),
        index=True,
        nullable=False,
    )

# in harvest record
Index(
    "ix_harvest_record_source_type_identifier_created_success",
    harvest_source_id,
    record_type,
    identifier,
    date_created.desc(),
    postgresql_where=text("status = 'success'"),
    postgresql_include=["action"],
)

Notes

the storing of catalog in "dcatus_catalog" could occur at the end where all processed "objects" (e.g. dataset, data_series, catalog_record) are removed from it. basically, the stored catalog would be a shallow copy with all "objects" removed beforehand

Before closing this want to re-run complete example in dev and verify working as expected.

@jbrown-xentity

@jbrown-xentity are you verifying this or is @Aguiardavidm planning to do it?

@SueValente

@SueValente I got this one...

We have the following example... https://github.com/GSA/dcat-us/blob/835115bbd131959d91a8a4766031d9a08de974ca/jsonschema/examples/Catalog/good/complete_example.json
I would expect there to be 6 harvest records total.
See latest job on the dev harvest source. 5 updated, 1 added, total of 6. I believe this is complete.

Navigation Menu

Repository navigation

Develop harvest for DCAT-US3.0 classes/objects (other than dataset) #6000

User Story

Acceptance Criteria

Background

Security Considerations (required)

Sketch

Notes

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Develop harvest for DCAT-US3.0 classes/objects (other than dataset) #6000

Description

User Story

Acceptance Criteria

Background

Security Considerations (required)

Sketch

Activity

rshewitt commented on Jul 28, 2026

Notes

jbrown-xentity commented on Aug 14, 2026

SueValente commented on Aug 17, 2026

jbrown-xentity commented on Aug 17, 2026

jbrown-xentity commented on Aug 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions