Scope and document CSW harvest source type

What: Some federal agencies publish ISO 19115 records via CSW (Catalog Service for the Web) endpoints rather than WAFs. CSW is an OGC-standard query protocol that supports filtered harvest by date, extent, and keyword. The harvester currently has no CSW source type.

This ticket scopes the CSW source type -- it does not build it. The output is a technical design document or GitHub epic that can be estimated and scheduled.

Scope the following:

Identify which agencies currently have active CSW endpoints that would benefit from a Data.gov harvest path (NOAA NCEI, USGS, EPA, NGA are likely candidates)
Document the CSW GetCapabilities / GetRecords / GetRecordById interaction pattern
Define the CSW source type schema: endpoint URL, filter parameters, output format, max records per page, authentication requirements
Map CSW pagination to the WAF-like modified_date filtering the harvester already uses
Identify whether existing Python CSW client libraries (e.g., OWSLib) are suitable or whether custom logic is needed
Estimate implementation effort

Done When:

A scoping document is produced and attached to this ticket
Known CSW endpoints and agencies are listed
Effort estimate is produced
A follow-on implementation epic is written if the work is approved

The CSW harvest type in CKAN was broken before we transitioned. It was not maintained upstream, and the data.gov team had neither the resources nor expertise to fix it.

I found this sheet, but it doesn't contain the actual source URL. Even back in 2020, we only had 7 CSW sources, and the implementation was mixed.

My recommendation would be to engage with FGDC and the geospatial community on their favorite/best practices on making metadata available, and consider collaboration on a new harvest type other than WAF, as that is a very old way to transfer files and make them public.

here's some geospatial endpoints we could consider for harvesting

Endpoint / ecosystem	What you would harvest	chris notes
ArcGIS Hub DCAT / OGC API Records	Hub catalog items + layers	Data.gov is already harvesting ArcGIS Hub DCAT endpoints. Example: https://harvest.data.gov/harvest_source/a248778a-3468-4c45-99de-1abb590e86d0. Esri is working to support DCAT-US 3.0. Metadata completeness depends on the underlying ArcGIS items.
ArcGIS Hub search API	Provides catalog items and layers through a REST API.	Its collection/item structure resembles STAC API and OGC API – Records, but it is not necessarily compliant with either standard. Example Hub: https://disasters-geoplatform.hub.arcgis.com/search REST API: https://disasters-geoplatform.hub.arcgis.com/api/search/v1/collections/all In GeoPlatform's STAC Browser: https://stac.geoplatform.gov/#/external/disasters-geoplatform.hub.arcgis.com/api/search/v1/collections/all
ArcGIS Portal / ArcGIS Online REST	Portal content items and underlying services	Could be used to harvest Portal content items and inspect their underlying services. This would be an Esri-specific integration rather than an open catalog standard. Portal items may also include or export ISO metadata if it was authored in ArcGIS.
OGC API – Records	Catalog metadata records	Modern successor to CSW. Part 1: Core 1.0 was published in May 2025, so it is still new and not widely deployed. Uses modern REST, JSON, HTML and OpenAPI conventions. This is the better strategic choice for new catalog implementations, but there may not yet be many federal production endpoints to harvest.
CSW	ISO/FGDC/etc. catalog records	Has a much larger installed base, especially in government and scientific spatial-data infrastructures. 2007: CSW 2.0.2 and the ISO Metadata Application Profile. 2016: Catalogue Services 3.0 General Model and HTTP binding. CSW 2.0.2 remains more widely implemented than CSW 3.0. My thought is that catalogs using CSW 2.0.2 should consider OGC API – Records as the future direction.
STAC / STAC API	Collections, Items, assets	Designed for spatiotemporal assets such as satellite imagery and derived products. STAC API supports spatial, temporal and property searches—for example, finding imagery over an area with less than 10% cloud cover. A catalog may contain millions of STAC Items. Harvesting STAC Collections would likely be valuable. Indexing every Item could create an enormous number of records and duplicate STAC API functionality. GeoPlatform Voyager STAC item example: https://sit-nsdi.geoplatform.info/nsdi-search?filters=format%3A%7C%3Astac+item&settings=any
THREDDS	Scientific dataset catalog + OPeNDAP/WMS/WCS access	Provides a catalog of scientific datasets and links to OPeNDAP, WMS, WCS and other access services. When ncISO is enabled, THREDDS generates ISO 19115/19115-2 XML from NetCDF CF and ACDD metadata. Data.gov could crawl a THREDDS catalog and retrieve an ISO record for each dataset. Record completeness depends on the quality of the source NetCDF attributes.
ERDDAP	Scientific/environmental dataset inventory	Automatically generates an ISO 19115 record for each dataset. Provides machine-readable ways to enumerate its dataset inventory. NOAA operates several ERDDAP installations, making this a potentially useful federal source. OPeNDAP, tabledap and griddap URLs could be retained as distributions in the harvested record.
OPeNDAP	Primarily a data-access protocol, not a discovery catalog.	A bare OPeNDAP endpoint exposes variables, dimensions and attributes, but generally does not produce an ISO discovery record. OPeNDAP URLs would normally appear as distributions within ISO or DCAT-US records generated by THREDDS, ERDDAP or another metadata system.
OGC API – Features	Feature collections	OGC API – Features / WFS: feature collections and feature types. OGC API – Coverages / WCS: raster or coverage collections. OGC API – Maps / WMS / WMTS: maps, layers and tiles. OGC API – EDR: environmental data collections. SensorThings API: Things, Sensors, ObservedProperties and Datastreams. These are primarily data-access services rather than metadata catalogs. Data.gov could create collection- or service-level discovery records from them, but you would not want to harvest every feature, cell, tile or observation. The newer OGC APIs use common modern web patterns. The legacy WFS, WCS, WMS and WMTS services predate OGC API – Common. }
WFS	Feature types
WCS	Raster/coverage collections
WMS / WMTS	Map layers
SensorThings / OGC EDR	Datastreams/environmental collections

ERDDAP endpoints (a number are harvested into Data.gov via the WAF):

NOAA NCEI
https://www.ncei.noaa.gov/erddap/
Currently lists more than 100 datasets. NCEI dataset inventory

NOAA CoastWatch West Coast / Southwest Fisheries Science Center
https://coastwatch.pfeg.noaa.gov/erddap/
This is also the reference production installation cited by the ERDDAP project. ERDDAP project

NOAA Pacific Marine Environmental Laboratory
https://data.pmel.noaa.gov/pmel/erddap/
Includes ocean, atmosphere, Arctic and research-cruise data. PMEL ERDDAP

NOAA Integrated Ocean Observing System
https://erddap.ioos.us/erddap/
NOAA IOOS national data services; distinct from the regional IOOS association servers. IOOS ERDDAP

NOAA PolarWatch
https://polarwatch.noaa.gov/erddap/
Polar satellite and environmental datasets.

NOAA Pacific Islands Ocean Observing System/OceanWatch
https://oceanwatch.pifsc.noaa.gov/erddap/
NOAA Pacific Islands Fisheries Science Center datasets.

A related federally funded—but not directly federal-agency-operated—example is:

NSF Ocean Observatories Initiative
OOI exposes roughly 600 datasets through ERDDAP.
It is an NSF-funded research infrastructure operated by a consortium. OOI ERDDAP information

THREDDS endpoints (a number are harvested into Data.gov via the WAF):

NOAA Physical Sciences Laboratory
Human catalog: https://psl.noaa.gov/thredds/catalog.html
XML catalog: https://psl.noaa.gov/thredds/catalog.xml
Includes climate, reanalysis, observational and research datasets. PSL THREDDS

NOAA National Data Buoy Center
Human catalog: https://dods.ndbc.noaa.gov/thredds/catalog.html
XML catalog: https://dods.ndbc.noaa.gov/thredds/catalog.xml
Includes buoy, OceanSITES and HF-radar surface-current data. NDBC THREDDS

NOAA Center for Operational Oceanographic Products and Services
Human catalog: https://opendap.co-ops.nos.noaa.gov/thredds/catalog.html
XML catalog: https://opendap.co-ops.nos.noaa.gov/thredds/catalog.xml
Includes operational ocean forecast-system products such as DBOFS, GOMOFS and NYOFS. CO-OPS THREDDS

NOAA Atlantic Oceanographic and Meteorological Laboratory
Human catalog: https://cwcgom.aoml.noaa.gov/thredds/catalog.html
XML catalog: https://cwcgom.aoml.noaa.gov/thredds/catalog.xml
Includes SST, chlorophyll, ocean acidification, currents, carbon-flux and other ocean products. AOML THREDDS

NOAA Pacific Islands OceanWatch
XML catalog: https://oceanwatch.pifsc.noaa.gov/thredds/catalog.xml
Includes satellite and oceanographic products for the Pacific Islands region.

i have an appointment today and will only make the first half of the meeting so we're postponing the discussion until next time (2 weeks)

FGDC recommendation is to "fast track already vetted [standards] by the standards development organization". ISO and CSDGM are still widely used and based on the meeting aren't going anywhere. There is awareness of the requirement to migrate to DCAT-US 3.0.

Navigation Menu

Repository navigation

Scope and document CSW harvest source type #6045

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Scope and document CSW harvest source type #6045

Description

Activity

jbrown-xentity commented on Sep 14, 2026

rshewitt commented on Sep 17, 2026

rshewitt commented on Sep 21, 2026

rshewitt commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions