Repository navigation
SPIKE-1: Confirm MDTranslator xlink resolution behavior #6037
Description
What: Many federal ISO 19139 records use xlink:href to reference external endpoints for contact information, coordinate reference system definitions, and responsible party data. The Census Bureau WAF records are a confirmed example -- contact, originator, and CRS are all xlink-referenced to NGDC (NOAA) docucomp endpoints rather than inline values.
The harvester's download_file() function fetches raw XML and passes it to MDTranslator. It is not known whether MDTranslator dereferences xlinks internally before parsing.
Investigate and document:
- Submit the Census concity ISO record and other test WAFs to the MDTranslator service and inspect the output:
- Reader:
iso19115_2_datagov - Writer:
dcat_us
- Reader:
- Does the output contain populated contact, publisher, and CRS fields, or are they empty?
- If empty: are they empty because xlinks were not resolved, or because MDTranslator does not map those fields at all?
- Check which fields in the output correspond to
fill_placeholdersdefaults vs. values derived from the source record - Test other ISO records that have inline (non-xlink) contact values to compare output
Done When:
- Findings documented with MDTranslator input/output for the test records
- Clear answer on whether xlinks are resolved
- If xlinks are not resolved, follow-up implementation ticket written (see IMPL-5)
Notes:
The NGDC xlink endpoints referenced in the Census record are:
- Contact:
https://www.ngdc.noaa.gov/docucomp/1df27e57-4768-42de-909b-52f530601fba - Originator:
https://www.ngdc.noaa.gov/docucomp/ddd21bfb-2229-465b-95b2-bee36200b0e5 - CRS (NAD83):
https://www.ngdc.noaa.gov/docucomp/65f8b220-95ed-11e0-aa80-0800200c9a66
These endpoints may or may not be publicly accessible; test reachability first.
Activity
rshewitt commentedon Jun 5, 2026
there's a function in mdtranslator called
convert_xlink_to_elemwhich fetches thexlink:hrefdoc, parses it as an xml element, and places it into the xml tree in its intended position. currently, this only occurs forgmd:contact//gmd:CI_ResponsiblePartyelements where thegmd:contactelement is a direct child of the root. We can add this functionality whereverxlink:hrefis supported. An important thing to consider is that every time we add this mdtranslator gets slower because it's making an http request, follows redirects, and does retries with a limit of 10.The reason to add this functionality is to pull more information from the input ISO doc which can result in more datasets on catalog or more enriched existing datasets.
cody-seibert-gsa commentedon Sep 21, 2026
I ran real Census content through the local
mdtranslatorservice, using the readeriso19115_2_datagovand the writerdcat_us, with no mocking — the xlink fetches actually hit the network. I tested three real ISO records this way: mdTranslator's own checked-in Census TIGER/Line fixture (tl_2013_72_sldu.shp.xml), which already contains all three xlink patterns named in this issue; the three NGDC URLs listed above spliced into an existing NOAA/NMFS fixture; and an unrelated EPA EnviroAtlas record with no xlinks at all, as a fully-inline baseline for comparison.Here's what came back:
contactPointgmd:contactxlink is followed for real.module_metadata_info.rbexplicitly callsconvert_xlink_to_elemon it, and mdTranslator's own upstream unit test (test_get_contact_xlink_href) already covers this exact path.publisherpublishercame from its owncitedResponsiblePartywithrole=publisherinstead, which shows the citation-to-publisher mapping itself works fine. It's specifically the xlink path into that same mechanism that never fires.citedResponsibleParty)module_responsibility.rbonly looks for an inlineCI_ResponsiblePartychild, and when it doesn't find one it returnsnilsilently, with no warning logged anywhere.referenceSystemInfo)dcat_usis the only one that never reads the internal:referenceSystemdata, even for readers likeiso19115_3that do parse it. DCAT-US itself has no field to hold a CRS value either way, so resolving this xlink wouldn't help even if someone implemented it.One more thing worth flagging is that the harvester's own
fill_placeholders()adds a third fallback layer on top of whatever MDTranslator returns. IfcontactPointorpublishercome back empty, the harvester fills them with a genericdatagovsupport@gsa.govaddress or the harvest source's registered organization name. That means an unresolved xlink doesn't show up as an obviously empty field in the catalog, since it gets masked by a placeholder that looks legitimate. CRS has no such fallback, because nothing in the pipeline touches it at any stage.My findings point toward scoping any fix narrowly to the originator field, since that's a small change that mirrors the existing contact-resolution pattern one level down in the citation. I'd leave CRS out of scope entirely, since resolving that xlink wouldn't do anything until someone first adds CRS support to the
dcat_uswriter, which is a separate and bigger piece of work.This maps directly onto the Option A/B/C decision already underway in #6043 — based on the content there, that looks like the implementation ticket this issue's Done When section points to as IMPL-5, so hopefully these findings help move that decision forward. Thanks @rshewitt for the earlier pointer to
convert_xlink_to_elem, which is what made tracing this down straightforward.rshewitt commentedon Sep 22, 2026
this is good investigation @cody-seibert-gsa and thanks for your writeup. I think @SueValente used AI to summarize where xlink attributes are most common in our xml sources (or maybe just one census waf source?) which showed "contact information, coordinate reference system definitions, and responsible party data" so that most likely explains why those specific things were listed in the ticket description.
this is a good exercise for finding out how we can pull out more information from our xml sources.
citedResponsiblePartybeing an example because we use it in our dcatus output but, as you noticed,referenceSystemInfoisn't used in the dcatus output so resolving an xlink in that data isn't worthwhile.we're planning on decommissioning mdtranslator so i'd hold off on all work for that service.
cody-seibert-gsa commentedon Sep 22, 2026
@rshewitt let me know if there are other things worth looking into on this, but I'll take my name off of it for now