Document and decide xlink resolution approach #6043

What: Based on SPIKE findings, implement the agreed approach for handling xlinks in ISO records.

If Option A (resolve at harvest time):

• Add an xlink resolution step before MDTranslator is called
• For each xlink:href in the raw ISO XML, make an HTTP GET request to the referenced URL
• Replace the xlink element with the fetched content inline
• Handle failures gracefully: log, continue without the field, do not fail the record
• Add timeout and retry logic appropriate for external service calls
• Test with the Census NGDC endpoints

If Option B (pass through and accept gaps):

• Document the known gap in a harvester README or decision record
• Confirm that fill_placeholders handles all fields that would be empty due to unresolved xlinks
• Add a warning log when an unresolved xlink is detected

If Option C (require agencies to use inline values):

• Draft publisher guidance for resources.data.gov explaining the limitation
• Add a harvest warning (not error) when unresolved xlinks are detected in a record

Done When:

• PM has chosen an option
• Chosen option is implemented
• Decision is documented in the harvester repo

Notes:

• Option A adds latency per-record. For a WAF with hundreds of records this is significant. Consider whether xlink resolution should be cached across records in the same harvest run.
• The NGDC docucomp endpoints used by Census are the most common xlink targets in federal ISO records. Test specifically with these.
• Option C is only viable if agencies can actually update their ISO records -- not always within their control.