Skip to content

harvester validator doesn't enforce file size for fetch via url #6081

Closed
GSA/datagov-harvester
#729

Description

@rshewitt

validator accepts fetching a dcatus document via url but should raise an exception when the file size of document exceeds 10mb. this source which is ~20mb can be fetched no problem. that's not the intended behavior.

How to reproduce

use the source above in the validator and fetch it via url

Expected behavior

error to be displayed on the page indicating the file is too big

Actual behavior

file is fetched and processed normally

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

Activity

  1. self-assigned this
    on Jun 29, 2026
  2. moved this from 📥 Queue (ready but not in sprint) to 🏗 In Progress in data.gov team boardon Jun 29, 2026
  3. linked a pull request that will close this issueAdd error when JSON file > 10mb. #729on Jun 29, 2026
  4. moved this from 🏗 In Progress to 📟 Sprint Backlog (Top Priority) in data.gov team boardon Jun 29, 2026
  5. moved this from 📟 Sprint Backlog (Top Priority) to 👀 Needs Review in data.gov team boardon Jun 29, 2026
  6. moved this from 👀 Needs Review to ✔ Done in data.gov team boardon Jul 2, 2026
  7. moved this from ✔ Done to 🗄 Closed in data.gov team boardon Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions