Skip to content

update harvester nginx client_max_body_size size to 10mb #6080

Closed
GSA/datagov-harvester
#711

Description

@rshewitt

harvester proxy throws 413 Request Entity Too Large when uploading a file larger than what it accepts (nginx default client_max_body_size is 1mb).

How to reproduce

upload a json file larger than 1mb. here's a source we harvest that's 20mb

Expected behavior

as long as the uploaded file is <= 10mb the validator should process it normally

Actual behavior

nginx complains the file is too big

Sketch

set max body size for validate route within server block

    location /validate {
        client_max_body_size 10M; 
    }

Activity

  1. moved this to 📟 Sprint Backlog in data.gov team boardon Jun 22, 2026
  2. FuhuXia commented on Jun 22, 2026

    @FuhuXia
    Member
  3. jaredb96 commented on Jun 24, 2026

    @jaredb96

    Tried uploading the large JSON on my local setup with no nginx proxy, it gave the same error for being too large. I think even if the nginx proxy accepts the large file the actual flask app will still reject it / throw the same error.

  4. self-assigned this
    on Jun 24, 2026
  5. rshewitt commented on Jun 24, 2026

    @rshewitt
    ContributorAuthor

    yeah we want to have 2 lines of defense for file uploads (proxy & app)

  6. linked a pull request that will close this issueAdd upload limit increase for nginx proxy. #711on Jun 24, 2026
  7. moved this from 📟 Sprint Backlog to 👀 Needs Review in data.gov team boardon Jun 24, 2026
  8. moved this from 👀 Needs Review to ✔ Done in data.gov team boardon Jun 25, 2026
  9. JessieD-SSA commented on Jun 26, 2026

    @JessieD-SSA

    Hi team - FYI - I just tested this again with the ssa.gov/data.json file I had shared earlier in the week. Uploading the JSON file works normally now but I still get an error if I try to copy/paste it into the page:
    {"detail":{},"message":"Request Entity Too Large"}

    It's a 6MB file.

  10. moved this from ✔ Done to 🗄 Closed in data.gov team boardon Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSoftware defect or bug

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions