Facet searches appear to be broken

[nocollier]

At this time, we are getting 400 Bad requests for facet searches:

https://esgf-node.ornl.gov/esgf-1-5-bridge/?limit=0&facets=source_id

This is breaking Metagrid which depends heavily on this feature. Other types of searches appear fine.

https://esgf-node.ornl.gov/esgf-1-5-bridge/?limit=1&source_id=CESM2

This makes me wonder if something in Globus backend has changed and our facet queries are no longer properly formed?

@sturoscy-personal @sashakames @bstrdsmkr

[jpnavarro]

Can we get payload that esgf-node is posting to Globus Search at https://search.api.globus.org/v1/index/a8ef4320-9e5a-4793-837b-c45161ca1845/search?

[jpnavarro]

Globus Search API call that reproduces the problem, and a fixed variant with size=1000000.

fail11.sh
fail11fix.sh

[lliming]

Globus has completed a hotfix release that solved the problem in production operation.

Currently, Globus Search API is enforcing a "soft limit" of 5000 for individual facet sizes. This means that regardless of what a client requests for a facet size, you'll never get back more than 5000. (This soft limit has been in place for a while now: at least a couple of months.)

They've also (temporarily) restored very high "hard limits" for the facet size requests. (Higher than 2 billion for individual facets, and ridiculously high for total across all facets.) If a client explicitly requests a facet size greater than the temporarily-high hard limits, the client will get a 4** error on the API request.

Globus is requesting that we change esg_fastapi to replace the current facet size request of 2000000000 (2 billion) to 5000. Because there's already a soft limit of 5000, this will not reduce the behavior of the search response in practice.

So, please keep this issue open until a PR is merged that has the requested facet size changed to 5000. (Or something else is proposed and agreed to?)

[nocollier]

#147 illustrates how to change these limits, it may have been an earlier iteration on what number to put. I can click 'merge' (it passes tests) but I am not sure how to make sure this will fix Metagrid problems or deploy the new version. @jpnavarro RE: slack question in esgf-1-5 on Dec 4.

[bstrdsmkr]

Testing endpoint setup with this change at https://test-globus-facet-limit-esgf-fastapi.apps.onyx.ccs.ornl.gov/esgf-1-5-bridge

[nocollier]

To be clear, I had pushed that before they asked for it to be lowered than my PR. We should test it at 5_000

[bstrdsmkr]

To be clear, I had pushed that before they asked for it to be lowered than my PR. We should test it at 5_000

Doh! Good catch @nocollier -- if you want to push an update to 5k, I'll update the testing deployment

I think @sashakames and @downiec are good w/ it on the Metagrid side (please speak up if not!), so I'd like to get this in front of the other client teams to be sure it doesn't cause them any trouble.

I remember there was a technical reason we waived off a similar change before (aside from the idea of being clear what users should expect), but I can't for the life of me find it in my notes now...