Skip to content

Fido client patch and QoL update - #124

Merged
MJWeberg merged 1 commit into
USNavalResearchLaboratory:mainfrom
MJWeberg:main
Sep 3, 2025
Merged

Fido client patch and QoL update#124
MJWeberg merged 1 commit into
USNavalResearchLaboratory:mainfrom
MJWeberg:main

Conversation

@MJWeberg

@MJWeberg MJWeberg commented Sep 3, 2025

Copy link
Copy Markdown
Collaborator

Summary of Changes

Patched some issues with the EISClient for SunPy's Fido and added a few quality-of-life improvements. Closes #120

  • Updated the search patterns to use the new parse-format added in SunPy v6.1 (following this migration guide).
  • Patched an issue with the client returning results when a different instrument is queried.
  • Added the "MSSL" data mirror for HDF5 files as an additional Provider. This should give better download speeds for European users.
  • Reenabled tests of the EISClient and added an automatic check for server issues (so the tests can be skipped as needed)
  • Updated the default behavior for a faster and smoother user experience (see below for details).
  • Updated doc strings and user guide

Updated EISClient Behavior

By default, the client only returns entries for the HDF5 data files and fetching a data.h5 file will automatically download the matching head.h5 too. This change cuts the overall search time by 50 - 60%, eliminates duplicate entries, and ensures that users always download the files they need. Searching for and fetching head.h5 files will only return and download the header files, should a user need or wish to do so. The old behavior of retuning a list of all files associated with each observation can produced by setting the a.eispac.FileType("any") search parameter.

In general, the FITS files are really only useful to IDL users who wish to analyze the data using older IDL routines, since they contain the exact same data as the HDF5 files, but are missing some of the essential metadata that must be provided by the SSW routines. To our knowledge, no one was using the Fido client to download the FITS files. This is evidenced by the fact that the FITS level-1 data comes in pairs of "l1" (data) and "er" (errors) files generated by the "EIS_PREP" routine in SolarSoft/IDL and the old EISClient was actually only searching for and downloading the "er" files (a problem no one has mentioned or raised before). This issue has also been patched. Now, fetching a FITS entry will download both the "l1" and "er" files.

@MJWeberg MJWeberg self-assigned this Sep 3, 2025
@MJWeberg MJWeberg added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request labels Sep 3, 2025
@MJWeberg
MJWeberg merged commit 6ed4df9 into USNavalResearchLaboratory:main Sep 3, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update Fido client to use latest version of Scraper

1 participant