Skip to content

FIX: require zarr>=3 for DataTree.to_zarr compatibility - #399

Merged
kmuehlbauer merged 1 commit into
openradar:mainfrom
syedhamidali:fix/zarr-v3-compat
Sep 1, 2026
Merged

kmuehlbauer merged 1 commit into
openradar:mainfrom
syedhamidali:fix/zarr-v3-compat

Conversation

@syedhamidali

@syedhamidali syedhamidali commented Aug 14, 2026

Copy link
Copy Markdown
Member

Summary

The multiple-sweeps-into-volume-scan notebook fails on DataTree.to_zarr(). There are actually two problems, both surfaced by current xarray:

  1. zarr<3's open_group() rejects the zarr_format argument that recent xarray forwards → TypeError: open_group() got an unexpected keyword argument 'zarr_format'.
  2. With zarr>=3, its codecs can't serialize numpy MaskedArray — and xradar's IRIS reader returns masked moments (e.g. VRADH) → ValueError: cannot reshape array of size N into shape (4N,).

Fix

  1. Bump the zarr pin to >=3 (requirements_dev.txt, environment.yml, ci/notebooktests.yml).
  2. Fill masked moments in the notebook before to_zarr (small fill_masked helper via map_over_datasets).

Verified

With zarr 3.3.0 + xarray 2026.7.0 (the exact combo that fails in CI), on a real IRIS/Sigmet file (which yields a masked VRADH): the to_zarr + open_datatree roundtrip succeeds and VRADH values are preserved (masked → NaN).

Note: I couldn't run the full notebook locally (its data is on the IDEAM s3-radaresideam bucket), but reproduced and fixed the exact failing operation with representative IRIS data.

@codecov

codecov Bot commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.23%. Comparing base (7b81ba0) to head (a9051c2).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #399   +/-   ##
=======================================
  Coverage   94.23%   94.23%           
=======================================
  Files          29       29           
  Lines        6452     6452           
=======================================
  Hits         6080     6080           
  Misses        372      372           
Flag Coverage Δ
notebooktests 0.00% <ø> (?)
unittests 94.23% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kmuehlbauer

Copy link
Copy Markdown
Collaborator

Thanks @syedhamidali, for trying. There is an issue in the writer. Maybe @aladinor has a clue?

)

Current xarray forwards a `zarr_format` argument to `zarr.open_group()`,
which `zarr<3` rejects. Bump the pin to `>=3` in requirements_dev.txt,
environment.yml and ci/notebooktests.yml.

zarr v3 then surfaces a second issue: its codecs cannot serialize numpy
MaskedArrays, which xradar's IRIS reader returns for moments such as VRADH.
Fill masked entries in the multiple-sweeps-into-volume-scan notebook before
`to_zarr`.

Verified with real IRIS data (zarr 3.3.0 + xarray 2026.7.0): the to_zarr +
open_datatree roundtrip succeeds and VRADH values are preserved (masked -> NaN).
@kmuehlbauer
kmuehlbauer merged commit 10a31b4 into openradar:main Sep 1, 2026
11 checks passed
@kmuehlbauer

Copy link
Copy Markdown
Collaborator

Thanks @syedhamidali! This will unlock other PR, which got stuck because of zarr < 3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Zarr compatibility issue: TypeError on to_zarr() with zarr_format argument

2 participants