Fix writing scalar datasets with compound dtype

[Copilot]

Motivation

Writing scalar (0-dimensional) numpy arrays with compound dtypes fails with IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed. This occurs in PyNWB when writing ElectrodeGroup.position which uses a compound dtype (x, y, z).

The fix follows HDF5IO's approach for handling scalar compound dtypes by using len(np.shape(data)) == 0 to detect scalars and routing them to __scalar_fill__ instead of __list_fill__.

How to test the behavior?

from pynwb import NWBFile
from hdmf_zarr import NWBZarrIO
from datetime import datetime
from uuid import uuid4

nwbfile = NWBFile(
    session_description="test",
    session_start_time=datetime.now(),
    identifier=str(uuid4()),
)

device = nwbfile.create_device(name="Device", description="test")
electrode_group = nwbfile.create_electrode_group(
    name="ElectrodeGroup",
    description="test",
    device=device,
    location="unknown",
    position=(1.0, 2.0, 3.0),  # Scalar compound dtype
)

with NWBZarrIO('test.nwb', 'w') as io:
    io.write(nwbfile)  # Previously raised IndexError, now succeeds

with NWBZarrIO('test.nwb', 'r') as io:
    read_nwbfile = io.read()
    assert read_nwbfile.electrode_groups['ElectrodeGroup'].position[0] == (1.0, 2.0, 3.0)

Changes:

• Compound dtype handling: Add elif len(np.shape(data)) == 0: check to route scalar compound dtypes to __scalar_fill__ (matches HDF5IO approach)
• Regular dataset flow: Add elif len(np.shape(data)) == 0: check before hasattr(data, "__len__") to catch scalars without explicit dtype specification
• __scalar_fill__(): Update to properly wrap scalar compound data with np.array([data], dtype=dtype) for writing to shape=(1,) datasets
• get_type(): Detect 0-dimensional arrays via data.ndim == 0 and return data.dtype directly to avoid TypeError
• Tests: Add test_write_scalar_compound, test_write_scalar_compound_with_dtype_spec, and test_read_scalar_compound

Checklist

• Did you update CHANGELOG.md with your changes?
• Does the PR clearly describe the problem and the solution?
• Have you reviewed our Contributing Guide?
• Does the PR use "Fix #XXX" notation to tell GitHub to close the relevant issue numbered XXX when the PR is merged?

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[Bug]: Writing scalar dataset with compound dtype</issue_title>
<issue_description>### What happened?

See comment in #276 (review) for reference.

This had come up recently in hdmf, but there seems to also be an error with hdmf-zarr writing scalar datasets with a compound dtype.

Steps to Reproduce

from pynwb import NWBFile
from uuid import uuid4
from datetime import datetime

from hdmf_zarr import NWBZarrIO

nwbfile = NWBFile(
    session_description="no description.",
    session_start_time=datetime.now(),
    identifier=str(uuid4()),
)

device = nwbfile.create_device(name="Device", description="no description.")

electrode_group = nwbfile.create_electrode_group(
    name="ElectrodeGroup",
    description="no description.",
    device=device,
    location="unknown",
    position=(1.0, 2.0, 3.0),
)

with NWBZarrIO('test.nwb', 'w') as io:
    io.write(nwbfile)  # errors here
    
with NWBZarrIO('test.nwb', 'r') as io:
    read_nwbfile = io.read()
    
    print(read_nwbfile.electrode_groups['ElectrodeGroup'].position.shape)

Traceback

File "/Users/smprince/.vscode/extensions/ms-python.debugpy-2025.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 127, in _run_module_code
    _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)
  File "/Users/smprince/.vscode/extensions/ms-python.debugpy-2025.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
    exec(code, run_globals)
  File "/Users/smprince/Documents/code/hdmf-zarr/docs/issue_testing.py", line 24, in <module>
    io.write(nwbfile)
  File "/Users/smprince/Documents/code/hdmf/src/hdmf/utils.py", line 578, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smprince/Documents/code/hdmf-zarr/src/hdmf_zarr/backend.py", line 347, in write
    super(ZarrIO, self).write(**kwargs)
  File "/Users/smprince/Documents/code/hdmf/src/hdmf/utils.py", line 578, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smprince/Documents/code/hdmf/src/hdmf/backends/io.py", line 99, in write
    self.write_builder(f_builder, **kwargs)
  File "/Users/smprince/Documents/code/hdmf/src/hdmf/utils.py", line 578, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smprince/Documents/code/hdmf-zarr/src/hdmf_zarr/backend.py", line 522, in write_builder
    self.write_group(
  File "/Users/smprince/Documents/code/hdmf/src/hdmf/utils.py", line 578, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smprince/Documents/code/hdmf-zarr/src/hdmf_zarr/backend.py", line 621, in write_group
    self.write_group(
  File "/Users/smprince/Documents/code/hdmf/src/hdmf/utils.py", line 578, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smprince/Documents/code/hdmf-zarr/src/hdmf_zarr/backend.py", line 621, in write_group
    self.write_group(
  File "/Users/smprince/Documents/code/hdmf/src/hdmf/utils.py", line 578, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smprince/Documents/code/hdmf-zarr/src/hdmf_zarr/backend.py", line 632, in write_group
    self.write_dataset(
  File "/Users/smprince/Documents/code/hdmf/src/hdmf/utils.py", line 578, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smprince/Documents/code/hdmf-zarr/src/hdmf_zarr/backend.py", line 1186, in write_dataset
    dset = self.__list_fill__(parent, name, data, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smprince/Documents/code/hdmf-zarr/src/hdmf_zarr/backend.py", line 1355, in __list_fill__
    data = data[:]  # load the data in case we come from HDF5 or another on-disk data source we don't know
           ~~~~^^^
IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed

Operating System

Windows

Python Version

3.12

Package Versions

No response</issue_description>

Comments on the Issue (you are @copilot in this section)

@mavaylon1 I will take this one. @stephprince you mentioned this comes up in hdmf. Do you know any open or closed issues on the matter?

• Fixes [Bug]: Writing scalar dataset with compound dtype #277

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[rly]

@copilot For writing a scalar dataset with compound dtype, let's try to be as similar as possible to HDF5IO's approach in https://github.com/hdmf-dev/hdmf/blob/dev/src/hdmf/backends/hdf5/h5tools.py. Specifically, write_dataset has these lines:

        elif isinstance(options['dtype'], list):
            # do some stuff to figure out what data is a reference
            refs = list()
            for i, dts in enumerate(options['dtype']):
                if self.__is_ref(dts):
                    refs.append(i)
            # If one or more of the parts of the compound data type are references then we need to deal with those
            if len(refs) > 0:
                try:
                    _dtype = self.__resolve_dtype__(options['dtype'], data)
                except Exception as exc:
                    msg = 'cannot add %s to %s - could not determine type' % (name, parent.name)
                    raise Exception(msg) from exc
                dset = parent.require_dataset(name, shape=(len(data),), dtype=_dtype, **options['io_settings'])
                self.__set_written(builder)
                self.logger.debug("Queueing reference resolution and set attribute on dataset '%s' containing "
                                  "object references. attributes: %s"
                                  % (name, list(attributes.keys())))

                @self.__queue_ref
                def _filler():
                    self.logger.debug("Resolving object references and setting attribute on dataset '%s' "
                                      "containing attributes: %s"
                                      % (name, list(attributes.keys())))
                    ret = list()
                    for item in data:
                        new_item = list(item)
                        for i in refs:
                            new_item[i] = self.__get_ref(item[i])
                        ret.append(tuple(new_item))
                    dset = parent[name]
                    dset[:] = ret
                    self.set_attributes(dset, attributes)

                return
            # If the compound data type contains only regular data (i.e., no references) then we can write it as usual
            elif len(np.shape(data)) == 0:
                dset = self.__scalar_fill__(parent, name, data, options)
            else:
                dset = self.__list_fill__(parent, name, data, options)

where the branch to use __scalar_fill__ was added somewhat recently to resolve a similar issue. Try an approach like this instead.

[Copilot]

@copilot For writing a scalar dataset with compound dtype, let's try to be as similar as possible to HDF5IO's approach in https://github.com/hdmf-dev/hdmf/blob/dev/src/hdmf/backends/hdf5/h5tools.py. S...

Updated the implementation to match HDF5IO's approach. Added len(np.shape(data)) == 0 checks in both the compound dtype handling section and the regular dataset flow to route scalar compound dtypes to __scalar_fill__, similar to how HDF5IO handles this case. Also updated __scalar_fill__ to properly wrap scalar compound data for writing. See commit ea7d482.

[rly]

Merge #310 first, then update this PR

[codecov-commenter]

Codecov Report

❌ Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.57%. Comparing base (faf7cf7) to head (a5360b2).

Files with missing lines	Patch %	Lines
src/hdmf_zarr/backend.py	81.81%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #307      +/-   ##
==========================================
- Coverage   85.78%   85.57%   -0.21%     
==========================================
  Files           5        5              
  Lines        1238     1248      +10     
  Branches      234      239       +5     
==========================================
+ Hits         1062     1068       +6     
- Misses        120      122       +2     
- Partials       56       58       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.