Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/source/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Open raw and converted files
.. _api-open_raw:

.. automodule:: echopype
:members: open_raw, open_converted, combine_echodata
:members: open_raw, open_converted

Data processing subpackages
---------------------------
Expand Down
82 changes: 0 additions & 82 deletions docs/source/open-converted.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -50,88 +50,6 @@
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Combine EchoData objects"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{Warning}\n",
"The `combine_echodata` function will be deprecated in version `v0.12.1`. Instead, we recommend using `ep.calibrate.compute_Sv` to compute volume backscattering strength (Sv) for each `EchoData` object individually, then combining the resulting `Sv` datasets rather than combining `EchoData` objects directly. See [Combine volume backscattering strength (Sv) datasets](#combine-volume-backscattering-strength-sv-datasets) for more details.\n",
":::"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Data collected by the same instrument deployment across multiple files can be combined into a single [`EchoData` object](data-format:echodata-object) using [`combine_echodata`](echopype.combine_echodata). With the release of echopype version `0.6.3`, one can now combine a large number of files in parallel (using [Dask](https://www.dask.org/)) while maintaining a stable memory usage. This is done under-the-hood by concatenating data directly into a Zarr store, which corresponds to the final combined `EchoData` object. \n",
"\n",
"To use `combine_echodata`, the following criteria must be met: \n",
"- Each `EchoData` object must have the same `sonar_model`\n",
"- The `EchoData` objects to be combined must correspond to different raw data files (i.e., no duplicated files)\n",
"- The `EchoData` objects in the list must be of sequential order in time. Specifically, the first timestamp of each `EchoData` object must be smaller (earlier) than the first timestamp of the subsequent `EchoData` object\n",
"- The `EchoData` objects must contain the same frequency channels and the same number of channels\n",
"- The following attribute criteria must be satisfied for all groups under each of the `EchoData` objects to be combined:\n",
" - the names of all attributes must be the same\n",
" - the values of all attributes must be identical (other than the attributes `date_created` or `conversion_time`; these attributes should have the same data type)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{Attention}\n",
"In previous versions, `combine_echodata` corrected reversed timestamps and stored the uncorrected timestamps in the `Provenance` group.\n",
"Starting from `0.6.3`, `combine_echodata` will preserve time coordinates that have reversed timestamps and no correction is performed. \n",
":::"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"The first step in combining data is to establish a Dask client with a scheduler. On a local machine, this can be done as follows:\n",
"```python\n",
"client = Client() # create client with local scheduler\n",
"```\n",
"With distributed resources, we highly recommend reviewing the Dask documentation for [deploying Dask clusters](https://docs.dask.org/en/latest/deploying.html). \n",
"\n",
"Next, we assemble a list of `EchoData` objects. This list can be from converted files (netCDF or Zarr) as in the example below, or from in-memory `EchoData` objects:\n",
"```python\n",
"ed_list = []\n",
"for converted_file in [\"convertedfile1.zarr\", \"convertedfile2.zarr\"]:\n",
" ed_list.append(ep.open_converted(converted_file)) # already converted files are lazy-loaded\n",
"```\n",
"\n",
"Finally, we apply `combine_echodata` on this list to combine all the data into a single `EchoData` object. Here, we will store the final combined form in the Zarr path `path_to/combined_echodata.zarr` and use the client we established above: \n",
"```python\n",
"combined_ed = ep.combine_echodata(\n",
" ed_list, \n",
" zarr_path=\"path_to/combined_echodata.zarr\", \n",
" client=client\n",
")\n",
"```\n",
"Once executed, `combine_echodata` returns a lazy loaded `EchoData` object (obtained from `zarr_path`) with all data from the input `EchoData` objects combined."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{Note}\n",
"As shown in the above example, the path of the combined Zarr store is given by the keyword argument `zarr_path`, \n",
"and the Dask client that parallel tasks will be submitted to is given by the keyword argument `client`.\n",
"When either (or both) of these are not provided, default values listed in the `Notes` section in [`combine_echodata`](echopype.combine_echodata) will be used.\n",
":::"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down
2 changes: 0 additions & 2 deletions echopype/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
from . import calibrate, clean, commongrid, consolidate, mask, utils
from .convert.api import open_raw
from .echodata.api import open_converted
from .echodata.combine import combine_echodata
from .utils.io import init_ep_dir
from .utils.log import verbose

Expand All @@ -17,7 +16,6 @@
__all__ = [
"calibrate",
"clean",
"combine_echodata",
"commongrid",
"consolidate",
"mask",
Expand Down
Loading
Loading