Indefinite hangs on large downloads for ATL03 granules

[liangyue2018]

In Harmony Python client, downloads are executed via a thread pool, e.g.:
future = self.executor.submit(self._download_file, url, directory, overwrite)

My workflow uses harmony_client.download(...) for many granules. For some URLs (typically large files), the download appears to hang indefinitely (no completion, no exception). When a connection stalls, the worker thread can block forever, and Future.cancel() / executor.shutdown() cannot stop a running thread.

So how to handle this issue? For example, is it possible to add a timeout setting to harmony_client._download_file?

The following is a partial implementation of the download.

# download data files
def _submit_download(url):
    return harmony_client.download(url, directory=save_dir, overwrite=True)

pending = {_submit_download(url): (url, 1) for url in urls}
downloaded = []
failed = []
max_attempts = 5
timeout = 1800

while pending:
    for f in as_completed(list(pending.keys()), timeout=timeout):
        url, attempt = pending.pop(f)
        result = None
        try:
            result = f.result()

            # check validity
            if not isGranuleValid(result):
                os.remove(result)
                raise RuntimeError(f"Invalid HDF5 file: {result}")
            
            # rename files
            fpath = os.path.dirname(result)
            fname = os.path.basename(result)
            fname = f"ATL03_{fname.split('ATL03_')[1]}"
            name, ext = os.path.splitext(fname)
            if error_count > 0 and not name.endswith('_subsetted'):
                name += '_subsetted'
            fnew = os.path.join(fpath, f"{name}{ext}")
            os.rename(result, fnew)
            
            downloaded.append(fnew)
        except Exception as e:
            print(f"WARNING: Download failed (attempt {attempt}/{max_attempts}): {e}.")
            if result and os.path.exists(result):
                os.remove(result)
            if attempt < max_attempts:
                time.sleep(30)
                pending[_submit_download(url)] = (url, attempt + 1)
            else:
                failed.append(url)