Download fails with "error decoding response body"

[seanses]

Describe the bug

Users report some downloads fail with RuntimeError: Task error: File reconstruction error: CAS Client Error: Format error: I/O error: error decoding response body. One user explicitly mentioned that this only happened when he is logged in.

System info

See #869 (comment) & #869 (comment)

[nleroy917]

I'm also experiencing this while uploading large datasets.

Specifically, the POST /v2/shards streaming shard finalization call gets cut after ~5 min under concurrent large-shard commits:

ConnectionError: error decoding response body

Versions/stack:

• hf-xet 1.6.0
• huggingface_hub 1.28.0
• Python 3.14.7
• Operating in AWS us-east-1.

During a bulk migration (8 nodes, one org token, using the upload_folder API in batches of ~32 GiB or 3–4 parquet files each) the data transfer completes at ~260 MiB/s per node, but shard finalization fails after ~5 min:

_upload_pipeline.py:512 _committer_loop → _process_batch
_upload_pipeline.py:524 batch.xet_commit.wait_to_finish()
ConnectionError: Network error: Request error: error decoding response body, domain: no-url

I noticed that similar commits are fast when the server isn't loaded. A solo 3×8 GiB folder upload from the same host/token was able to commit in seconds (25.8 GB end-to-end in 62 s). Moreover, one of 8 concurrent nodes landed its commit in ~ 124 s while three neighbors timed out in the same minute. So, this kinda looks like shard-validation latency variance server-side racing an ~5-minute timeout on the v1.6.0 streaming-finalization response?

[rajatarya]

Hi @nleroy917 : Thanks for adding repro steps here. I believe this issue could be resolved by a change we made last week and are testing now. Can you retry using this pre-release of hf-xet and see if the timeouts continue?

pip install --pre hf-xet==1.6.1a0

The HTTP library our client and server both build on (hyper) has a rare bug in its HTTP/1 write path: the connection loop can see the socket as ready to write, then park itself with nothing guaranteed to wake it again. The shard upload stalls partway through the body with both ends alive and waiting - and once it is parked the upload never completes and eventually times out. It's fixed upstream in hyper 1.10.0; we were pinned to 1.8.1.