download_file: do not HEAD pre-signed S3 URIs

Running HEAD on a pre-signed S3 URI
gives a 403 response because the URI
is signed for GET. Download pre-signed
URIs all at once instead.

Fixes #939

Author: pjonsson

CHANGELOG.md

@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Convert setup.py to pyproject.toml ([#920](https://github.com/Open-EO/openeo-python-client/issues/920))
 - Make `_DerivedFrom._from_url` more resilient against unresolvable/unparsable `derived_from` links ([#928](https://github.com/Open-EO/openeo-python-client/issues/928), eu-cdse/openeo-cdse-infra#1338)
+- Download pre-signed S3 URIs all at once ([#939](https://github.com/Open-EO/openeo-python-client/issues/939))
 
 ### Removed
 

openeo/rest/_connection.py

@@ -9,6 +9,7 @@
 import urllib3.util
 from requests import Response
 from requests.auth import AuthBase
+from urllib.parse import parse_qs, urlparse
 
 import openeo
 from openeo.rest import (
@@ -305,8 +306,12 @@ def download_url(
         chunk_size: int = DEFAULT_DOWNLOAD_CHUNK_SIZE,
         range_size: int = DEFAULT_DOWNLOAD_RANGE_SIZE,
     ) -> None:
-        head = self.head(url, stream=True)
-        if head.ok and head.headers.get("Accept-Ranges") == "bytes" and "Content-Length" in head.headers:
+        query_params = parse_qs(urlparse(url).query)
+        # Pre-signed S3 URIs are signed for "GET" and will result in a 403 when running
+        # HEAD on them, so download them all at once instead.
+        signed = "X-Amz-Signature" in query_params or "Signature" in query_params
+        head = None if signed else self.head(url, stream=True)
+        if head and head.ok and head.headers.get("Accept-Ranges") == "bytes" and "Content-Length" in head.headers:
             file_size = int(head.headers["Content-Length"])
             self._download_ranged(
                 url=url, target=target, file_size=file_size, chunk_size=chunk_size, range_size=range_size