[core][sandbox] Support a Docker Hub pull-through mirror

doc/source/ray-core/sandboxes.md

@@ -237,6 +237,30 @@ print(result.stdout)
 ray.get(sb.delete.remote())
 ```
 
+## Container images
+
+Sandboxes boot from OCI container images. The image manager pulls an image straight from the registry's HTTP API (anonymously, with no Docker daemon and no credentials), extracts its root filesystem into `/tmp/ray/sandbox/images` on the node, and caches it for reuse by subsequent sandboxes on that node using the same image. Sandboxes with write access to the filesystem get their own private writable overlay on top of the cached root filesystem.
+
+### Route Docker Hub pulls through a mirror
+
+Because image pulls are anonymous, every node pulling from Docker Hub consumes the anonymous pull-rate limit and downloads the image over the WAN. In a large cluster, concurrent pulls of multi-GB images can quickly hit the rate limit or saturate network bandwidth, causing image pulls to fail or become slow.
+
+Set `RAY_SANDBOX_REGISTRY_MIRROR` to route Docker Hub pulls through a registry mirror. Ray rewrites only Docker Hub image references. Pulls from other registries, such as GHCR or a private registry, are left unchanged.
+
+The value is `host[:port][/repo-prefix]`. Ray prepends the repository prefix to the repository path, which is the form pull-through caches expect:
+
+| Mirror | Example value | `python:3.10-slim` resolves to |
+| --- | --- | --- |
+| [ECR pull-through cache](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html) | `<acct>.dkr.ecr.<region>.amazonaws.com/dockerhub` | `<acct>.dkr.ecr.<region>.amazonaws.com/dockerhub/library/python` |
+| [Artifact Registry remote repository](https://cloud.google.com/artifact-registry/docs/repositories/remote-repo) | `<region>-docker.pkg.dev/<project>/<repo>` | `<region>-docker.pkg.dev/<project>/<repo>/library/python` |
+| In-cluster [`registry:2`](https://distribution.github.io/distribution/recipes/mirror/) proxy | `http://registry.default.svc.cluster.local:5000` | `http://registry.default.svc.cluster.local:5000/library/python` |
+
+Keep the following in mind:
+
+* **A bare host means HTTPS.** Write an explicit `http://` prefix for a plain-HTTP mirror, which an in-cluster `registry:2` proxy typically is.
+* **The mirror is authoritative.** Unlike Docker's registry-mirrors behavior, Ray does not fall back to Docker Hub. If the mirror is unreachable or does not contain the image, the pull fails.
+* **The mirror must allow anonymous pulls.** Ray talks to a mirror exactly as it talks to any registry, over the same anonymous bearer-token flow. If your mirror normally requires authentication, expose it to Ray through network-level access instead, such as a VPC endpoint or cluster-internal service.
+
 ## Networking and DNS
 
 Sandboxes support four network modes. The default is `none`, which follows the safe-defaults principle. Use `public` when a sandbox needs internet access.
@@ -331,7 +355,7 @@ For detailed signatures, parameters, and return types, see {ref}`ray-sandbox-ref
 
 * **`runsc` not found in `$PATH`**: Verify that gVisor's `runsc` binary is installed on all Ray worker nodes and sits in a directory on the system `$PATH`, such as `/usr/local/bin/runsc`.
 * **cgroup or permission errors**: In containerized environments such as Kubernetes without root permissions, keep the default `rootless=True`. Where cgroups are restricted, set `RAY_SANDBOX_IGNORE_CGROUPS=1`.
-* **Image pull failures**: Verify that the node can reach the container registry, such as Docker Hub or GHCR, or pre-populate the image cache directory at `/tmp/ray/sandbox/images`.
+* **Image pull failures**: Verify that the node can reach the container registry, such as Docker Hub or GHCR, or pre-populate the image cache directory at `/tmp/ray/sandbox/images`. When many nodes pull large images at once, Docker Hub's anonymous rate limits are a likely cause; see [Route Docker Hub pulls through a mirror](#route-docker-hub-pulls-through-a-mirror).
 
 ## Next steps
 

python/ray/experimental/sandbox/_internal/image_utils.py

@@ -62,6 +62,60 @@ def sanitize_image_name(image: str) -> str:
 )
 
 
+_REGISTRY_MIRROR_ENV = "RAY_SANDBOX_REGISTRY_MIRROR"
+
+
+def registry_base_url(registry: str) -> str:
+    """Return the registry as a base URL.
+
+    Bare hosts default to https. An explicit ``http://`` scheme is honored,
+    which in-cluster pull-through proxies (a plain ``registry:2``) need.
+
+    Args:
+        registry: Registry host, optionally carrying an explicit scheme.
+
+    Returns:
+        The registry with a scheme, without a trailing slash.
+    """
+    if registry.startswith(("http://", "https://")):
+        return registry
+    return f"https://{registry}"
+
+
+def apply_registry_mirror(registry: str, repo: str) -> Tuple[str, str]:
+    """Route Docker Hub pulls through a configured pull-through mirror.
+
+    ``RAY_SANDBOX_REGISTRY_MIRROR`` names a registry that mirrors Docker Hub
+    as ``host[:port][/repo-prefix]`` — e.g. an ECR pull-through cache
+    (``<acct>.dkr.ecr.<region>.amazonaws.com/dockerhub``), an Artifact
+    Registry remote repository, or an in-cluster ``registry:2`` proxy. It
+    avoids Docker Hub's anonymous rate limits and pulls over the local
+    network instead of the WAN. Only Docker Hub pulls are rewritten; other
+    registries pass through untouched. When set, the mirror is
+    authoritative (no fallback to the upstream), and it is used with the
+    same anonymous token flow as any registry.
+
+    Args:
+        registry: Registry host chosen by ``parse_image_ref``.
+        repo: Repository path chosen by ``parse_image_ref``.
+
+    Returns:
+        The possibly rewritten ``(registry, repo)`` pair.
+    """
+    mirror = os.environ.get(_REGISTRY_MIRROR_ENV, "").strip().strip("/")
+    if not mirror or registry != "registry-1.docker.io":
+        return registry, repo
+    scheme = ""
+    for candidate in ("http://", "https://"):
+        if mirror.startswith(candidate):
+            scheme, mirror = candidate, mirror[len(candidate) :]
+            break
+    host, _, prefix = mirror.partition("/")
+    if scheme:
+        host = scheme + host
+    return host, f"{prefix}/{repo}" if prefix else repo
+
+
 def parse_image_ref(image_ref: str) -> Tuple[str, str, str]:
     """Parse image reference string into (registry, repository, tag_or_digest).
 
@@ -120,7 +174,7 @@ def get_registry_auth_headers(
     timeout: float = 30.0,
 ) -> Dict[str, str]:
     """Retrieve bearer authentication token headers for registry repository."""
-    url = f"https://{registry}/v2/{repo}/manifests/{reference}"
+    url = f"{registry_base_url(registry)}/v2/{repo}/manifests/{reference}"
     req = urllib.request.Request(url, headers={"User-Agent": _USER_AGENT})
     try:
         urllib.request.urlopen(req, timeout=timeout)
@@ -384,6 +438,7 @@ def pull_and_extract_container_image(
                     )
                 try:
                     registry, repo, reference = parse_image_ref(image)
+                    registry, repo = apply_registry_mirror(registry, repo)
                     auth_headers = get_registry_auth_headers(
                         registry,
                         repo,
@@ -401,7 +456,9 @@ def pull_and_extract_container_image(
                     }
                     auth_header = auth_headers.get("Authorization")
 
-                    manifest_url = f"https://{registry}/v2/{repo}/manifests/{reference}"
+                    manifest_url = (
+                        f"{registry_base_url(registry)}/v2/{repo}/manifests/{reference}"
+                    )
                     req = _registry_request(manifest_url, headers, auth_header)
                     with urllib.request.urlopen(req, timeout=timeout_seconds) as resp:
                         manifest_data = json.loads(resp.read().decode("utf-8"))
@@ -422,7 +479,7 @@ def pull_and_extract_container_image(
                             chosen_digest = manifest_data["manifests"][0]["digest"]
 
                         sub_req = _registry_request(
-                            f"https://{registry}/v2/{repo}/manifests/{chosen_digest}",
+                            f"{registry_base_url(registry)}/v2/{repo}/manifests/{chosen_digest}",
                             headers,
                             auth_header,
                         )
@@ -435,9 +492,7 @@ def pull_and_extract_container_image(
                     config_desc = manifest_data.get("config")
                     if config_desc and "digest" in config_desc:
                         config_digest = config_desc["digest"]
-                        config_url = (
-                            f"https://{registry}/v2/{repo}/blobs/{config_digest}"
-                        )
+                        config_url = f"{registry_base_url(registry)}/v2/{repo}/blobs/{config_digest}"
                         config_req = _registry_request(config_url, headers, auth_header)
                         try:
                             with urllib.request.urlopen(
@@ -460,7 +515,9 @@ def pull_and_extract_container_image(
 
                     for layer in layers:
                         digest = layer["digest"]
-                        blob_url = f"https://{registry}/v2/{repo}/blobs/{digest}"
+                        blob_url = (
+                            f"{registry_base_url(registry)}/v2/{repo}/blobs/{digest}"
+                        )
                         blob_req = _registry_request(blob_url, headers, auth_header)
                         with urllib.request.urlopen(
                             blob_req, timeout=timeout_seconds

python/ray/experimental/sandbox/tests/test_image_manager.py

@@ -510,6 +510,52 @@ def test_prepare_oci_bundle_no_resolv_without_host_side_networking(tmp_path):
         assert _prepare(mgr, tmp_path, network=network) is None
 
 
+def test_registry_mirror_rewrites_docker_hub_only(monkeypatch):
+    """RAY_SANDBOX_REGISTRY_MIRROR reroutes Docker Hub pulls (with an
+    optional repository prefix, as ECR pull-through caches require) and
+    leaves every other registry untouched."""
+    from ray.experimental.sandbox._internal.image_utils import (
+        apply_registry_mirror,
+    )
+
+    monkeypatch.delenv("RAY_SANDBOX_REGISTRY_MIRROR", raising=False)
+    assert apply_registry_mirror("registry-1.docker.io", "library/python") == (
+        "registry-1.docker.io",
+        "library/python",
+    )
+
+    monkeypatch.setenv("RAY_SANDBOX_REGISTRY_MIRROR", "mirror.local:5000")
+    assert apply_registry_mirror("registry-1.docker.io", "library/python") == (
+        "mirror.local:5000",
+        "library/python",
+    )
+
+    monkeypatch.setenv(
+        "RAY_SANDBOX_REGISTRY_MIRROR",
+        "123.dkr.ecr.us-east-2.amazonaws.com/dockerhub/",
+    )
+    assert apply_registry_mirror("registry-1.docker.io", "library/python") == (
+        "123.dkr.ecr.us-east-2.amazonaws.com",
+        "dockerhub/library/python",
+    )
+    # Non-Docker-Hub registries are never rewritten.
+    assert apply_registry_mirror("ghcr.io", "org/repo") == ("ghcr.io", "org/repo")
+
+    # An explicit scheme is honored (plain-HTTP in-cluster proxies) and
+    # flows through URL construction; https is stripped to the default.
+    from ray.experimental.sandbox._internal.image_utils import registry_base_url
+
+    monkeypatch.setenv("RAY_SANDBOX_REGISTRY_MIRROR", "http://mirror.local:5000")
+    registry, _ = apply_registry_mirror("registry-1.docker.io", "library/python")
+    assert registry == "http://mirror.local:5000"
+    assert registry_base_url(registry) == "http://mirror.local:5000"
+
+    monkeypatch.setenv("RAY_SANDBOX_REGISTRY_MIRROR", "https://mirror.local/dockerhub")
+    registry, repo = apply_registry_mirror("registry-1.docker.io", "library/python")
+    assert (registry, repo) == ("https://mirror.local", "dockerhub/library/python")
+    assert registry_base_url("plain.host") == "https://plain.host"
+
+
 def test_extract_tar_layer_preserves_mtimes(tmp_path):
     """Archived mtimes survive extraction: apt inside the sandbox validates
     its package lists with If-Modified-Since from the file mtime, so a