fix(url): also emit #datalabBackendUrl=<full URL> fragment

docs/04_automation_and_utility.md

@@ -1,5 +1,6 @@
 ---
 log:
+2026-05-27: `colab url` now emits BOTH the `?dbu=<urlencoded path>` query parameter (existing) AND a new `#datalabBackendUrl=<full URL>` hash fragment (new). Format: `https://<host>/notebooks/empty.ipynb?dbu=%2Ftun%2Fm%2F<endpoint>#datalabBackendUrl=<host>/tun/m/<endpoint>`. Why both: some Colab frontend code paths consult the hash fragment first and ignore `dbu` entirely, so the previously-emitted query-only form failed silently for those users (the frontend fell through to allocating a fresh VM via `/tun/m/assign`). The fragment value is a FULL URL with scheme + host (NOT just the path) and is emitted RAW (no URL encoding) because browsers don't decode the fragment before passing `location.hash` to page JS — Colab's parser calls `new URL(rawString)` directly. The fragment host always matches `--host` so Colab's same-origin enforcement on embedded backend URLs doesn't block the connection, and sandbox/dev users (`--host https://colab.sandbox.google.com`) get a sandbox fragment automatically. Three new test cases in `tests/test_url.py` cover the raw-encoding requirement (`%3A`/`%2F` must NOT appear in the fragment), the both-signals-present invariant, and `--open` propagating the fragment to `webbrowser.open()`. Integration-verified live against synthetic session state with three host shapes (default, sandbox, trailing-slash); all produced correctly-shaped URLs with no `//` artifacts.
 2026-05-07: Added a developer-only `colab whoami` subcommand (hidden from `colab --help`). Mints an access token via the same `auth.get_credentials(...)` path the rest of the CLI uses (honoring the global `--auth=...` flag), refreshes the credentials, then queries `https://oauth2.googleapis.com/tokeninfo` to print the email, scopes, audience, and expiry of whatever the CLI is about to send. Built specifically to short-circuit the "why is my call to colab.pa.googleapis.com 403-ing" debugging loop — the answer is almost always "missing scope" or "wrong identity", both of which `whoami` makes immediately visible. Hidden via `app.command(hidden=True)`; reachable via `colab whoami` or `colab whoami --help`. Suppressed from the daily-update banner check (added to `_AUTO_UPDATE_SUPPRESSED` in `cli.py`) so the banner doesn't obscure the auth output.
 2026-05-11: Removed the local-file update source (`update_file_path` setting and `_fetch_local` helper); `colab update` now consults PyPI only. Switched the default `update_url` to the canonical PyPI JSON API (`https://pypi.org/pypi/google-colab-cli/json`), which already exposes the `info.version` schema the auto-update subsystem expects. Re-added `colab update --install` as a public self-install path that runs `pip install -U google-colab-cli` against the current `sys.executable`; Linux-only (other platforms exit non-zero with an explanatory message), and a silent no-op when the cached `latest_version` is already at or below the current install.
 2026-05-12: Added an optional `timeout=` parameter to `ColabRuntime.execute_code` that flows through to both the `execute()` and `execute_interactive()` branches. `colab auth` and `colab drivemount` now pass `timeout=600` (10 min) via a shared `INTERACTIVE_AUTOMATION_TIMEOUT_SEC` constant in `commands/automation.py`. Background: `jupyter_kernel_client` defaults to a 10s wall-clock timeout that is consumed even when the kernel is idle waiting on `input_request`. With the drivefs hook intercepting that request and prompting the user to OAuth in their browser, any user that takes >10s to click through (essentially everyone) hit `TimeoutError` and saw "drivemount failed" even though the mount had actually succeeded server-side. The fix is scoped narrowly to the two human-in-the-loop subcommands; non-interactive paths (`colab exec`, `colab run`, `colab install`, `colab repl --pipe`, `colab console --pipe`) keep the upstream default since they receive continuous iopub traffic that resets the practical inactivity ceiling.

src/colab_cli/commands/utility.py

@@ -62,14 +62,27 @@ def url(
 ):
     """Print a browser URL that connects to an existing session.
 
-    Format: ``https://<host>/notebooks/empty.ipynb?dbu=<urlencoded path>``,
+    Format: ``https://<host>/notebooks/empty.ipynb?dbu=<urlencoded path>#datalabBackendUrl=<host>/tun/m/<endpoint>``,
     where the path is ``/tun/m/<endpoint>``. When opened, the Colab frontend
     skips ``/tun/m/assign`` and attaches the kernel to our existing VM.
 
-    The ``dbu`` query parameter is the ``datalab_backend_url`` development
-    flag. Because it's a development flag, URL-overriding it may be gated
-    by the Colab frontend; some users may need to use the hash-based
-    ``#datalabBackendUrl=...`` form instead.
+    Two backend-URL signals are embedded:
+
+    - ``?dbu=<urlencoded path>`` — the ``datalab_backend_url`` development
+      query flag. The frontend resolves the value against
+      ``window.location.origin``.
+
+    - ``#datalabBackendUrl=<full URL>`` — the hash-fragment form. Some
+      frontend code paths consult this first and ignore ``dbu``, so we
+      emit both for robustness. The fragment value is a FULL URL (with
+      scheme + host) and is intentionally NOT URL-encoded — browsers do
+      not decode the fragment before passing ``location.hash`` to page
+      JS, and Colab's hash parser expects the raw string.
+
+    The fragment's host always matches ``--host`` (the page origin), so
+    Colab's same-origin enforcement on the embedded backend URL doesn't
+    block the connection, and sandbox/dev users get a sandbox fragment
+    automatically.
     """
     # Imported here (not at module top) to mirror the lazy-state pattern used
     # elsewhere in this module and avoid a circular import via colab_cli.common.
@@ -83,14 +96,26 @@ def url(
         typer.echo(f"[colab] Session '{name}' not found.", err=True)
         raise typer.Exit(code=1)
 
-    # Strip a trailing slash so we don't produce `https://host//notebooks/...`.
+    # Strip a trailing slash so we don't produce `https://host//notebooks/...`
+    # or `https://host//tun/m/...` in the fragment URL.
     host_clean = host.rstrip("/")
-    # `dbu` value is the path `/tun/m/<endpoint>`. URL-encode it (incl. the
-    # slashes via `safe=""`) so the value survives any downstream non-strict
-    # query-string re-parsing — this is also the form shown in real Colab
-    # connect URLs in the wild.
-    dbu_value = quote(f"/tun/m/{s.endpoint}", safe="")
-    connect_url = f"{host_clean}/notebooks/empty.ipynb?dbu={dbu_value}"
+    backend_path = f"/tun/m/{s.endpoint}"
+    # `dbu` value is the backend path. URL-encode it (incl. the slashes via
+    # `safe=""`) so the value survives any downstream non-strict query-string
+    # re-parsing — this is also the form shown in real Colab connect URLs.
+    dbu_value = quote(backend_path, safe="")
+    # `#datalabBackendUrl=` value is the FULL backend URL, raw (un-encoded):
+    # the browser does not decode the fragment before passing it to page JS,
+    # and Colab's hash parser calls `new URL(rawString)` directly. Pinning
+    # the host to `host_clean` (not hardcoding research.google.com) keeps
+    # this aligned with the page origin so same-origin enforcement passes
+    # for sandbox / dev hosts too.
+    fragment_value = f"{host_clean}{backend_path}"
+    connect_url = (
+        f"{host_clean}/notebooks/empty.ipynb"
+        f"?dbu={dbu_value}"
+        f"#datalabBackendUrl={fragment_value}"
+    )
 
     # Print the URL on its own line with no `[colab]` prefix so the output
     # is pipeable (`colab url -s s1 | xclip`, etc.).

tests/test_url.py

@@ -17,17 +17,26 @@
 
 URL format:
 
-    https://<host>/notebooks/empty.ipynb?dbu=<urlencoded-/tun/m/endpoint>
-
-The `dbu` query parameter is the Colab frontend's `datalab_backend_url`
-development flag; the frontend resolves the value against
-`window.location.origin` and attaches the kernel to the supplied
-`/tun/m/<endpoint>` path instead of allocating a fresh VM.
-
-Note: `dbu` is a development flag, so URL-overriding it may be gated by
-the Colab frontend; some users may need to use the hash-based
-`#datalabBackendUrl=...` form instead (we don't support that today; file
-an issue if you need it).
+    https://<host>/notebooks/empty.ipynb?dbu=<urlencoded-/tun/m/endpoint>#datalabBackendUrl=<host>/tun/m/<endpoint>
+
+Two backend-URL signals are embedded:
+
+-   `?dbu=<urlencoded path>` -- the Colab frontend's
+    `datalab_backend_url` development query flag. The frontend resolves
+    the value against `window.location.origin` and attaches the kernel
+    to the supplied `/tun/m/<endpoint>` path instead of allocating a
+    fresh VM.
+
+-   `#datalabBackendUrl=<full URL>` -- the hash-fragment form. Some
+    Colab frontend code paths consult this first and ignore `dbu`, so we
+    emit both for robustness. The fragment value is a FULL URL (with
+    scheme + host) and is intentionally NOT URL-encoded -- browsers do
+    not decode fragment values before passing them to page JS, and
+    Colab's hash parser expects the raw string.
+
+The fragment's host always matches the page origin (`--host`), so
+same-origin enforcement in the frontend doesn't block the connection
+and sandbox/dev users get a sandbox fragment automatically.
 """
 
 from unittest.mock import MagicMock, patch
@@ -59,10 +68,11 @@ def _parse_url_output(output: str) -> str:
 def test_url_explicit_session(mock_common_state):
     """`colab url -s NAME` prints the connect URL for that session.
 
-    Format: ``https://<host>/notebooks/empty.ipynb?dbu=<urlencoded path>``.
+    Format: ``https://<host>/notebooks/empty.ipynb?dbu=<urlencoded path>#datalabBackendUrl=<host>/tun/m/<endpoint>``.
     The path must land on `empty.ipynb` so the user sees a usable notebook
     UI; the `dbu` query param tells the frontend to skip /tun/m/assign and
-    attach to our existing endpoint.
+    attach to our existing endpoint; the `#datalabBackendUrl=` fragment
+    is the alternative signal some frontend code paths consult.
     """
     s = _make_session(name="my-sess", endpoint="ep-XYZ")
     mock_common_state.store.get.return_value = s
@@ -90,6 +100,12 @@ def test_url_explicit_session(mock_common_state):
     # downstream re-parsing that treats the query string non-strictly.
     assert "dbu=%2Ftun%2Fm%2Fep-XYZ" in url
 
+    # Fragment: raw (not URL-encoded), full URL form, host matches page origin.
+    assert (
+        parsed.fragment
+        == "datalabBackendUrl=https://colab.research.google.com/tun/m/ep-XYZ"
+    )
+
 
 def test_url_resolves_unique_session(mock_common_state):
     """`colab url` (no -s) uses the unique-session resolution path."""
@@ -120,10 +136,16 @@ def test_url_session_not_found(mock_common_state):
 
 
 def test_url_custom_host(mock_common_state):
-    """`--host` overrides the default frontend host. `dbu` is a path-only
-    value (resolved against `window.location.origin` in the frontend), so
-    the host swap only affects the page origin, not the embedded backend
-    path."""
+    """`--host` overrides the default frontend host AND the host used in
+    the `#datalabBackendUrl=` fragment.
+
+    `dbu` itself is a path-only value (resolved against
+    `window.location.origin` in the frontend), so the host swap only
+    affects the page origin, not the embedded backend path. But the
+    fragment carries a full URL, and the Colab frontend enforces
+    same-origin between page and embedded backend URL -- so the fragment
+    host MUST match `--host` for the swap to work end-to-end.
+    """
     s = _make_session(endpoint="ep1")
     mock_common_state.store.get.return_value = s
     mock_common_state.resolve_session.return_value = "s1"
@@ -138,11 +160,17 @@ def test_url_custom_host(mock_common_state):
     assert parsed.netloc == "colab.sandbox.google.com"
     assert parsed.path == "/notebooks/empty.ipynb"
     assert parse_qs(parsed.query).get("dbu") == ["/tun/m/ep1"]
+    # Fragment host tracks --host (NOT pinned to research.google.com).
+    assert (
+        parsed.fragment
+        == "datalabBackendUrl=https://colab.sandbox.google.com/tun/m/ep1"
+    )
 
 
 def test_url_host_normalises_trailing_slash(mock_common_state):
     """`--host https://example.com/` (with trailing slash) must not produce
-    a double slash before `/notebooks/empty.ipynb`."""
+    a double slash anywhere -- not before `/notebooks/empty.ipynb` in the
+    page URL, AND not before `/tun/m/...` in the fragment value."""
     s = _make_session(endpoint="ep2")
     mock_common_state.store.get.return_value = s
     mock_common_state.resolve_session.return_value = "s1"
@@ -154,6 +182,11 @@ def test_url_host_normalises_trailing_slash(mock_common_state):
     assert result.exit_code == 0
     assert "https://colab.research.google.com//notebooks/" not in result.output
     assert "https://colab.research.google.com/notebooks/empty.ipynb" in result.output
+    # Same guarantee for the fragment URL.
+    assert "https://colab.research.google.com//tun/" not in result.output
+    assert (
+        "datalabBackendUrl=https://colab.research.google.com/tun/m/ep2" in result.output
+    )
 
 
 def test_url_endpoint_with_special_chars_is_encoded(mock_common_state):
@@ -210,6 +243,76 @@ def test_url_no_open_by_default(mock_common_state):
     mock_open.assert_not_called()
 
 
+def test_url_fragment_is_not_url_encoded(mock_common_state):
+    """The `#datalabBackendUrl=...` fragment value is a full URL and must
+    be embedded raw (no percent-encoding). Browsers do not decode the
+    fragment before passing `location.hash` to page JS, and the Colab
+    parser expects to call `new URL(rawString)` directly. If we encoded
+    `:` -> `%3A` or `/` -> `%2F` here, the parser would see
+    `https%3A%2F%2Fcolab...` and fail.
+
+    Concretely we should see the literal `://` and unescaped `/` in the
+    fragment, NOT their percent-encoded counterparts.
+    """
+    s = _make_session(endpoint="ep-RAW")
+    mock_common_state.store.get.return_value = s
+    mock_common_state.resolve_session.return_value = "s1"
+
+    result = runner.invoke(app, ["url", "-s", "s1"])
+    assert result.exit_code == 0, result.output
+    url = _parse_url_output(result.output)
+    parsed = urlparse(url)
+
+    # The fragment must contain the literal scheme + slashes...
+    assert "datalabBackendUrl=https://" in url
+    assert "/tun/m/ep-RAW" in parsed.fragment
+    # ...and MUST NOT contain percent-encoded versions of `:`, `/`.
+    assert "%3A" not in parsed.fragment
+    assert "%2F" not in parsed.fragment
+
+
+def test_url_both_signals_present(mock_common_state):
+    """Invariant: every printed URL has BOTH `?dbu=` and `#datalabBackendUrl=`.
+
+    Either alone is unreliable across Colab frontend revisions; we emit
+    both so the frontend can use whichever it consults first.
+    """
+    s = _make_session(endpoint="ep-BOTH")
+    mock_common_state.store.get.return_value = s
+    mock_common_state.resolve_session.return_value = "s1"
+
+    result = runner.invoke(app, ["url", "-s", "s1"])
+    assert result.exit_code == 0, result.output
+    url = _parse_url_output(result.output)
+    assert "?dbu=" in url
+    assert "#datalabBackendUrl=" in url
+
+
+def test_url_open_flag_includes_fragment(mock_common_state):
+    """`--open` opens the SAME URL it printed, including the fragment.
+
+    `webbrowser.open()` must receive the URL with the `#datalabBackendUrl=`
+    fragment intact -- otherwise the browser may attach to a fresh VM via
+    `/tun/m/assign` instead of our existing session.
+    """
+    s = _make_session(endpoint="ep-OPEN2")
+    mock_common_state.store.get.return_value = s
+    mock_common_state.resolve_session.return_value = "s1"
+
+    with patch("webbrowser.open") as mock_open:
+        result = runner.invoke(app, ["url", "-s", "s1", "--open"])
+
+    assert result.exit_code == 0, result.output
+    mock_open.assert_called_once()
+    opened_url = mock_open.call_args[0][0]
+    assert (
+        "#datalabBackendUrl=https://colab.research.google.com/tun/m/ep-OPEN2"
+        in opened_url
+    )
+    # And it's the same URL that got printed.
+    assert opened_url in result.output
+
+
 def test_url_output_is_pipeable(mock_common_state):
     """The printed URL line must be machine-parseable: a single line with no
     leading `[colab]` chatter, so `colab url -s s1 | xclip` works.