perf(dashboard): take provider dials and whole-table reads off page loads

docs/configuration.md

@@ -87,12 +87,12 @@ pricing:
 | `stream_missing_usage_policy` | string | `"estimate"` | How to bill a streamed response that completes with no provider usage data: `"estimate"` (charge the up-front estimate), `"fail"` (charge estimate and mark errored), or `"allow_free"` (don't bill). |
 | `budget_estimate_default_output_tokens` | int | `1024` | Output-token count assumed when reserving budget for a request with no declared max output; reconciled to actual usage on completion. |
 | `streaming_keepalive_interval_ms` | int | `15000` | Idle interval after which a streaming response emits a transport keepalive while it waits on the provider: a `ping` event on `/v1/messages`, an SSE comment line (`: keepalive`) on `/v1/chat/completions` and `/v1/responses`. Keeps an intermediary with a read timeout (Cloudflare's default Proxy Read Timeout is 125s) from severing a connection during a long time-to-first-token. Keepalives start once the provider stream is open and never touch usage accounting or extend a first-chunk/failover deadline. `0` disables. |
-| `model_discovery` | bool | `true` | Auto-discover models for `GET /v1/models` from the configured providers: the `providers` block plus anything added at runtime on the Providers page. A provider that is callable through its credential environment variable alone is not discovered until it has an entry. |
-| `model_cache_ttl_seconds` | int | `300` | TTL for the in-memory model-discovery cache (`0` disables caching). |
+| `model_discovery` | bool | `true` | Auto-discover models for `GET /v1/models` from the configured providers: the `providers` block plus anything added at runtime on the Providers page. A provider that is callable through its credential environment variable alone is not discovered until it has an entry. Setting this to `false` also stops the background refresher, so the gateway makes no unattended `list_models` calls; the operator-facing `/v1/models/discoverable` and `/v1/providers/health` still dial when asked. |
+| `model_cache_ttl_seconds` | int | `300` | TTL for the in-memory model-discovery cache, and the interval at which a background task re-dials every configured provider to refill it (floored at 30s; a round that saw a failure comes back on `model_discovery_negative_ttl_seconds` instead). While it is above `0`, `GET /v1/models`, `/v1/models/discoverable` and `/v1/providers/health` answer from that cache rather than dialing on the request path, and so does `GET /v1/models/{model_id}`, which never dials at all. The one read that still dials is the first one to ask about a provider that has never been dialed (a freshly started worker whose background refresh has not landed yet), so a cold worker reports what a provider actually serves instead of claiming it has no models. Setting this to `0` disables caching: reads then dial for themselves, and no background refresher runs. Force a live re-dial with `?refresh=true` on `/v1/models/discoverable` or `/v1/providers/health`; a health check within a few seconds of the last dial reuses it rather than starting another. |
 | `model_discovery_timeout_seconds` | float | `10.0` | Per-provider timeout for a live model-discovery (`list_models`) call. Bounds how long an unreachable or slow provider can stall discovery before it is treated as failed. |
-| `model_discovery_negative_ttl_seconds` | float | `30.0` | How long a failed model-discovery result is remembered before the provider is dialed again, so an unreachable provider is not re-tried on every request (`0` disables negative caching). |
+| `model_discovery_negative_ttl_seconds` | float | `30.0` | How long a failed model-discovery result is remembered before that provider is dialed again (`0` disables negative caching). This governs a read that dials: a cold provider, or any read while `model_cache_ttl_seconds` is `0`. Once the background refresher owns the dialing, how quickly a recovered provider reappears is bounded by `model_cache_ttl_seconds` (the refresh interval) rather than by this. |
 | `models_dev_metadata` | bool | `true` | Enrich the dashboard's model detail with metadata (modalities, capabilities, knowledge cutoff) fetched from the public models.dev catalog. Set `false` to disable the outbound call; the gateway then falls back to the bundled genai-prices data. |
-| `models_dev_cache_ttl_seconds` | int | `86400` | TTL in seconds for the cached models.dev catalog (`0` disables caching). |
+| `models_dev_cache_ttl_seconds` | int | `86400` | TTL for the cached models.dev catalog, and the interval at which a background task refetches it (floored at 5 minutes). While it is above `0`, `GET /v1/models/metadata` answers from that cache rather than waiting on the fetch. A failed fetch is held for one minute, not for the refresh interval, so a transient models.dev outage costs a minute of enrichment rather than a day. `0` disables caching, which means every read fetches instead. |
 | `files_enabled` | bool | `true` | Enable the `/v1/files` upload/storage endpoints (standalone mode). |
 | `files_backend` | string | `"local"` | Blob backend for uploaded file bytes (`"local"` filesystem for now). |
 | `files_local_dir` | string | `"./otari-files"` | Directory the `local` files backend writes uploaded bytes to. |

docs/public/openapi.json

@@ -1536,6 +1536,18 @@
       "DiscoverableProvider": {
         "description": "One provider instance's discovery result.",
         "properties": {
+          "checked_at": {
+            "anyOf": [
+              {
+                "type": "string"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "description": "When this instance was last dialed, ISO 8601. Null when it has not been checked yet, which is what the first read after a restart sees while the background refresh runs.",
+            "title": "Checked At"
+          },
           "discovery_unsupported": {
             "default": false,
             "description": "True when discovery failed only because this backend serves no model-listing endpoint. The provider may still handle requests for models declared in config.",
@@ -6121,6 +6133,17 @@
             ],
             "title": "Api Key Id"
           },
+          "api_key_name": {
+            "anyOf": [
+              {
+                "type": "string"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "title": "Api Key Name"
+          },
           "attempt_count": {
             "anyOf": [
               {
@@ -6375,6 +6398,17 @@
             ],
             "title": "Total Tokens"
           },
+          "user_alias": {
+            "anyOf": [
+              {
+                "type": "string"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "title": "User Alias"
+          },
           "user_id": {
             "anyOf": [
               {
@@ -6477,6 +6511,17 @@
             ],
             "title": "Key"
           },
+          "label": {
+            "anyOf": [
+              {
+                "type": "string"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "title": "Label"
+          },
           "requests": {
             "title": "Requests",
             "type": "integer"
@@ -9469,8 +9514,22 @@
     },
     "/v1/models/discoverable": {
       "get": {
-        "description": "List every model the configured provider credentials can reach.\n\nOperator-facing counterpart to GET /v1/models, which serves a curated catalog\nto API callers. This reports each provider separately and keeps its error, so\na provider with a bad key is distinguishable from one with no models. It is\nmaster-key gated because a provider error message describes the gateway's own\nconfiguration.",
+        "description": "List every model the configured provider credentials can reach.\n\nOperator-facing counterpart to GET /v1/models, which serves a curated catalog\nto API callers. This reports each provider separately and keeps its error, so\na provider with a bad key is distinguishable from one with no models. It is\nmaster-key gated because a provider error message describes the gateway's own\nconfiguration.\n\nAnswers from the discovery cache, which a background refresher keeps warm, so\nthe call does not wait on a slow or unreachable provider. Each provider\ncarries the ``checked_at`` its result was produced at; a null one has not been\ndialed yet. Pass ``refresh=true`` to force a live re-dial of every provider.",
         "operationId": "list_discoverable_models_v1_models_discoverable_get",
+        "parameters": [
+          {
+            "description": "Re-dial every provider instead of answering from the discovery cache.",
+            "in": "query",
+            "name": "refresh",
+            "required": false,
+            "schema": {
+              "default": false,
+              "description": "Re-dial every provider instead of answering from the discovery cache.",
+              "title": "Refresh",
+              "type": "boolean"
+            }
+          }
+        ],
         "responses": {
           "200": {
             "content": {
@@ -9481,6 +9540,16 @@
               }
             },
             "description": "Successful Response"
+          },
+          "422": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            },
+            "description": "Validation Error"
           }
         },
         "security": [
@@ -9499,7 +9568,7 @@
     },
     "/v1/models/metadata": {
       "get": {
-        "description": "Per-model metadata for the dashboard's detail view, from models.dev.\n\nCovers every model models.dev lists under a configured provider, keyed by the\n``instance:model`` selector the dashboard uses. ``available`` is false when\nenrichment is disabled (``models_dev_metadata``) or models.dev could not be\nreached; the response is then empty and the UI falls back to bundled data.\nMaster-key gated: it describes the gateway's configured providers.",
+        "description": "Per-model metadata for the dashboard's detail view, from models.dev.\n\nCovers every model models.dev lists under a configured provider, keyed by the\n``instance:model`` selector the dashboard uses. ``available`` is false when\nenrichment is disabled (``models_dev_metadata``) or models.dev could not be\nreached; the response is then empty and the UI falls back to bundled data.\nMaster-key gated: it describes the gateway's configured providers.\n\nAnswers from the cached catalog, kept warm by a background refresher, so the\ndashboard never waits on the models.dev fetch timeout.",
         "operationId": "list_model_metadata_v1_models_metadata_get",
         "responses": {
           "200": {

docs/public/otari.postman_collection.json

@@ -1258,7 +1258,7 @@
         {
           "name": "List Discoverable Models",
           "request": {
-            "description": "List every model the configured provider credentials can reach.\n\nOperator-facing counterpart to GET /v1/models, which serves a curated catalog\nto API callers. This reports each provider separately and keeps its error, so\na provider with a bad key is distinguishable from one with no models. It is\nmaster-key gated because a provider error message describes the gateway's own\nconfiguration.",
+            "description": "List every model the configured provider credentials can reach.\n\nOperator-facing counterpart to GET /v1/models, which serves a curated catalog\nto API callers. This reports each provider separately and keeps its error, so\na provider with a bad key is distinguishable from one with no models. It is\nmaster-key gated because a provider error message describes the gateway's own\nconfiguration.\n\nAnswers from the discovery cache, which a background refresher keeps warm, so\nthe call does not wait on a slow or unreachable provider. Each provider\ncarries the ``checked_at`` its result was produced at; a null one has not been\ndialed yet. Pass ``refresh=true`` to force a live re-dial of every provider.",
             "header": [],
             "method": "GET",
             "url": {
@@ -1270,14 +1270,22 @@
                 "models",
                 "discoverable"
               ],
-              "raw": "{{baseUrl}}/v1/models/discoverable"
+              "query": [
+                {
+                  "description": "Re-dial every provider instead of answering from the discovery cache.",
+                  "disabled": true,
+                  "key": "refresh",
+                  "value": ""
+                }
+              ],
+              "raw": "{{baseUrl}}/v1/models/discoverable?refresh="
             }
           }
         },
         {
           "name": "List Model Metadata",
           "request": {
-            "description": "Per-model metadata for the dashboard's detail view, from models.dev.\n\nCovers every model models.dev lists under a configured provider, keyed by the\n``instance:model`` selector the dashboard uses. ``available`` is false when\nenrichment is disabled (``models_dev_metadata``) or models.dev could not be\nreached; the response is then empty and the UI falls back to bundled data.\nMaster-key gated: it describes the gateway's configured providers.",
+            "description": "Per-model metadata for the dashboard's detail view, from models.dev.\n\nCovers every model models.dev lists under a configured provider, keyed by the\n``instance:model`` selector the dashboard uses. ``available`` is false when\nenrichment is disabled (``models_dev_metadata``) or models.dev could not be\nreached; the response is then empty and the UI falls back to bundled data.\nMaster-key gated: it describes the gateway's configured providers.\n\nAnswers from the cached catalog, kept warm by a background refresher, so the\ndashboard never waits on the models.dev fetch timeout.",
             "header": [],
             "method": "GET",
             "url": {

src/gateway/api/routes/models.py

@@ -18,10 +18,12 @@
 from gateway.services.model_access import is_model_allowed, resolve_request_allowlist
 from gateway.services.model_catalog_service import (
     ModelCatalogEntry,
+    background_catalog_enabled,
     build_metadata_map,
     load_models_dev_catalog,
 )
 from gateway.services.model_discovery_service import (
+    background_discovery_enabled,
     discover_all_models,
     discover_models_with_status,
     get_model_cache,
@@ -120,6 +122,13 @@ class DiscoverableProvider(BaseModel):
             "The provider may still handle requests for models declared in config."
         ),
     )
+    checked_at: str | None = Field(
+        default=None,
+        description=(
+            "When this instance was last dialed, ISO 8601. Null when it has not been checked yet, "
+            "which is what the first read after a restart sees while the background refresh runs."
+        ),
+    )
     models: list[DiscoverableModel]
 
 
@@ -432,7 +441,16 @@ async def list_models(
     # Phase 1: auto-discovered models from upstream providers.
     if config.model_discovery:
         try:
-            discovered = await discover_all_models(config, provider_filter=provider)
+            # Cache-only when a refresher owns the dialing. This endpoint is
+            # reachable with any API key, so it deliberately has no ``refresh``
+            # escape hatch: forcing a fanout across every configured provider is
+            # an operator action, and lives on the master-key-gated
+            # /v1/models/discoverable and /v1/providers/health instead.
+            discovered = await discover_all_models(
+                config,
+                provider_filter=provider,
+                serve_stale=background_discovery_enabled(config),
+            )
         except Exception:
             logger.exception("Model discovery failed unexpectedly")
             discovered = []
@@ -535,6 +553,10 @@ def _permitted(model_id: str) -> bool:
 @router.get("/models/discoverable", dependencies=[Depends(verify_master_key)])
 async def list_discoverable_models(
     config: Annotated[GatewayConfig, Depends(get_config)],
+    refresh: Annotated[
+        bool,
+        Query(description="Re-dial every provider instead of answering from the discovery cache."),
+    ] = False,
 ) -> DiscoverableModelsResponse:
     """List every model the configured provider credentials can reach.
 
@@ -543,14 +565,22 @@ async def list_discoverable_models(
     a provider with a bad key is distinguishable from one with no models. It is
     master-key gated because a provider error message describes the gateway's own
     configuration.
+
+    Answers from the discovery cache, which a background refresher keeps warm, so
+    the call does not wait on a slow or unreachable provider. Each provider
+    carries the ``checked_at`` its result was produced at; a null one has not been
+    dialed yet. Pass ``refresh=true`` to force a live re-dial of every provider.
     """
-    discoveries = await discover_models_with_status(config)
+    serve_stale = background_discovery_enabled(config) and not refresh
+    discoveries = await discover_models_with_status(config, serve_stale=serve_stale, force=refresh)
+    cache = get_model_cache()
     providers = [
         DiscoverableProvider(
             provider=discovery.provider,
             ok=discovery.error is None,
             error=discovery.error,
             discovery_unsupported=discovery.discovery_unsupported,
+            checked_at=checked.isoformat() if (checked := cache.checked_at(discovery.provider)) else None,
             models=sorted(
                 (
                     DiscoverableModel(id=model.id, key=f"{discovery.provider}:{model.id}")
@@ -577,8 +607,11 @@ async def list_model_metadata(
     enrichment is disabled (``models_dev_metadata``) or models.dev could not be
     reached; the response is then empty and the UI falls back to bundled data.
     Master-key gated: it describes the gateway's configured providers.
+
+    Answers from the cached catalog, kept warm by a background refresher, so the
+    dashboard never waits on the models.dev fetch timeout.
     """
-    catalog = await load_models_dev_catalog(config)
+    catalog = await load_models_dev_catalog(config, serve_stale=background_catalog_enabled(config))
     entries = build_metadata_map(config, catalog)
     return ModelMetadataResponse(
         available=catalog is not None,
@@ -626,16 +659,28 @@ async def get_model(
     )
     pricing = (await db.execute(stmt)).scalar_one_or_none()
 
-    # Check the discovery cache for this model (respecting TTL).
-    # Parse provider from model_id ("provider:model_name") for a targeted lookup
-    # instead of scanning all cached providers.
+    # Check the discovery cache for this model. Parse provider from model_id
+    # ("provider:model_name") for a targeted lookup instead of scanning all
+    # cached providers.
+    #
+    # Read stale-tolerantly, matching the listing above, because nothing on the
+    # request path renews ``cached_at`` any more: the refresher sleeps its
+    # interval *after* each round finishes, so an entry stored at T is already
+    # expired when the next round starts and stays expired until that round's
+    # dials complete. A TTL-bounded peek here would 404 a model that GET
+    # /v1/models is listing in the same instant, for any provider model with no
+    # pricing row and no genai-prices fallback. This endpoint never dials, so
+    # serving the last known answer is the only way to agree with the listing.
     discovered_model = None
     discovered_provider = None
     if config.model_discovery and ":" in model_id:
         provider_prefix, model_name = model_id.split(":", 1)
         cache = get_model_cache()
-        ttl = config.model_cache_ttl_seconds
-        cached_models = cache.get(provider_prefix, ttl)
+        if background_discovery_enabled(config):
+            stale = cache.stale(provider_prefix)
+            cached_models = stale.models if stale is not None and stale.error is None else None
+        else:
+            cached_models = cache.get(provider_prefix, config.model_cache_ttl_seconds)
         if cached_models is not None:
             for model in cached_models:
                 if model.id == model_name: