Feat/multi backend engine

Makefile

@@ -6,10 +6,11 @@ PY   := $(VENV)/bin/python
 PYTEST := $(PY) -m pytest
 
 COMPOSE := docker compose -f deploy/docker-compose.yaml
+COMPOSE_MIXED := docker compose -f deploy/docker-compose.mixed.yaml
 
 .PHONY: help test test-backend test-router test-schema \
         dev-backend dev-frontend build-frontend install-frontend \
-        up down logs ps build
+        up down logs ps build up-mixed down-mixed logs-mixed
 
 help:
 	@echo "Targets:"
@@ -26,6 +27,10 @@ help:
 	@echo "  logs            Tail logs from all services"
 	@echo "  ps              Show service status"
 	@echo "  build           Build images without starting"
+	@echo "  --- mixed-engine HA (deploy/docker-compose.mixed.yaml) ---"
+	@echo "  up-mixed        vLLM + SGLang backends sharing one Postgres/router/dashboard"
+	@echo "  down-mixed      Stop + remove the mixed stack"
+	@echo "  logs-mixed      Tail logs from the mixed stack"
 
 up:
 	$(COMPOSE) up -d --build
@@ -42,6 +47,16 @@ ps:
 build:
 	$(COMPOSE) build
 
+# --- mixed-engine HA deployment (vLLM + SGLang backends, shared Postgres) ---
+up-mixed:
+	$(COMPOSE_MIXED) up -d --build
+
+down-mixed:
+	$(COMPOSE_MIXED) down
+
+logs-mixed:
+	$(COMPOSE_MIXED) logs -f
+
 test: test-backend test-router test-schema
 
 test-backend:

README.md

@@ -29,6 +29,7 @@ becomes a routable model; the router load-balances across instances; and a bundl
 ## Highlights
 
 - **One router controls the whole fleet** — a single OpenAI- & Anthropic-compatible origin fronts every model. Route by the `model` field across `/v1/chat/completions`, `/v1/messages`, `/v1/embeddings`, `/v1/rerank`, `/v1/score`, `/tokenize` and more; the router resolves the group and load-balances its instances, so clients never address an instance directly.
+- **Two inference engines, one control plane — vLLM + SGLang** — choose the engine per model (the *Add Model* dialog has an engine selector); an engine-aware scheduler places each model on a backend that can run it, and the same router / dashboard / monitoring front both. Run a **vLLM-only** stack (`make up`) or a **mixed vLLM + SGLang** fleet (`make up-mixed`). See [docs/mixed-engine-deployment.md](docs/mixed-engine-deployment.md).
 - **Add a model by pasting `vllm serve …`** — parsed into a form and layered on as a dynamic overlay; the router hot-reloads, no `config.yaml` edits.
 - **Lifecycle + self-healing** — per-instance state machine (`stopped → starting → ready → sleeping → failed`), VRAM pre-flight guard, GPU auto-placement, crash auto-restart with backoff.
 - **Autoscaling with a warm-standby tier** — per group, keep `min_ready` replicas warm and scale up on queue depth (wake first, else cold-start) to `max_ready`; fold idle replicas back down `ready → sleep → stop`. vLLM **sleep mode** (level-1) frees a replica's VRAM but wakes in seconds, so scaling down needn't mean a minute-long cold start. Set it from config.yaml or the dashboard; a live Grafana dashboard + alerts are bundled.
@@ -59,6 +60,11 @@ make up                              # build + start the whole stack
 
 `make down` stops it · `make logs` tails all services · `make ps` shows status.
 
+**Two deployment modes:**
+
+- **`make up`** — the default **vLLM-only** stack.
+- **`make up-mixed`** — a **vLLM + SGLang** fleet: a vLLM backend and a SGLang backend sharing one Postgres, router, dashboard and Grafana. Add a SGLang model from *Add Model → engine: `sglang`* and it is auto-placed on the SGLang backend. `make down-mixed` / `make logs-mixed` manage it. See [docs/mixed-engine-deployment.md](docs/mixed-engine-deployment.md).
+
 ```bash
 curl http://localhost:8887/v1/models     # router: configured model groups
 curl http://localhost:5000/api/models    # backend: lifecycle state of each instance
@@ -100,6 +106,8 @@ Request/response shapes and auth details are in [docs/API.md](docs/API.md).
 
 ## Architecture
 
+### vLLM-only (`make up`)
+
 ```mermaid
 flowchart LR
     Client([Clients])
@@ -131,11 +139,56 @@ The **router only routes** — the **backend owns model lifecycle**. The fronten
 backend, and Grafana sit behind nginx on a single origin; backend, router, and Prometheus
 share one network namespace so the spawned vLLM instances are reachable on `localhost`.
 
+### Mixed vLLM + SGLang (`make up-mixed`)
+
+Each engine runs as its own backend container (they can't share a netns), sharing one
+Postgres (scheduling / desired intent), one router, one dashboard and one monitoring stack.
+Each backend publishes its ready instances as **routable addresses** to a shared file_sd that
+Prometheus scrapes. See [docs/mixed-engine-deployment.md](docs/mixed-engine-deployment.md).
+
+```mermaid
+flowchart LR
+    Client([Clients])
+    FE["<b>frontend</b><br/>nginx · :8884"]
+    GF["<b>grafana</b><br/>/grafana"]
+    PG[("<b>postgres</b><br/>shared store · scheduling/desired")]
+    PR["<b>prometheus</b> · :9090<br/>scrapes routable file_sd targets"]
+    RT["<b>router</b> · :8887<br/>OpenAI-compatible LB"]
+
+    subgraph vbe["vLLM backend (engine.Dockerfile)"]
+        BV["<b>backend</b> · :5071<br/>NODE_ENGINES=vllm"]
+        VINS["vLLM instances"]
+    end
+    subgraph sbe["SGLang backend (engine-sglang.Dockerfile)"]
+        BS["<b>backend</b> · :5072<br/>NODE_ENGINES=sglang"]
+        SINS["SGLang instances"]
+    end
+
+    Client --> FE
+    FE -->|/api| BV
+    FE -->|/v1| RT
+    FE -->|/grafana| GF
+    BV -->|launch| VINS
+    BS -->|launch| SINS
+    RT -->|route| VINS
+    RT -->|route| SINS
+    BV <-->|leader/schedule| PG
+    BS <-->|converge desired| PG
+    PR -->|scrape| VINS
+    PR -->|scrape| SINS
+    GF -->|query| PR
+```
+
+The leader's **engine-aware scheduler** places each model on a backend that can run its engine;
+a control action landing on the wrong node is deferred to the owning one. SGLang serves
+OpenMetrics, so Prometheus stores its metrics as `sglang_*` (underscore) while vLLM keeps colons.
+
 ## Documentation
 
 | Topic | |
 |---|---|
 | Deployment & topology | [docs/deployment.md](docs/deployment.md) |
+| Mixed-engine (vLLM + SGLang) | [docs/mixed-engine-deployment.md](docs/mixed-engine-deployment.md) |
 | Configuration (`config.yaml`) | [docs/configuration.md](docs/configuration.md) |
 | Features in depth | [docs/features.md](docs/features.md) |
 | Monitoring (Prometheus + Grafana) | [docs/monitoring.md](docs/monitoring.md) |

README_zh-CN.md

@@ -30,6 +30,7 @@
 ## 功能亮點
 
 - **一個 router 掌控整個集群** — 單一 OpenAI 與 Anthropic 相容入口統管所有模型。以 `model` 欄位路由 `/v1/chat/completions`、`/v1/messages`、`/v1/embeddings`、`/v1/rerank`、`/v1/score`、`/tokenize` 等端點；router 自動解析群組並在實例間負載平衡，客戶端永遠不直接連到單一實例。
+- **兩種推理引擎、同一個控制平面 — vLLM + SGLang** — 每顆模型可各自選引擎（*新增模型*對話框有引擎選擇器）；engine-aware 排程器把每顆模型擺到「跑得動它」的 backend 上，並由同一個 router／控制台／監控統一前置。可只跑 **vLLM**（`make up`），或跑 **vLLM + SGLang 混合**集群（`make up-mixed`）。見 [docs/mixed-engine-deployment_zh-CN.md](docs/mixed-engine-deployment_zh-CN.md)。
 - **貼上 `vllm serve …` 即可新增模型** — 解析成表單、以動態 overlay 疊加；router 熱重載。
 - **生命週期** — 每實例狀態機（`stopped → starting → ready → sleeping → failed`）、VRAM 預檢防呆、GPU 自動擺放、崩潰指數退避自動重啟。
 - **自動擴縮（含暖待命層）** — 每群組保留 `min_ready` 暖機副本，依佇列深度擴容（優先喚醒、其次冷啟）到 `max_ready`；閒置時逐階縮回 `ready → sleep → stop`。vLLM **sleep mode**（level-1）釋放副本 VRAM 但秒級喚醒，所以縮容不必付出數分鐘冷啟代價。config.yaml 或控制台皆可設定，內建即時 Grafana 面板與告警。
@@ -59,6 +60,11 @@ make up                              # 建置並啟動整套服務
 
 `make down` 停止 · `make logs` 追蹤所有服務日誌 · `make ps` 看狀態。
 
+**兩種啟動方式：**
+
+- **`make up`** — 預設的**純 vLLM** 集群。
+- **`make up-mixed`** — **vLLM + SGLang** 混合集群：一個 vLLM backend 與一個 SGLang backend 共用同一顆 Postgres、router、控制台與 Grafana。從 *新增模型 → 引擎：`sglang`* 新增的模型會自動擺到 SGLang backend。對應 `make down-mixed`／`make logs-mixed`。見 [docs/mixed-engine-deployment_zh-CN.md](docs/mixed-engine-deployment_zh-CN.md)。
+
 ```bash
 curl http://localhost:8887/v1/models     # router：列出設定的模型群組
 curl http://localhost:5000/api/models    # 後端：每個實例的生命週期狀態
@@ -98,6 +104,8 @@ curl http://localhost:8887/v1/chat/completions \
 
 ## 架構
 
+### 純 vLLM（`make up`）
+
 ```mermaid
 flowchart LR
     Client([Clients 用戶端])
@@ -129,11 +137,54 @@ flowchart LR
 Grafana 都在 nginx 之後以單一來源對外；backend、router、Prometheus 共用一個 network
 namespace，所以被拉起的 vLLM 實例可在 `localhost` 互相連到。
 
+### vLLM + SGLang 混合（`make up-mixed`）
+
+兩個引擎各跑一個 backend 容器（無法共用 netns），共用一顆 Postgres（排程／desired 意圖）、
+一個 router、一個 dashboard 與一套監控；各 backend 把自己 ready 的實例以**可路由位址**寫進
+共享 file_sd，Prometheus 一起抓。見 [docs/mixed-engine-deployment_zh-CN.md](docs/mixed-engine-deployment_zh-CN.md)。
+
+```mermaid
+flowchart LR
+    Client([Clients 用戶端])
+    FE["<b>frontend</b><br/>nginx · :8884"]
+    GF["<b>grafana</b><br/>/grafana"]
+    PG[("<b>postgres</b><br/>共用 store · 排程/desired")]
+    PR["<b>prometheus</b> · :9090<br/>抓可路由 file_sd targets"]
+    RT["<b>router</b> · :8887<br/>OpenAI 相容負載平衡"]
+
+    subgraph vbe["vLLM backend (engine.Dockerfile)"]
+        BV["<b>backend</b> · :5071<br/>NODE_ENGINES=vllm"]
+        VINS["vLLM 實例"]
+    end
+    subgraph sbe["SGLang backend (engine-sglang.Dockerfile)"]
+        BS["<b>backend</b> · :5072<br/>NODE_ENGINES=sglang"]
+        SINS["SGLang 實例"]
+    end
+
+    Client --> FE
+    FE -->|/api| BV
+    FE -->|/v1| RT
+    FE -->|/grafana| GF
+    BV -->|拉起| VINS
+    BS -->|拉起| SINS
+    RT -->|路由| VINS
+    RT -->|路由| SINS
+    BV <-->|leader/排程| PG
+    BS <-->|收斂 desired| PG
+    PR -->|scrape| VINS
+    PR -->|scrape| SINS
+    GF -->|查詢| PR
+```
+
+leader 的 **engine-aware 排程器**把每顆模型擺到「跑得動它引擎」的 backend；落錯 node 的控制
+動作會延後給擁有者執行。SGLang 走 OpenMetrics，指標入庫為底線的 `sglang_*`，vLLM 則保留冒號。
+
 ## 文件
 
 | 主題 | |
 |---|---|
 | 部署與架構 | [docs/deployment_zh-CN.md](docs/deployment_zh-CN.md) |
+| 混合引擎（vLLM + SGLang） | [docs/mixed-engine-deployment_zh-CN.md](docs/mixed-engine-deployment_zh-CN.md) |
 | 配置（`config.yaml`） | [docs/configuration_zh-CN.md](docs/configuration_zh-CN.md) |
 | 功能特色（詳細） | [docs/features_zh-CN.md](docs/features_zh-CN.md) |
 | 監控（Prometheus + Grafana） | [docs/monitoring_zh-CN.md](docs/monitoring_zh-CN.md) |

apps/backend/app/api/models.py

@@ -23,13 +23,16 @@
     SleepError,
     VRAMInsufficient,
 )
-from app.services.vllm_command import parse_vllm_command
+from app.services.vllm_command import parse_command as parse_engine_command
 
 router = APIRouter(prefix="/models", tags=["models"])
 
 
 class ParseRequest(BaseModel):
     command: str
+    # Which engine's CLI to parse ('vllm' | 'sglang'). Omitted = sniff from the
+    # command (sglang.launch_server -> sglang, else vLLM).
+    engine: Optional[str] = None
 
 
 class InstanceSpec(BaseModel):
@@ -52,20 +55,18 @@ class CreateModelRequest(BaseModel):
 
 @router.get("", response_model=list[ModelView])
 async def list_models(request: Request, manager: ModelManager = Depends(get_manager)):
-    # HA Phase 3d: a non-leader replica reports the fleet from the shared store's
-    # observed state (the leader/owning agents backfill it), since its own registry
-    # is idle. The leader (and a single-host collapsed deploy) uses its live
-    # registry — identical to before.
-    elector = getattr(request.app.state, "leader", None)
-    prefer_store = elector is not None and not elector.is_leader
-    return [ModelView(**v) for v in await manager.fleet_views(prefer_store=prefer_store)]
+    # HA: in Postgres/multi-node mode the fleet view comes from the shared store
+    # (each node backfills its *owned* observed state) — on leader and follower
+    # alike, since with per-node actuation (Phase 7) no single registry is complete.
+    # SQLite collapsed: the local registry is the truth — identical to before.
+    return [ModelView(**v) for v in await manager.fleet_views(prefer_store=manager.prefer_store_view())]
 
 
 @router.post("/parse", dependencies=[Depends(require_operator)])
 async def parse_command(body: ParseRequest, manager: ModelManager = Depends(get_manager)):
-    """Parse a pasted vLLM command into editable fields + conflict hints."""
+    """Parse a pasted vLLM / SGLang command into editable fields + conflict hints."""
     try:
-        parsed = parse_vllm_command(body.command)
+        parsed = parse_engine_command(body.command, body.engine)
     except ValueError as e:
         raise HTTPException(status.HTTP_400_BAD_REQUEST, str(e))
     inst = parsed["instance"]
@@ -167,6 +168,13 @@ async def unload_lora(key: str, name: str, manager: ModelManager = Depends(get_m
 
 @router.get("/{key}", response_model=ModelView)
 async def get_model(key: str, manager: ModelManager = Depends(get_manager)):
+    # HA: like list_models, prefer the shared store's observed state in multi-node
+    # mode so a model owned by another node shows its real state (not this node's
+    # idle registry). Falls back to the local registry (collapsed / not in store).
+    if manager.prefer_store_view():
+        for v in await manager.fleet_views(prefer_store=True):
+            if v.get("key") == key:
+                return ModelView(**v)
     try:
         return ModelView.from_instance(await manager.get(key))
     except ModelNotFound:

apps/backend/app/api/observability.py

@@ -9,8 +9,10 @@
 import json
 from typing import Optional
 
+from urllib.parse import quote
+
 from fastapi import APIRouter, Depends, HTTPException, Request, status
-from fastapi.responses import StreamingResponse
+from fastapi.responses import JSONResponse, StreamingResponse
 
 from app.api.deps import get_manager
 from app.api.schemas import ModelView
@@ -62,10 +64,35 @@ async def requests_log(request: Request, model_key: Optional[str] = None, limit:
     return await _store(request).recent_requests(model_key=model_key, limit=limit)
 
 
+async def _proxy_to_owner(request: Request, manager, key: str, suffix: str):
+    """If `key` runs on another node (HA), GET that node's backend API for this
+    node-local data (logs/metrics live as files on the owning node) and relay the
+    JSON. Returns None when the model is local (caller reads locally)."""
+    api_url = await manager.owning_node_api_url(key)
+    if not api_url:
+        return None
+    url = f"{api_url}/api/models/{quote(key, safe='')}/{suffix}"
+    headers = {}
+    auth = request.headers.get("authorization")
+    if auth:
+        headers["authorization"] = auth
+    try:
+        resp = await request.app.state.http_client.get(
+            url, params=dict(request.query_params), headers=headers, timeout=10.0
+        )
+    except Exception:
+        raise HTTPException(status.HTTP_502_BAD_GATEWAY,
+                            f"failed to reach owning node for {key}")
+    return JSONResponse(status_code=resp.status_code, content=resp.json())
+
+
 @router.get("/models/{key}/logs")
 async def model_logs(
-    key: str, tail: int = 200, manager: ModelManager = Depends(get_manager)
+    request: Request, key: str, tail: int = 200, manager: ModelManager = Depends(get_manager)
 ):
+    proxied = await _proxy_to_owner(request, manager, key, "logs")
+    if proxied is not None:
+        return proxied
     try:
         inst = await manager.get(key)
     except ModelNotFound:
@@ -77,11 +104,14 @@ async def model_logs(
 
 
 @router.get("/models/{key}/metrics")
-async def model_metrics(key: str, manager: ModelManager = Depends(get_manager)):
-    """vLLM startup capacity/memory/compile metrics parsed from the engine log.
+async def model_metrics(request: Request, key: str, manager: ModelManager = Depends(get_manager)):
+    """vLLM/SGLang startup capacity/memory/compile metrics parsed from the engine log.
 
     Only meaningful once the instance is READY (the metrics are printed at the end
     of model loading); returns {ready: false} otherwise so the UI hides the panel."""
+    proxied = await _proxy_to_owner(request, manager, key, "metrics")
+    if proxied is not None:
+        return proxied
     try:
         inst = await manager.get(key)
     except ModelNotFound:
@@ -94,17 +124,22 @@ async def model_metrics(key: str, manager: ModelManager = Depends(get_manager)):
     return {"ready": True, **parse_startup_metrics(head)}
 
 
-async def model_snapshot_stream(registry, interval: float = 1.0, heartbeat_every: float = 15.0):
+async def model_snapshot_stream(manager, interval: float = 1.0, heartbeat_every: float = 15.0):
     """SSE generator: emit the full model snapshot whenever it changes.
 
-    Cheap snapshot-diff (no pub/sub bus): re-snapshot the registry each interval,
-    emit on change, and send a heartbeat comment if nothing changed for a while.
+    Cheap snapshot-diff (no pub/sub bus): re-read the fleet each interval, emit on
+    change, send a heartbeat if nothing changed for a while. In HA/multi-node mode
+    the view comes from the shared store (each node only actuates its own models, so
+    the local registry alone is incomplete — Phase 7); collapsed uses the registry.
     """
     last_sig: Optional[str] = None
     since_emit = 0.0
     while True:
-        snap = await registry.snapshot()
-        payload = [ModelView.from_instance(i).model_dump(mode="json") for i in snap]
+        payload = sorted(
+            (ModelView(**v).model_dump(mode="json")
+             for v in await manager.fleet_views(prefer_store=manager.prefer_store_view())),
+            key=lambda m: m["key"],
+        )
         sig = json.dumps(payload, sort_keys=True, default=str)
         if sig != last_sig:
             last_sig = sig
@@ -122,6 +157,6 @@ async def model_snapshot_stream(registry, interval: float = 1.0, heartbeat_every
 async def stream_models(request: Request):
     """Live model state via Server-Sent Events (replaces frontend polling)."""
     return StreamingResponse(
-        model_snapshot_stream(request.app.state.registry),
+        model_snapshot_stream(request.app.state.manager),
         media_type="text/event-stream",
     )

apps/backend/app/api/schemas.py

@@ -14,6 +14,8 @@ class ModelView(BaseModel):
 
     key: str
     kind: ModelKind
+    # Which inference engine backs this instance ("vllm" / "sglang" / …).
+    engine: str = "vllm"
     model_tag: Optional[str] = None
     host: str
     port: int
@@ -35,6 +37,7 @@ def from_instance(cls, inst: ModelInstance) -> "ModelView":
         return cls(
             key=inst.key,
             kind=inst.kind,
+            engine=inst.engine,
             model_tag=inst.model_tag,
             host=inst.host,
             port=inst.port,