diff --git a/llamacppllmbasev3/Chart.yaml b/llamacppllmbasev3/Chart.yaml index 141917f88..a3f6cdcd1 100644 --- a/llamacppllmbasev3/Chart.yaml +++ b/llamacppllmbasev3/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v2 -appVersion: b10454 +appVersion: b10588 description: Generic llama.cpp + llm-init base; the GGUF model is supplied at install time via env name: llamacppllmbasev3 type: application -version: 1.2.33 \ No newline at end of file +version: 1.2.37 \ No newline at end of file diff --git a/llamacppllmbasev3/OlaresManifest.yaml b/llamacppllmbasev3/OlaresManifest.yaml index d14f45661..01bf01017 100644 --- a/llamacppllmbasev3/OlaresManifest.yaml +++ b/llamacppllmbasev3/OlaresManifest.yaml @@ -7,9 +7,10 @@ metadata: description: "Generic llama.cpp engine base. Pick any GGUF model at install via env." appid: llamacppllmbasev3 title: llama.cpp Engine Base - version: '1.2.33' + version: '1.2.37' categories: - AI + - models tags: - engine sharedEntrances: @@ -35,27 +36,37 @@ workloadReplicas: llminit: 1 spec: onlyAdmin: true - versionName: 'b10454' + versionName: 'b10588' upgradeDescription: | - Chart 1.2.33 — bump engine to `server-cuda12-b10454` and llm-init to `v1.3.8`. + Chart 1.2.37 — bump engine to `server-cuda12-b10588`. **Images** - - llama.cpp: `docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10362` → `server-cuda12-b10454` (latest published beclab CUDA 12 server tag; amd64 + arm64). Upstream GitHub is already at b10456; no beclab b10455/b10456 image yet. - - llm-init: `docker.io/beclab/llm-init:v1.3.5` → `v1.3.8`. + - llama.cpp: `docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10454` → `server-cuda12-b10588` (amd64 + arm64). + - llm-init: unchanged at `v1.5.0`. - **Muse Glimmer 30B (Meta)** — requires **b10353+**; **b10454 includes** the `muse-glimmer` architecture ([llama.cpp #26841](https://github.com/ggml-org/llama.cpp/pull/26841)). Older builds (e.g. b10331) fail with `unknown model architecture: 'muse-glimmer'`. + 134 upstream commits; core libraries move to llama.cpp 0.2.0 / ggml 0.21.0. - Clone this base and point env at [meta-models/Muse-Glimmer-30B-GGUF](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF) or community quants (e.g. unsloth). Typical layout: - - Main GGUF + `mmproj` in `MODEL_SOURCE` (comma-separated) for vision - - DFlash drafter (`dflash-kquant.gguf`) + `--spec-type draft-dflash` or `draft-dspark` in `ENGINE_ARGS` for speculative decode + **New model architectures** — clone this base and point `MODEL_SOURCE` / `MODEL_NAME` at the GGUF: + - BailingMoE3 ([#26608](https://github.com/ggml-org/llama.cpp/pull/26608)), with DSpark speculative decode ([#27508](https://github.com/ggml-org/llama.cpp/pull/27508)) + - Granite SWA / GraniteMoE SWA ([#25505](https://github.com/ggml-org/llama.cpp/pull/25505)) + - dots3-note, including vision + audio through `mmproj` ([#27060](https://github.com/ggml-org/llama.cpp/pull/27060), [#27524](https://github.com/ggml-org/llama.cpp/pull/27524)) + - DSpark drafters for LFM2 ([#27383](https://github.com/ggml-org/llama.cpp/pull/27383)) and speculators-format checkpoints ([#26275](https://github.com/ggml-org/llama.cpp/pull/26275)) + - Nemotron 3 Ultra conversion fix ([#27101](https://github.com/ggml-org/llama.cpp/pull/27101)) - **Also in b10362 → b10454:** - - Chat templates receive OpenAI `reasoning_effort` ([llama.cpp #26941](https://github.com/ggml-org/llama.cpp/pull/26941)); Qwen3.8 uses `xhigh` / `medium` / `low` via `--chat-template-kwargs` or the request body - - MTP draft type auto-detect; DFlash / DSpark backend sampling - - MiniMax Text01 / M1 conversion support - - Server `/metrics` and `/slots` stay available during decode; mtmd and jinja fixes + **GPU and performance** + - DGX Spark (`nvidia-gb10`): faster batch-size-1 decode for dense models via MMVQ `nwarps=8` ([#26843](https://github.com/ggml-org/llama.cpp/pull/26843)) + - CUDA MMVQ→MMQ decode crossover is now tuned per GPU and quant type ([#26079](https://github.com/ggml-org/llama.cpp/pull/26079)); cuBLAS handles use a static workspace ([#26574](https://github.com/ggml-org/llama.cpp/pull/26574)) + - Layer auto-fit accounts for `n_streams` ([#27496](https://github.com/ggml-org/llama.cpp/pull/27496)); V is built as a view of K in iSWA attention ([#27392](https://github.com/ggml-org/llama.cpp/pull/27392)) + - Fixes: backend split scheduler race ([#26040](https://github.com/ggml-org/llama.cpp/pull/26040)) and a null-pointer deref in speculative decode ([#27404](https://github.com/ggml-org/llama.cpp/pull/27404)) - See https://github.com/ggml-org/llama.cpp/releases for full upstream release notes. + **Server and API** + - `/metrics` and `/slots` stay reachable while the server is sleeping ([#27376](https://github.com/ggml-org/llama.cpp/pull/27376)) + - Model endpoints become private when `--api-key` authentication is enabled ([#26347](https://github.com/ggml-org/llama.cpp/pull/26347)); this chart runs without an engine API key, so the internal entrance is unaffected + - `--mmproj-device` places the vision projector on a chosen device ([#23255](https://github.com/ggml-org/llama.cpp/pull/23255)); WebP inputs are accepted via ffmpeg ([#27520](https://github.com/ggml-org/llama.cpp/pull/27520)) + - Embedding output combined with draft-MTP is fixed ([#27400](https://github.com/ggml-org/llama.cpp/pull/27400)) — relevant to `MODEL_MODE=embedding` + - mtmd fixes for LFM2 image tiling, DeepSeek-OCR and Granite preprocessing; refreshed built-in WebUI + + See https://github.com/ggml-org/llama.cpp/compare/b10454...b10588 for the full upstream diff. fullDescription: | **IMPORTANT NOTE** This app is a template and cannot be used on its own. To use it, set the environment variables below to connect a specific model. diff --git a/llamacppllmbasev3/owners b/llamacppllmbasev3/owners index 48e63a537..54214d082 100644 --- a/llamacppllmbasev3/owners +++ b/llamacppllmbasev3/owners @@ -1,7 +1,3 @@ owners: -- 'LittleLollipop' -- 'TShentu' -- 'hysyeah' -- 'pengpeng' -- 'harveyff' -- 'FantasticCode2019' + - username: '@beclab' + title: 'Olares' diff --git a/llamacppllmbasev3/templates/llamacpp.yaml b/llamacppllmbasev3/templates/llamacpp.yaml index 10c95802b..7640a301f 100644 --- a/llamacppllmbasev3/templates/llamacpp.yaml +++ b/llamacppllmbasev3/templates/llamacpp.yaml @@ -73,12 +73,14 @@ spec: defaultMode: 0555 containers: - name: llamacpp - image: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10454 + image: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10588 imagePullPolicy: IfNotPresent command: ["/bin/sh", "/llm-init/wrappers/llamacpp.sh"] env: - name: ENGINE_KIND value: llamacpp + - name: WRAPPER_COMMON + value: /run/llm-init/wrappers/common.sh - name: MODEL_NAME value: "{{ $modelName }}" - name: MODEL_MODE diff --git a/llamacppllmbasev3/templates/llm-init.yaml b/llamacppllmbasev3/templates/llm-init.yaml index 2ec8b8931..d069c644f 100644 --- a/llamacppllmbasev3/templates/llm-init.yaml +++ b/llamacppllmbasev3/templates/llm-init.yaml @@ -109,7 +109,7 @@ spec: # v1.2.4 defaults to the stable LFS download path (hf_xet disabled # unless HF_ENABLE_XET=true); LFS streams each file to disk with a # bounded RAM footprint, avoiding the hf_xet OOM (huggingface_hub#3300). - image: docker.io/beclab/llm-init:v1.3.8 + image: docker.io/beclab/llm-init:v1.5.0 imagePullPolicy: IfNotPresent env: - name: ENGINE_KIND diff --git a/llamacppllmbasev3/templates/wrappers.yaml b/llamacppllmbasev3/templates/wrappers.yaml index 87c5b5615..7f94d1752 100644 --- a/llamacppllmbasev3/templates/wrappers.yaml +++ b/llamacppllmbasev3/templates/wrappers.yaml @@ -1,8 +1,8 @@ # Engine wrapper script, mounted into the llama.cpp (engine) container at -# /llm-init/wrappers. Inlined from llm-init deploy/wrappers with Olares -# chart specifics (-hf "$MODEL_NAME"). Prefers ${RUN_DIR}/engine_args from -# Model Console model-card SSOT; supervise_engine relaunches on -# engine_restart bumps. +# /llm-init/wrappers. Sources the helpers llm-init publishes into RUN_DIR +# with Olares chart specifics (-hf "$MODEL_NAME"). Prefers +# ${RUN_DIR}/engine_args from the Model Console model-card SSOT; +# supervise_engine relaunches on engine_restart bumps. apiVersion: v1 kind: ConfigMap metadata: @@ -16,29 +16,19 @@ data: set -eu ENGINE=llamacpp - SENTINEL=${SENTINEL:-/run/llm-init/model_download_finish} - ENGINE_ARGS_FILE=${ENGINE_ARGS_FILE:-/run/llm-init/engine_args} - ENGINE_RESTART_FILE=${ENGINE_RESTART_FILE:-/run/llm-init/engine_restart} - WAIT_INTERVAL=${WAIT_INTERVAL:-5} - WAIT_TIMEOUT=${WAIT_TIMEOUT:-3600} - ENGINE_ARGS_WAIT_TIMEOUT=${ENGINE_ARGS_WAIT_TIMEOUT:-60} - RESTART_POLL_INTERVAL=${RESTART_POLL_INTERVAL:-2} - ENGINE_STOP_GRACE_SECONDS=${ENGINE_STOP_GRACE_SECONDS:-15} - log() { printf '[wrapper-%s] %s\n' "$ENGINE" "$*"; } - - wait_for_engine_args_handoff() { - elapsed=0 - while [ ! -f "$ENGINE_ARGS_FILE" ]; do - if [ "$elapsed" -ge "$ENGINE_ARGS_WAIT_TIMEOUT" ]; then - log "timed out after ${ENGINE_ARGS_WAIT_TIMEOUT}s waiting for $ENGINE_ARGS_FILE; using chart ENGINE_ARGS" - return 0 - fi - log "waiting for $ENGINE_ARGS_FILE (elapsed=${elapsed}s)" - sleep "$WAIT_INTERVAL" - elapsed=$((elapsed + WAIT_INTERVAL)) - done - log "engine_args handoff $ENGINE_ARGS_FILE found after ${elapsed}s" - } + # Source the shared helpers llm-init publishes into RUN_DIR. That copy can + # arrive late - the two containers are separate Deployments with no ordering + # between them - so waiting here is the normal path. + WRAPPER_COMMON=${WRAPPER_COMMON:-"$(dirname "$0")/common.sh"} + _wc=0 + while [ ! -f "$WRAPPER_COMMON" ] && [ "$_wc" -lt "${WRAPPER_COMMON_WAIT:-3600}" ]; do + printf '[wrapper] waiting for %s (elapsed=%ss)\n' "$WRAPPER_COMMON" "$_wc" + sleep 5 + _wc=$((_wc + 5)) + done + [ -f "$WRAPPER_COMMON" ] || { printf '[wrapper] %s never appeared\n' "$WRAPPER_COMMON" >&2; exit 1; } + unset _wc + . "$WRAPPER_COMMON" looks_like_hf_repo_id() { s=$1 @@ -57,15 +47,10 @@ data: esac } - load_engine_args_from_handoff() { - if [ -f "$ENGINE_ARGS_FILE" ]; then - ENGINE_ARGS=$(tr -d '\r\n' < "$ENGINE_ARGS_FILE") - export ENGINE_ARGS - log "loaded ENGINE_ARGS from $ENGINE_ARGS_FILE (${#ENGINE_ARGS} bytes)" - fi - # Upgrade path: hostPath model-spec may already have engine_args - # without --embedding. Handoff would otherwise override the chart - # ENGINE_ARGS that appends the flag. Force it when MODEL_MODE says so. + # Upgrade path: a hostPath model-spec may already carry engine_args without + # --embedding, and the handoff overrides the chart ENGINE_ARGS that appends + # the flag. Force it back when MODEL_MODE says this is an embedding model. + force_embedding_flag() { case "${MODEL_MODE:-}" in embedding) case " ${ENGINE_ARGS:-} " in @@ -80,113 +65,7 @@ data: esac } - read_restart_gen() { - if [ -f "$ENGINE_RESTART_FILE" ]; then - tr -d '\r\n' < "$ENGINE_RESTART_FILE" - else - printf '0' - fi - } - - stop_engine_child() { - child=$1 - if [ -z "$child" ]; then - return 0 - fi - if ! kill -0 "$child" 2>/dev/null; then - wait "$child" 2>/dev/null || true - return 0 - fi - log "stopping engine pid=$child (SIGTERM to process group)" - kill -TERM -"$child" 2>/dev/null || kill -TERM "$child" 2>/dev/null || true - elapsed=0 - while kill -0 "$child" 2>/dev/null; do - if [ "$elapsed" -ge "$ENGINE_STOP_GRACE_SECONDS" ]; then - log "engine pid=$child still alive after ${ENGINE_STOP_GRACE_SECONDS}s; SIGKILL" - kill -KILL -"$child" 2>/dev/null || kill -KILL "$child" 2>/dev/null || true - break - fi - sleep 1 - elapsed=$((elapsed + 1)) - done - wait "$child" 2>/dev/null || true - } - - supervise_engine() { - _supervise_child="" - _supervise_exiting=0 - - _supervise_on_signal() { - _supervise_exiting=1 - log "received stop signal; forwarding to engine" - if [ -n "${_supervise_child}" ]; then - stop_engine_child "$_supervise_child" - fi - exit 143 - } - trap '_supervise_on_signal' TERM INT - - gen=$(read_restart_gen) - while [ "$_supervise_exiting" -eq 0 ]; do - load_engine_args_from_handoff - log "launching engine (restart_gen=$gen)" - # Job control → background job is a process-group leader. - # Do NOT use `setsid CMD`: CMD is the shell function - # run_llamacpp, which setsid cannot execvp (CrashLoop on - # Linux images that ship util-linux setsid). - set -m 2>/dev/null || true - "$@" & - _supervise_child=$! - child=$_supervise_child - restarted=0 - while kill -0 "$child" 2>/dev/null; do - sleep "$RESTART_POLL_INTERVAL" - if [ "$_supervise_exiting" -eq 1 ]; then - break - fi - new=$(read_restart_gen) - if [ "$new" != "$gen" ]; then - log "restart requested (gen $gen -> $new); stopping pid=$child" - stop_engine_child "$child" - _supervise_child="" - gen=$new - restarted=1 - break - fi - done - if [ "$_supervise_exiting" -eq 1 ]; then - break - fi - if [ "$restarted" -eq 1 ]; then - settle=0 - while [ "$settle" -lt 10 ]; do - sleep 1 - settle=$((settle + 1)) - if ! kill -0 "$child" 2>/dev/null; then - break - fi - done - continue - fi - wait "$child" - status=$? - _supervise_child="" - log "engine exited status=$status (no restart requested)" - exit "$status" - done - } - - elapsed=0 - while [ ! -f "$SENTINEL" ]; do - if [ "$elapsed" -ge "$WAIT_TIMEOUT" ]; then - log "timed out after ${WAIT_TIMEOUT}s waiting for $SENTINEL" - exit 1 - fi - log "waiting for $SENTINEL (elapsed=${elapsed}s)" - sleep "$WAIT_INTERVAL" - elapsed=$((elapsed + WAIT_INTERVAL)) - done - log "sentinel $SENTINEL found after ${elapsed}s" + wait_for_sentinel wait_for_engine_args_handoff if ! looks_like_hf_repo_id "${MODEL_NAME:-}"; then @@ -217,6 +96,7 @@ data: # Preserve container CMD extras before ENGINE_ARGS word-splitting. extra_argv="$*" load_engine_args_from_handoff + force_embedding_flag ARGV="" if [ -n "${ENGINE_ARGS:-}" ]; then # shellcheck disable=SC2086 @@ -232,7 +112,7 @@ data: fi log "starting llama.cpp with -hf=$MODEL_NAME alias=${MODEL_NAME:-unset} argv=${ARGV:-} (bin=$LLAMA_SERVER_BIN)" # shellcheck disable=SC2086 - "$LLAMA_SERVER_BIN" \ + exec "$LLAMA_SERVER_BIN" \ -hf "$MODEL_NAME" \ --alias "${MODEL_NAME:-llm-init-model}" \ --host 0.0.0.0 \