diff --git a/docs/backend/wsl.md b/docs/backend/wsl.md index 5bcce3f..ef5eb52 100644 --- a/docs/backend/wsl.md +++ b/docs/backend/wsl.md @@ -1,10 +1,9 @@ # `vla.cpp` on Windows (WSL2 + CUDA) `vla.cpp` targets Linux and macOS. On Windows the supported path is **WSL2** -with an Ubuntu distribution: the toolchain (`libzmq`, `protobuf`, `pkg-config`, -the bash `patches/patch.sh` script) and the CUDA build all run natively inside -the Linux environment, while still using the host NVIDIA GPU through the -WSL CUDA driver. +with an Ubuntu distribution: the toolchain (`libzmq`, `protobuf`, `pkg-config`) +and the CUDA build all run natively inside the Linux environment, while still +using the host NVIDIA GPU through the WSL CUDA driver. ## Prerequisites @@ -100,6 +99,72 @@ SmolVLA ships a combined GGUF plus a separate `mmproj` vision tower (see # vla-server: bound to tcp://*:5555. ready. ``` +## End-to-end evaluation (LIBERO client on WSL) + +The eval client (LIBERO simulator + preprocessing) can run in the *same* WSL +distribution as `vla-server`, so the full serving loop - sim reset/render → +client preprocess → `vla-server` inference → action → sim step → video - runs on +one Windows machine with no separate Linux host. + +Two WSL-specific gotchas: + +- **Work from the Linux filesystem, not `/mnt/c` or `/mnt/d`.** `uv`/venv installs + are slow and unreliable on the Windows-mounted drives (drvfs); `git clone` the + repo into your home directory (e.g. `~/vla.cpp`) and run from there. This also + sidesteps CRLF issues in the shell scripts. +- **Add your user to the `render` group** so MuJoCo's EGL renderer can open the + GPU render node (`/dev/dri/renderD128`, owned `root:render`). Without it, EGL + fails with `libEGL warning: failed to open /dev/dri/renderD128: Permission + denied` and falls back to software rendering: + + ```bash + sudo usermod -aG render "$USER" + # then, from PowerShell, restart WSL so the new group takes effect: + # wsl --shutdown + ``` + +Install the LIBERO client into an isolated `uv` venv. Set `UV_LINK_MODE=copy` and +a home-directory cache so `uv` does not try to hardlink across filesystems: + +```bash +export UV_LINK_MODE=copy +export UV_CACHE_DIR=$HOME/.cache/uv-vla +bash eval/sim/libero/setup_libero.sh +``` + +Then serve SmolVLA and drive one episode. The client renders with EGL and talks +to `vla-server` over ZeroMQ: + +```bash +# terminal 1 - server (GPU inference) +./build/vla-server --bind 'tcp://*:5555' "$VLA_GGUF" + +# terminal 2 - client (LIBERO sim + preprocessing) +export MUJOCO_GL=egl CUDA_VISIBLE_DEVICES=0 +eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \ + --arch smolvla --vla-addr tcp://localhost:5555 \ + --task libero_object --task-id 0 --n-episodes 1 --n-action-steps 1 \ + --output-dir outputs/wsl_smoke +``` + +The client writes `episode_000000.mp4` and `summary.txt` under +`outputs/wsl_smoke/smolvla/libero_object/task_0/`. + +### One-command runner + +`eval/run_libero_wsl.sh` wraps the two steps above - it starts `vla-server`, +waits for it to become ready, drives one client run, and stops the server on +exit. Pass the task-id and episode count as arguments, or run with none and it +prompts for them: + +```bash +bash eval/run_libero_wsl.sh # prompts for task-id + episodes +bash eval/run_libero_wsl.sh 3 5 # task-id 3, 5 episodes, no prompt +``` + +Override the served model, suite, or output dir via environment +(`VLA_GGUF`, `TASK_SUITE`, `OUTPUT_DIR`); see the header of the script. + ## Results Evo1 (libero, 1.20 GiB BF16 weights incl. InternViT vision tower), @@ -115,3 +180,18 @@ steady state: On libero_object task 0 (pick up the alphabet soup and place it in the basket), the episode succeeded after 171 steps at ~2114 ms/step client-side. + +SmolVLA (libero, 532 MB GGUF), **NVIDIA GeForce RTX 4050 Laptop GPU** (6 GB VRAM, +~1.07 GB allocated), with the full server + LIBERO client stack co-resident on a +single WSL2 (Ubuntu 24.04) machine, steady state: + +| Stage | CUDA (WSL2) | +|---------------|------------:| +| vision | ~84 ms | +| inference | ~46 ms | +| other | ~1 ms | +| **total/req** | **~131 ms** | + +On libero_object task 0 the episode succeeded end-to-end (`is_success: True`, +149 requests served) at ~155 ms/step client-side (including software EGL +rendering), producing a rendered MP4. diff --git a/eval/run_libero_wsl.sh b/eval/run_libero_wsl.sh new file mode 100644 index 0000000..60bb34b --- /dev/null +++ b/eval/run_libero_wsl.sh @@ -0,0 +1,130 @@ +#!/usr/bin/env bash +# Copyright 2026 VinRobotics +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# run_libero_wsl.sh - one-command LIBERO eval for the full serving stack on WSL2. +# +# Starts vla-server (CUDA) and drives run_sim_client_direct.py for a chosen +# task-id / episode count, both co-resident in one WSL distro, then stops the +# server. See docs/backend/wsl.md for the WSL2 + CUDA setup this assumes +# (built vla-server, installed LIBERO venv, user in the `render` group). +# +# Usage: +# bash eval/run_libero_wsl.sh # prompts for task-id + episodes +# bash eval/run_libero_wsl.sh 3 5 # task-id 3, 5 episodes (no prompt) +# TASK_SUITE=libero_spatial bash eval/run_libero_wsl.sh 0 1 +# +# Overridable via environment: +# VLA_GGUF path to the served GGUF +# (default: $HOME/data/vrfai/smolvla-libero-gguf/smolvla-libero.gguf) +# VLA_ARCH arch preset for the client (default: smolvla) +# TASK_SUITE LIBERO suite (default: libero_object) +# N_ACTION_STEPS actions replayed per predicted chunk (default: 1) +# CUDA_HOME CUDA toolkit dir (default: /usr/local/cuda-12.6) +# OUTPUT_DIR where videos/summaries land (default: $HOME/outputs/wsl_smoke) + +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" + +CUDA_HOME="${CUDA_HOME:-/usr/local/cuda-12.6}" +export PATH="$CUDA_HOME/bin:${PATH:-}" +export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}" + +VLA_ARCH="${VLA_ARCH:-smolvla}" +TASK_SUITE="${TASK_SUITE:-libero_object}" +N_ACTION_STEPS="${N_ACTION_STEPS:-1}" +VLA_GGUF="${VLA_GGUF:-$HOME/data/vrfai/smolvla-libero-gguf/smolvla-libero.gguf}" +OUTPUT_DIR="${OUTPUT_DIR:-$HOME/outputs/wsl_smoke}" + +SERVER_BIN="$REPO_ROOT/build/vla-server" +VENV_PY="$REPO_ROOT/eval/sim/libero/libero_uv/.venv/bin/python" +CLIENT="$REPO_ROOT/eval/client/run_sim_client_direct.py" +BIND_ADDR="${BIND_ADDR:-tcp://*:5555}" +CLIENT_ADDR="${CLIENT_ADDR:-tcp://localhost:5555}" + +# --- pick task-id + episode count: CLI args first, else prompt, else default --- +TASK_ID="${1:-}" +N_EPISODES="${2:-}" + +ask() { # ask ; only prompts on an interactive terminal + local __var="$1" __prompt="$2" __def="$3" __ans="" + if [ -z "${!__var}" ]; then + if [ -t 0 ]; then + read -r -p "$__prompt [$__def]: " __ans + fi + printf -v "$__var" '%s' "${__ans:-$__def}" + fi +} + +ask TASK_ID "Task id (0-9) in suite '$TASK_SUITE'" 0 +ask N_EPISODES "Number of episodes to run" 1 + +# --- validate numeric input --- +if ! [[ "$TASK_ID" =~ ^[0-9]+$ ]]; then + echo "ERROR: task-id must be a non-negative integer (got '$TASK_ID')" >&2; exit 1 +fi +if ! [[ "$N_EPISODES" =~ ^[1-9][0-9]*$ ]]; then + echo "ERROR: episodes must be a positive integer (got '$N_EPISODES')" >&2; exit 1 +fi + +# --- preflight --- +[ -x "$SERVER_BIN" ] || { echo "ERROR: vla-server not built at $SERVER_BIN (see docs/backend/wsl.md)" >&2; exit 1; } +[ -x "$VENV_PY" ] || { echo "ERROR: LIBERO venv missing; run: bash eval/sim/libero/setup_libero.sh" >&2; exit 1; } +[ -f "$VLA_GGUF" ] || { echo "ERROR: GGUF not found at $VLA_GGUF (set VLA_GGUF=...)" >&2; exit 1; } + +mkdir -p "$OUTPUT_DIR" +SERVER_LOG="$OUTPUT_DIR/vla-server.log" + +echo "[config] arch=$VLA_ARCH suite=$TASK_SUITE task-id=$TASK_ID episodes=$N_EPISODES" +echo "[config] gguf=$VLA_GGUF" +echo "[config] output=$OUTPUT_DIR" + +# --- start server, stop it on any exit --- +SERVER_PID="" +cleanup() { + if [ -n "$SERVER_PID" ] && kill -0 "$SERVER_PID" 2>/dev/null; then + echo "[cleanup] stopping vla-server (pid=$SERVER_PID)" + kill -INT "$SERVER_PID" 2>/dev/null || true + for _ in $(seq 1 10); do kill -0 "$SERVER_PID" 2>/dev/null || break; sleep 0.5; done + kill -KILL "$SERVER_PID" 2>/dev/null || true + fi +} +trap cleanup EXIT INT TERM + +echo "[server] starting (CUDA)..." +"$SERVER_BIN" --bind "$BIND_ADDR" "$VLA_GGUF" > "$SERVER_LOG" 2>&1 & +SERVER_PID=$! + +for _ in $(seq 1 180); do + if grep -q 'ready' "$SERVER_LOG" 2>/dev/null; then echo "[server] ready"; break; fi + if ! kill -0 "$SERVER_PID" 2>/dev/null; then + echo "ERROR: vla-server exited before ready; see $SERVER_LOG" >&2; tail -n 30 "$SERVER_LOG" >&2; exit 1 + fi + sleep 1 +done +grep -iE 'backend =|CUDA \(device' "$SERVER_LOG" | head -1 || true + +# --- run the sim client (EGL/GPU render path) --- +export MUJOCO_GL=egl CUDA_VISIBLE_DEVICES=0 VLA_ARCH="$VLA_ARCH" +echo "[client] running $N_EPISODES episode(s) on $TASK_SUITE/task_$TASK_ID..." +"$VENV_PY" "$CLIENT" \ + --arch "$VLA_ARCH" \ + --vla-addr "$CLIENT_ADDR" \ + --task "$TASK_SUITE" --task-id "$TASK_ID" \ + --n-episodes "$N_EPISODES" --n-action-steps "$N_ACTION_STEPS" \ + --output-dir "$OUTPUT_DIR" + +echo "[done] outputs under $OUTPUT_DIR/$VLA_ARCH/$TASK_SUITE/task_$TASK_ID/"