Skip to content

model_serving.md

Latest commit

 

History

History
172 lines (122 loc) · 4.14 KB

model_serving.md

File metadata and controls

172 lines (122 loc) · 4.14 KB
Raw

Model Download and vLLM Serving

This document describes how to prepare models and serve them through a local OpenAI-compatible vLLM endpoint.

No model weights are committed to git. Downloads should go under the ignored models/ directory on the machine that runs the experiment.

Model Configs

The first-stage base model configs are:

configs/models/qwen25_math_7b_instruct.yaml
configs/models/qwen25_05b_instruct.yaml

They define:

  • Hugging Face repo id
  • ModelScope repo id fallback
  • local model directory
  • served model name
  • vLLM host, port, dtype, tensor parallel size, memory utilization, and max model length

The default port is 8000 for all configs. The first version assumes one model is served at a time. If concurrent services are needed, override --port manually.

The default vLLM max model length is 4096, matching the Qwen2.5-Math context limit. EvalScope uses max_tokens: 2560, so prompts must stay short enough to leave completion room.

Download Models

Teacher:

PYTHONPATH=src python scripts/model/download_model.py \
  --config configs/models/qwen25_math_7b_instruct.yaml

Student:

PYTHONPATH=src python scripts/model/download_model.py \
  --config configs/models/qwen25_05b_instruct.yaml

The default source is auto, which tries Hugging Face first and falls back to ModelScope if the Hugging Face command fails.

Use Hugging Face explicitly:

PYTHONPATH=src python scripts/model/download_model.py \
  --config configs/models/qwen25_math_7b_instruct.yaml \
  --source hf

Use ModelScope explicitly:

PYTHONPATH=src python scripts/model/download_model.py \
  --config configs/models/qwen25_math_7b_instruct.yaml \
  --source modelscope

Start vLLM

Serve the teacher model:

PYTHONPATH=src python scripts/model/serve_vllm.py \
  --config configs/models/qwen25_math_7b_instruct.yaml

Serve the base student model:

PYTHONPATH=src python scripts/model/serve_vllm.py \
  --config configs/models/qwen25_05b_instruct.yaml

The script starts:

vllm serve <model_path> ... --generation-config vllm

Using --generation-config vllm keeps repository generation defaults from silently changing experiment behavior.

To inspect the command without starting vLLM:

PYTHONPATH=src python scripts/model/serve_vllm.py \
  --config configs/models/qwen25_05b_instruct.yaml \
  --print-command

Health Check

After starting vLLM, check the OpenAI-compatible endpoint:

PYTHONPATH=src python scripts/model/check_openai_server.py \
  --base-url http://127.0.0.1:8000/v1 \
  --model Qwen2.5-0.5B-Instruct

The check calls /v1/models, chooses the first returned model unless --model is provided, and sends one minimal chat completion request.

Teacher Distillation

Teacher distillation uses the same OpenAI-compatible endpoint configured in:

configs/data/teacher_distill.yaml

Default teacher API settings:

api_base: http://127.0.0.1:8000/v1
api_key: EMPTY
api_type: chat_completions

Typical server-side sequence:

PYTHONPATH=src python scripts/model/download_model.py \
  --config configs/models/qwen25_math_7b_instruct.yaml

PYTHONPATH=src python scripts/model/serve_vllm.py \
  --config configs/models/qwen25_math_7b_instruct.yaml

PYTHONPATH=src python scripts/model/check_openai_server.py \
  --base-url http://127.0.0.1:8000/v1 \
  --model Qwen2.5-Math-7B-Instruct

PYTHONPATH=src python scripts/data/distill_with_teacher.py \
  --config configs/data/teacher_distill.yaml

Run the health check and distillation commands in another shell after vLLM is started.

Serving Local Checkpoints

Later SFT or GRPO checkpoints do not need dedicated model config files. Serve them directly:

PYTHONPATH=src python scripts/model/serve_vllm.py \
  --model-path outputs/sft/sft_full/checkpoint-xxx \
  --served-model-name sft_full

Override vLLM settings when needed:

PYTHONPATH=src python scripts/model/serve_vllm.py \
  --model-path outputs/sft/sft_full/checkpoint-xxx \
  --served-model-name sft_full \
  --port 8001 \
  --max-model-len 32000

Only use extra ports when running multiple model services concurrently.