Skip to content

training.md

Latest commit

 

History

History
377 lines (281 loc) · 8.35 KB

training.md

File metadata and controls

377 lines (281 loc) · 8.35 KB
Raw

Training

ReasonTune-Lab uses ms-swift for the first-stage full-parameter SFT and GRPO experiments.

The mainline experiments are:

sft_full
grpo_base_full
grpo_sft_full

LoRA is intentionally deferred to later ablations.

Prerequisites

Before training, prepare:

models/Qwen2.5-0.5B-Instruct
data/sft/qwen25_math_7b_sft_train.jsonl
data/sft/qwen25_math_7b_sft_val.jsonl
data/rlvr/gsm8k_grpo_train.jsonl
data/rlvr/gsm8k_grpo_val.jsonl

The experiments use a multi-GPU H100 server. The current configs allocate:

SFT:  CUDA_VISIBLE_DEVICES=0,1,2 and NPROC_PER_NODE=3
GRPO: CUDA_VISIBLE_DEVICES=0,1,2,3 and NPROC_PER_NODE=4
MASTER_ADDR=127.0.0.1
MASTER_PORT=29501

The wrapper exports these variables and runs swift directly:

swift sft ...

ms-swift handles the distributed launch internally from NPROC_PER_NODE, MASTER_ADDR, and MASTER_PORT. Do not wrap the mainline config in an extra outer torchrun; that causes each worker to relaunch another torchrun group and can make every worker compete for the same rendezvous port.

Training outputs, logs, and metadata are local-only and ignored by git.

Configs

Training configs live under:

configs/train/sft_full.yaml
configs/train/grpo_base_full.yaml
configs/train/grpo_sft_full.yaml

They store the actual ms-swift CLI flags under training_args.

Each mainline config also enables SwanLab tracking:

tracking:
  report_to: swanlab
  project: ReasonTune-Lab
  run_name: <experiment_id>
  run_id:
  resume:

The wrapper passes --report_to swanlab and --run_name <experiment_id> to ms-swift, records tracking metadata, and sets SWANLAB_PROJECT in the training environment when tracking.project is configured.

For a new run, leave tracking.run_id, tracking.resume, and resume.resume_from_checkpoint empty. To continue an existing SwanLab chart and training state, set all of them explicitly:

tracking:
  report_to: swanlab
  project: ReasonTune-Lab
  run_name: sft_full
  run_id: <existing_swanlab_run_id>
  resume: must
resume:
  resume_from_checkpoint: outputs/sft/sft_full/v0-YYYYMMDD-HHMMSS/checkpoint-96

The wrapper exports SWANLAB_RUN_ID and SWANLAB_RESUME from these tracking fields. tracking.run_id controls SwanLab chart continuation; resume_from_checkpoint controls Trainer state continuation. When continuing by epoch count, set num_train_epochs to the total target epoch, such as 1.25, not the additional 0.25.

Dry Run

Use dry-run before launching a server job:

PYTHONPATH=src python scripts/train/run_swift.py \
  --config configs/train/sft_full.yaml \
  --dry-run

This writes run files and prints the command, but does not start training.

Formal runs redirect stdout and stderr to the configured log file, for example:

logs/sft_full.log

Full SFT

Run full-parameter SFT:

PYTHONPATH=src python scripts/train/run_swift.py \
  --config configs/train/sft_full.yaml

Default output:

outputs/sft/sft_full/
logs/sft_full.log

The first-pass SFT config uses:

dataset: data/sft/qwen25_math_7b_sft_train.jsonl
val_dataset: data/sft/qwen25_math_7b_sft_val.jsonl
model_type: qwen2
torch_dtype: bfloat16
max_length: 4096
learning_rate: 1e-5
deepspeed: null
num_train_epochs: 1

The evaluated sft_full_192 checkpoint was produced by resuming the first-pass run and setting num_train_epochs to a total target of 2. The committed config remains the one-epoch baseline template.

For ms-swift==4.2.0, Qwen2.5 text models use model_type: qwen2. qwen2_5 is not a valid text model type in this version; names such as qwen2_5_vl are for other model families. The SFT CLI still expects --torch_dtype; its torch_dtype is deprecated warning comes from lower-level Transformers code and does not mean that --dtype is accepted by swift sft.

Training-Time Validation

For SFT, the mainline config uses the fixed local validation split for lightweight training-time validation:

data/sft/qwen25_math_7b_sft_val.jsonl

This validation is the standard ms-swift / Hugging Face Trainer evaluation path, so it reports metrics such as eval_loss and token-level accuracy. It does not run generated-answer EvalScope evaluation inside the training loop.

eval_steps: 10
save_steps: 10
logging_steps: 10
load_best_model_at_end: false

Generated-answer accuracy should be measured after training, or on selected checkpoints, with the standalone EvalScope + vLLM evaluation flow documented in docs/evaluation.md.

Do not set eval_use_evalscope: true in the main SFT training config. Full generated-answer evaluation is too slow for frequent training steps and can be duplicated across distributed ranks.

If a previous distributed job is still using the default rendezvous port, PyTorch may fail with:

EADDRINUSE, message: address already in use

First check the occupied port on the server:

lsof -i :29500

or:

ss -ltnp | grep 29500

If the process is an old failed run, stop it. Otherwise launch with a different port:

PYTHONPATH=src python scripts/train/run_swift.py \
  --config configs/train/sft_full.yaml \
  --master-port 29517

For a 3-GPU run, keep the visible GPU list and process count aligned:

PYTHONPATH=src python scripts/train/run_swift.py \
  --config configs/train/sft_full.yaml \
  --cuda-visible-devices 0,1,2 \
  --nproc-per-node 3 \
  --master-port 29517

Direct Full GRPO

Run full-parameter GRPO directly from the base student model:

PYTHONPATH=src python scripts/train/run_swift.py \
  --config configs/train/grpo_base_full.yaml

Default output:

outputs/rlhf/grpo_base_full/
logs/grpo_base_full.log

The first-pass GRPO config uses:

dataset: data/rlvr/gsm8k_grpo_train.jsonl
val_dataset: data/rlvr/gsm8k_grpo_val.jsonl#20
max_completion_length: 2560
num_generations: 8
num_generations_eval: 4
reward_funcs: [accuracy]
reward_weights: [1.0]
per_device_eval_batch_size: 1
use_vllm: true
vllm_mode: colocate
learning_rate: 1e-6
deepspeed: zero2

Full GRPO from SFT

Run full-parameter GRPO from the SFT warm-start checkpoint:

PYTHONPATH=src python scripts/train/run_swift.py \
  --config configs/train/grpo_sft_full.yaml

The default model path is:

outputs/sft/sft_full/checkpoint-final

The SFT warm-start GRPO config keeps the same accuracy reward, max_completion_length: 2560, deepspeed: zero2, and evaluates / saves every 10 steps on a 20-sample validation subset for closer checkpoint inspection.

For GRPO evaluation, keep per_device_eval_batch_size * NPROC_PER_NODE divisible by num_generations_eval. The current GRPO configs use GPUs 0,1,2,3, NPROC_PER_NODE: 4, per_device_eval_batch_size: 1, and num_generations_eval: 4, giving a global eval batch size of 4. Training still uses num_generations: 8.

If the actual checkpoint name differs, override it:

PYTHONPATH=src python scripts/train/run_swift.py \
  --config configs/train/grpo_sft_full.yaml \
  --model-path outputs/sft/sft_full/checkpoint-1000

DeepSpeed

The configs recommend zero2 by default:

training_args:
  deepspeed: zero2

To disable DeepSpeed in YAML:

training_args:
  deepspeed:

or:

training_args:
  deepspeed: none

To override from the CLI:

PYTHONPATH=src python scripts/train/run_swift.py \
  --config configs/train/grpo_base_full.yaml \
  --deepspeed zero3

Disable it from the CLI:

PYTHONPATH=src python scripts/train/run_swift.py \
  --config configs/train/grpo_base_full.yaml \
  --deepspeed none

Resume

By default, configs do not resume from a checkpoint:

resume:
  resume_from_checkpoint:

Resume explicitly from the CLI:

PYTHONPATH=src python scripts/train/run_swift.py \
  --config configs/train/sft_full.yaml \
  --resume-from-checkpoint outputs/sft/sft_full/checkpoint-1000

This renders:

--resume_from_checkpoint outputs/sft/sft_full/checkpoint-1000

Run Files

Before launching training, the wrapper writes:

run_config.yaml
metadata.json
run_command.sh

to the configured metadata directory, usually the same as the output directory.

metadata.json records:

experiment_id
task
git_commit
config_path
model_path
dataset
output_dir
log_file
command
env

These files are intended to make server runs reproducible after configs change.