Merging Models

Environment Setup

For parameter merge and vendored oe_eval, the lightest repo-supported environment is:

uv venv handbook-tulu --python 3.11
source handbook-tulu/bin/activate
python -m install -r requirements-prim.txt

If you instead install pieces manually, make sure scipy is present in the same environment as transformers; newer transformers imports require it at startup.

Basic Commands

Export models

python utils/export_bilevel_source_models.py

Initial Evals

Evaluate Multitask Model

python merge.py model=tulu-8B-sft method=mtl eval=tulu3_dev_fast
python merge.py model=tulu-8B-sft method=mtl eval=tulu3_held_out

Merging

Expert weight order for tulu-8B-sft-fft-og

When passing per-expert merge weights, the order must match configs/merging/folder_model/tulu-8B-sft-fft-og.yaml:

1. coding
2. general
3. knowledge_recall
4. math_reasoning
5. multilingual
6. precise_if

Using the best individual-model scaling factor by avg from output_remote/mc_finalized/individual_models, the ordered raw weights are:

[0.5, 1.0, 0.3, 1.0, 0.9, 1.0]

For merge methods that support expert weighting, pass them with source_models_norm_weights:

python3 parameter_merge.py \
  folder_model=tulu-8B-sft-fft-og \
  method=average \
  eval=tulu3_dev_fast \
  'source_models_norm_weights=[0.5,1.0,0.3,1.0,0.9,1.0]'

This is separate from merging.scaling_factor, which remains the global merge-wide scaling factor for the chosen method.

Random Search

Random Search with Average

python3 random_search.py \
  folder_model=tulu-8B-sft-fft-og \
  eval=tulu3_dev_fast \
  merging.method=average \
  random_search.num_samples=50 \
  random_search.start_index=0 

TSV

python3 random_search.py \
  best_dev_method=tsv_tulu3_dev_best \
  random_search.num_samples=20 \
  random_search.start_index=0 

Gaussian Process

Run Gaussian Process with 4 parallel candidate evaluations:

python3 -m src.gaussian_process \
  optimization.algorithm=gaussian_process \
  optimization.gaussian_process.random_search_init_count=13 \
  optimization.num_steps=80 \
  merging.method=average \
  optimization.gaussian_process.parallel_evals=4 \
  'optimization.gaussian_process.parallel_eval_cuda_visible_devices=[0,1,2,3]' \
  optimization.gaussian_process.rbf_lengthscale=0.1 \
  optimization.gaussian_process.expected_improvement_exploration=0.02 \
  optimization.gaussian_process.run_name_prefix=average_method__random_init__rbf_lengthscale_0p1__expected_improvement_exploration_0p02

Report eval scores

python3 src/analysis/report_eval_scores.py \
  --eval-name tulu3_dev_fast outputs/parameter_merge/tulu-8B-sft-fft-og/mats_delta__scale_coef=_1.0___damping_lambda=_0.0___cg_max_iterations=_10_