Behavior Cues

Code for the paper Behavior Cue Reasoning: Monitorable Reasoning Improves Efficiency and Safety through Oversight.

This repository contains the paper code, prompt templates, scenario definitions, vendored experiment code, and reproduction notes for the Step 1 actor-SFT pipeline, Step 2 monitor-RL experiments, and Step 3 HazardWorld safety-monitoring experiments.

A Behavior Cue is an [answer]…[continue] (or [stop]) checkpoint emitted inside a model's <think>…</think> block:

<think>
[answer]
\boxed{24}
[continue]
…initial reasoning…
[answer]
\boxed{42}
[continue]
…more reasoning…
[answer]
\boxed{37}
[stop]
</think>
\boxed{37}

Step 1 teaches a base reasoning model to produce these checkpoints via SFT on data assembled by eliciting forced-stop answers from a non-checkpointing baseline at each reasoning chunk boundary.

Included artifacts

This release tracks source code, prompt templates, scenario definitions, configuration files, vendored code needed for the paper workflows, and documentation. It does not include generated trajectories, continuation labels, SFT JSONL rows, monitor datasets, replay parquet files, checkpoints, logs, wandb outputs, or internal smoke tests. Those generated artifacts are ignored by git; regenerate them with the documented commands or point scripts at your own local paths / Hub model IDs.

Quickstart

The repository uses separate virtual environments because the inference stack and SFT stack require incompatible torch / vllm / transformers package sets. The names below are intentionally descriptive so it is clear which stack is active.

# From the repository root.
python -m pip install --upgrade uv

# Inference env: trajectory generation, continuations, forced-token tests,
# fixation tests, and HazardWorld monitored decoding.
uv venv .venv-inference
source .venv-inference/bin/activate
uv pip install -r requirements.txt
python -B scripts/audit.py

# SFT env: actor SFT for AIME/TW/HazardWorld and HazardWorld monitor SFT.
deactivate
uv venv sft/.venv-sft
source sft/.venv-sft/bin/activate
uv pip install -r sft/requirements.txt

Monitor RL has a third, separate environment because its ray + flash-attn + vllm stack is heavier and conflicts with the Step 1 environments:

cd monitor_rl
bash setup_uv_env.sh
source .venv-monitor-rl/bin/activate

Environments

Two named top-level environments cover the Step 1 / Step 3 domain code:

Path	Purpose	Key pins
.venv-inference/	Inference: trajectory + continuation generation for AIME / TW / HW	vllm 0.15.1 / torch 2.9.1+cu128 / transformers 4.57.6 / gymnasium / tale-suite (git) / textworld / alfworld
sft/.venv-sft/	All Step-1 actor SFT and Step-3 monitor SFT	torch 2.6.0 / transformers 4.57.6 / trl 0.25.1 / accelerate 1.13.0

The torch versions are pinned at incompatible majors (requirements.txt installs vllm==0.15.1, whose stack uses torch==2.9.1; sft/requirements.txt pins torch==2.6.0), which is why the inference and SFT stacks live in separate environments.

The repo itself is not packaged; python <domain>/<script>.py from the repo root works, and common.* resolves via the repo-root sys.path.insert each script does at startup. Generated trajectories and continuation labels are written to data/ paths, while SFT-ready JSONL rows live under each domain's sft_data/ directory. The exact artifact paths are listed in docs/REPRODUCIBILITY.md; several scripts also expose explicit --input, --output_dir, or --save_dir flags.

Step 2 (monitor RL) has its own separate .venv-monitor-rl/ inside monitor_rl/; see that subdirectory's README.

Intended use and safety

This is research code for studying monitorable reasoning, oversight, and efficiency/safety tradeoffs in controlled benchmark settings. The monitors, rules, and trained models here are experimental artifacts, not deployable safety systems. Generated model behavior should not be treated as a safety guarantee, and any use outside the documented research workflows should be validated independently.

Documentation

The README is intentionally short. Detailed reproduction notes live in:

• docs/REPRODUCIBILITY.md — artifact paths, model registry, prompt variants, Step 1 actor-SFT commands, post-SFT regeneration, SFT resume behavior, auxiliary experiments, and Monitor-RL pointers.
• docs/HAZARDWORLD_MONITORING.md — Step 3 rule-based monitored decoding, replay annotations, learned Safe/Unsafe monitor SFT, and optional analyses.
• monitor_rl/README.md — vendored VERL-agent replay environment and AIME/TW monitor-RL launch details.
• hazard/MODIFICATIONS.md — HazardWorld vendored-code provenance and environment-level modifications.

Step 1 pipeline

The same four-stage flow runs in each domain, with one extra consolidation step for HazardWorld:

1. Baseline trajectories  ──► 2. Continuation labels  ──► 3. SFT JSONL  ──► 4. Actor SFT
   (no [answer] tokens)        (forced-stop answer per       (Behavior Cue
                                chunk boundary)               training data)

Output filenames include both the model save name and the prompt style, so basic and behavior_cue runs never collide on disk.

The detailed commands use the canonical Qwen3-8B path used by most examples in this repository. The same scripts also accept --model glm-z1 or --family glm-z1/--family glm where applicable; when using GLM, keep the model family and dataset names consistent across trajectory generation, continuation labeling, conversion, and SFT.

Concrete commands and artifact paths are in docs/REPRODUCIBILITY.md.

Step 2: Monitor RL

The monitor-RL experiment code is included for reproducibility. It trains a reasoning model to make [stop] / [continue] decisions over the SFT'd actor's saved trajectories (from Step 1.5), and lives in its own top-level subdirectory with its own .venv-monitor-rl/ environment (heavy vllm + flash-attn + ray stack):

monitor_rl/                    vendored VERL-agent fork — see monitor_rl/README.md
├── verl/                       upstream VERL trainer (kept verbatim)
├── agent_system/               paper-specific extensions
│   └── environments/env_package/replay/   the replay environment
├── examples/data_preprocess/   actor-trajectories → replay format converters
└── rl_scripts/                 curated PPO / GRPO launch scripts

Pipeline at a glance:

1. Convert Step 1.5 trajectories to replay format — per-domain converters in monitor_rl/examples/data_preprocess/.
2. Bake into parquet — examples/data_preprocess/prepare.py --mode text writes train.parquet / test.parquet under $HOME/data/verl-agent/text/.
3. Launch RL training — curated launch scripts live under monitor_rl/rl_scripts/.

Concrete reproduction commands live in monitor_rl/README.md. verl/ and agent_system/ import each other bidirectionally and must stay together — they're a single inseparable fork, which is why this subdirectory is vendored as a unit rather than split across the domain-grouped layout.

Step 3: Safety monitoring + rule-based enforcement

The HazardWorld safety-monitoring experiment code is included for reproducibility. Two monitoring paths live in hazard/: a pure-stdlib rule-based monitor used during decoding, and a learned binary Safe/Unsafe monitor trained from replay-annotated trajectories. See docs/HAZARDWORLD_MONITORING.md for commands, file roles, and optional analyses.

Layout

behavior-cues/
├── common/                shared infrastructure
│   ├── model_registry.py    canonical MODELS + VLLM_DEFAULTS
│   ├── prompts.py           PROMPT_STYLES + resolve_prompt_style
│   ├── gpu_setup.py         --gpu / VLLM_USE_V1 env setup (before vllm import)
│   ├── vllm_agent.py        verl-agent-format inference agent (TW + HW)
│   ├── continuations.py     ContinuationAgent + chunked-continuation loop
│   ├── trl_sft.py           PushToHub callback, load_data, SFTConfig builder
│   ├── forced_token.py      forced [stop]/[continue] adherence experiment
│   └── fixation.py          embedded-answer fixation experiment
├── aime/                  AIME math reasoning
│   ├── prompts.py           SYSTEM_PROMPT_{BASIC,BEHAVIOR_CUE}, PROMPT_TEMPLATES
│   ├── generate_trajectories.py
│   ├── generate_continuations.py
│   ├── trl_sft.py
│   ├── sft_data/             generated AIME SFT JSONL lives here
│   ├── forced_token_test.py
│   └── embedded_answer_fixation.py
├── tw/                    CookingWorld / TextWorld
│   ├── prompts.py           VERL_AGENT_PROMPT_{BASIC,BEHAVIOR_CUE}
│   ├── zero_shot_loop.py    baseline trajectory driver (verl-agent)
│   ├── generate_continuations.py
│   ├── trl_sft.py
│   ├── sft_data/             generated TW SFT JSONL lives here
│   └── forced_token_test.py
├── docs/                  reproduction and validation details
│   ├── REPRODUCIBILITY.md
│   └── HAZARDWORLD_MONITORING.md
├── hazard/                HazardWorld
│   ├── envs/gym_minigrid/   vendored modified gym_minigrid
│   ├── prompts.py           make_system_prompt + make_verl_agent_prompt
│   ├── text_llm.py          baseline trajectory driver (verl-agent)
│   ├── text_llm_monitored.py  monitored decoding driver (Step 3)
│   ├── rule_monitor.py      rule-based action safety classifier (Step 3)
│   ├── replay_trajectories.py  replay to log engine-truth safe/unsafe actions
│   ├── analyze_violations.py   post-hoc CoT-vs-violation analysis
│   ├── monitor_sft.py       binary Safe/Unsafe classifier SFT (Step 3)
│   ├── consolidate.py       merge per-scenario trajectory JSONL files
│   ├── generate_continuations.py
│   ├── trl_sft.py
│   ├── sft_data/             generated HazardWorld SFT JSONL lives here
│   ├── forced_token_test.py
│   ├── scenarios.json       fixed train/test scenario definitions
│   ├── MODIFICATIONS.md     HazardWorld provenance + environment changes
│   ├── LICENSE              upstream HazardWorld / SRL-NLC MIT license
│   └── NOTES.md             HazardWorld-specific notes
├── scripts/               cross-cutting data utilities
│   ├── continuations_to_sft.py   continuation labels -> SFT JSONL
│   ├── rewrite_system_prompt.py  swap an SFT JSONL's system prompt variant
│   └── audit.py                  parseability + first-party import check
├── NOTICE.md              third-party provenance and vendored-code notes
└── monitor_rl/            Step 2: vendored VERL-agent fork (see monitor_rl/README.md)

Citation

If you use this code, please cite the paper:

@article{cui2026behaviorcue,
  title={Behavior Cue Reasoning: Monitorable Reasoning Improves Efficiency and Safety through Oversight},
  author={Cui, Christopher Z. and Killian, Taylor W. and Ammanabrolu, Prithviraj},
  journal={arXiv preprint arXiv:2605.07021},
  year={2026},
  url={https://arxiv.org/abs/2605.07021}
}

This repository also builds on VERL-agent and HazardWorld. Please cite them where relevant:

@article{feng2025groupingroup,
  title={Group-in-Group Policy Optimization for LLM Agent Training},
  author={Feng, Lang and Xue, Zhenghai and Liu, Tingcong and An, Bo},
  journal={arXiv preprint arXiv:2505.10978},
  year={2025},
  url={https://arxiv.org/abs/2505.10978}
}

@article{yang2020safe,
  title={Safe Reinforcement Learning with Natural Language Constraints},
  author={Yang, Tsung-Yen and Hu, Michael and Chow, Yinlam and Ramadge, Peter J. and Narasimhan, Karthik},
  journal={arXiv preprint arXiv:2010.05150},
  year={2020},
  url={https://arxiv.org/abs/2010.05150}
}