Reasoning Up the Instruction Ladder
for Controllable Language Models

Our work provide a simple recipe that enables reasoning for instruction hierarchy on LLMs. Before thinking about how to solve a task, LLMs first think whether there is a conflict between system prompt and user prompt. This ability allows steering models' behavior by simply updating the system prompt.

Using safety as a downstream applicaiton, reasoning for instruction hierarchy generalize to the safety tasks out of the post-training domain, avoiding expensive data collection, annotation, and LLM-as-Judge during RL training, but still improve the safety performance significantly.

Use Finetined Checkpoint Directly

Qwen3-8B + VerIH

Training by Youself

Setup

# if you are using cluster, set cuda to 11.8.0 by
# module load cuda/11.8.0
conda create -n verih python=3.10
conda activate verih

git clone https://github.com/skai-research/VerIH
cd VerIH/RLVR
pip install -r requirements.txt
pip install flash_attn==2.8.0.post2
pip install -e .

RL Training Qwen3-8B with VerIH

sh run.sh # Train with 4xH100 for <20 hours

Dataset Generation Pipeline

# Use gpt-4o by default. Change API in dataset/generate_verih.py line 42
export OPENAI_API_KEY="Your key here"

python dataset/generate_verih.py
python dataset/make_parquet.py --template_type qwen3 

Evaluation

conda create -n verih_eval python=3.10
conda activate verih_eval

# Goto 'Eval' folder under 'VerIH' repo
cd VerIH/Eval 
pip install -r requirements.txt
pip install -e torchllms
pip install -e evals/safety-eval/utils

export MODEL_NAME="Qwen3-8B"
# Use RL training ckpt or your own model path
export MODEL_PATH="$HOME/VerIH/RLVR/Qwen3-8B-GRPO-01R-2048-verih/global_step_600/actor/huggingface" 

export TEST_MODE="CoTSysHint"
sh run_general.sh

export TEST_MODE="CoTSysHintGuardRules"
sh run_safety.sh

Acknowledge

• We run our experiments based on TinyZero and veRL.
• We use Qwen3 series base model Qwen3 and Phi-4-mini-reasoning.
• Our evaluation framework is modified based on torchllms.
• We integrate the evaluation codebase from IFBench, IFEval, IHEval, EvalScope, and safety-eval.

Citation

@inproceedings{zheng2026reasoning,
  title={Reasoning Up the Instruction Ladder for Controllable Language Models},
  author={Zheng, Zishuo and Balachandran, Vidhisha and Park, Chan Young and Brahman, Faeze and Kumar, Sachin},
  booktitle={Findings of the Association for Computational Linguistics: ACL 2026},
  pages={39332--39354},
  year={2026}
}