A lightweight parallel-question benchmark for evaluating language-steering behavior in LLMs across 32 languages.
Understanding and controlling the behavior of large language models (LLMs) is an increasingly important topic in multilingual NLP. Beyond prompting or fine-tuning, language steering, manipulating internal representations during inference, has emerged as a more efficient and interpretable technique for adapting models to a target language. Yet, no dedicated benchmarks or evaluation protocols exist to quantify the effectiveness of these techniques.
CLaS-Bench addresses this gap by providing:
This repository supports two models benchmarked in the paper:
| Model | Identifier |
|---|---|
| Llama 3.1 8B Instruct | meta-llama/Llama-3.1-8B-Instruct |
| Aya Expanse 8B | CohereLabs/aya-expanse-8b |
Note: The model choice must be specified in every shell script. The scripts can be extended to support other models with some effort.
Log in to Hugging Face before running any code:
HF_TOKEN=<your_token>
huggingface-cli login --token $HF_TOKEN
The benchmark consists of two parts: (I) Identification of language steering components and (II) Steering evaluation. Below, we detail the steps to reproduce the DiffMean results. Other methods can be run analogously using the appropriate scripts.
Most identified components are already computed and included in this repository.
cd CLaS-Bench/identification
# Step 1: Download CulturaX data for all 32 languages
./0-load_data.sh
# Step 2: Compute average activations and steering vectors
./2-diffmean.sh
# Step 3: Generate layer-wise inspection plots
python vis_diffmean.py
Evaluates two prompting strategies:
./1-baseline.sh
Scripts sweep through layers with different intervention strengths (α):
./2-diffmean-3.sh # α = 5.0
# Additional scripts: -1, -2, -4 for other α levels
./X-evaluate-language-forcing-diffmean.sh
./Y-evaluate-judge-diffmean-3.sh
# Additional scripts: -0, -1, -2 for other α levels
Produces plots for:
./B-vis-ablate-diffmean.sh
Runs the best configuration for each method based on ablation results:
./Z-get-steering-score-1.sh
To run only for DiffMean, comment out other methods in the script.
CLaS-Bench/
├── identification/ # Scripts for identifying steering components
│ ├── 0-load_data.sh
│ ├── 2-diffmean.sh
│ └── vis_diffmean.py
├── data/ # Data for all supported languages
│ └── lang.txt
├── 1-baseline.sh # Prompting baselines
├── 2-diffmean-*.sh # DiffMean steering scripts
├── X-evaluate-language-forcing-*.sh
├── Y-evaluate-judge-*.sh
├── B-vis-ablate-*.sh
├── Z-get-steering-score-*.sh
└── README.md
@inproceedings{gurgurov-etal-2026-clas,
title = "{CL}a{S}-Bench: A Cross-Lingual Alignment and Steering Benchmark",
author = "Gurgurov, Daniil and
Al Ghussin, Yusser and
Baeumel, Tanja and
Chou, Cheng-Ting and
Schramowski, Patrick and
Mosbach, Marius and
van Genabith, Josef and
Ostermann, Simon",
editor = "Liakata, Maria and
Moreira, Viviane P. and
Zhang, Jiajun and
Jurgens, David",
booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
month = jul,
year = "2026",
address = "San Diego, California, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.findings-acl.1086/",
doi = "10.18653/v1/2026.findings-acl.1086",
pages = "21591--21628",
ISBN = "979-8-89176-395-1",
abstract = "Understanding and controlling the behavior of large language models (LLMs) is an increasingly important topic in multilingual NLP. Beyond prompting or fine-tuning, $\textit{language steering}$, i.e.,{~}manipulating internal representations during inference, has emerged as a more efficient and interpretable technique for adapting models to a target language. Yet, no dedicated benchmarks or evaluation protocols exist to quantify the effectiveness of steering techniques. We introduce $CLaS-Bench$, a lightweight parallel-question benchmark for evaluating language-forcing behavior in LLMs across 32 languages, enabling systematic evaluation of multilingual steering methods. We evaluate a broad array of steering techniques, including residual-stream DiffMean interventions, probe-derived directions, language-specific neurons, PCA/LDA vectors, Sparse Autoencoders, and prompting baselines. Steering performance is measured along two axes: language control and semantic relevance, combined into a single harmonic-mean steering score. We find that across languages simple residual-based DiffMean method consistently outperforms all other methods. Moreover, a layer-wise analysis reveals that language-specific structure emerges predominantly in later layers and steering directions cluster based on language family. $CLaS-Bench$ is the first standardized benchmark for multilingual steering, enabling both rigorous scientific analysis of language representations and practical evaluation of steering as a low-cost adaptation alternative."
}
A lightweight parallel-question benchmark for evaluating language-steering behavior in LLMs across 32 languages.
Understanding and controlling the behavior of large language models (LLMs) is an increasingly important topic in multilingual NLP. Beyond prompting or fine-tuning, language steering, manipulating internal representations during inference, has emerged as a more efficient and interpretable technique for adapting models to a target language. Yet, no dedicated benchmarks or evaluation protocols exist to quantify the effectiveness of these techniques.
CLaS-Bench addresses this gap by providing:
This repository supports two models benchmarked in the paper:
| Model | Identifier |
|---|---|
| Llama 3.1 8B Instruct | meta-llama/Llama-3.1-8B-Instruct |
| Aya Expanse 8B | CohereLabs/aya-expanse-8b |
Note: The model choice must be specified in every shell script. The scripts can be extended to support other models with some effort.
Log in to Hugging Face before running any code:
HF_TOKEN=<your_token>
huggingface-cli login --token $HF_TOKEN
The benchmark consists of two parts: (I) Identification of language steering components and (II) Steering evaluation. Below, we detail the steps to reproduce the DiffMean results. Other methods can be run analogously using the appropriate scripts.
Most identified components are already computed and included in this repository.
cd CLaS-Bench/identification
# Step 1: Download CulturaX data for all 32 languages
./0-load_data.sh
# Step 2: Compute average activations and steering vectors
./2-diffmean.sh
# Step 3: Generate layer-wise inspection plots
python vis_diffmean.py
Evaluates two prompting strategies:
./1-baseline.sh
Scripts sweep through layers with different intervention strengths (α):
./2-diffmean-3.sh # α = 5.0
# Additional scripts: -1, -2, -4 for other α levels
./X-evaluate-language-forcing-diffmean.sh
./Y-evaluate-judge-diffmean-3.sh
# Additional scripts: -0, -1, -2 for other α levels
Produces plots for:
./B-vis-ablate-diffmean.sh
Runs the best configuration for each method based on ablation results:
./Z-get-steering-score-1.sh
To run only for DiffMean, comment out other methods in the script.
CLaS-Bench/
├── identification/ # Scripts for identifying steering components
│ ├── 0-load_data.sh
│ ├── 2-diffmean.sh
│ └── vis_diffmean.py
├── data/ # Data for all supported languages
│ └── lang.txt
├── 1-baseline.sh # Prompting baselines
├── 2-diffmean-*.sh # DiffMean steering scripts
├── X-evaluate-language-forcing-*.sh
├── Y-evaluate-judge-*.sh
├── B-vis-ablate-*.sh
├── Z-get-steering-score-*.sh
└── README.md
@inproceedings{gurgurov-etal-2026-clas,
title = "{CL}a{S}-Bench: A Cross-Lingual Alignment and Steering Benchmark",
author = "Gurgurov, Daniil and
Al Ghussin, Yusser and
Baeumel, Tanja and
Chou, Cheng-Ting and
Schramowski, Patrick and
Mosbach, Marius and
van Genabith, Josef and
Ostermann, Simon",
editor = "Liakata, Maria and
Moreira, Viviane P. and
Zhang, Jiajun and
Jurgens, David",
booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
month = jul,
year = "2026",
address = "San Diego, California, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.findings-acl.1086/",
doi = "10.18653/v1/2026.findings-acl.1086",
pages = "21591--21628",
ISBN = "979-8-89176-395-1",
abstract = "Understanding and controlling the behavior of large language models (LLMs) is an increasingly important topic in multilingual NLP. Beyond prompting or fine-tuning, $\textit{language steering}$, i.e.,{~}manipulating internal representations during inference, has emerged as a more efficient and interpretable technique for adapting models to a target language. Yet, no dedicated benchmarks or evaluation protocols exist to quantify the effectiveness of steering techniques. We introduce $CLaS-Bench$, a lightweight parallel-question benchmark for evaluating language-forcing behavior in LLMs across 32 languages, enabling systematic evaluation of multilingual steering methods. We evaluate a broad array of steering techniques, including residual-stream DiffMean interventions, probe-derived directions, language-specific neurons, PCA/LDA vectors, Sparse Autoencoders, and prompting baselines. Steering performance is measured along two axes: language control and semantic relevance, combined into a single harmonic-mean steering score. We find that across languages simple residual-based DiffMean method consistently outperforms all other methods. Moreover, a layer-wise analysis reveals that language-specific structure emerges predominantly in later layers and steering directions cluster based on language family. $CLaS-Bench$ is the first standardized benchmark for multilingual steering, enabling both rigorous scientific analysis of language representations and practical evaluation of steering as a low-cost adaptation alternative."
}