ECCV 2026 (Oral, Spotlight) · Paper (arXiv) · Project Page · Model checkpoints · Training subset
TL;DR — Attention-sink tokens in LVLMs can help or hurt downstream accuracy depending on the layer and the task; a tiny per-layer learned gate (Layer-wise Sink Gating, LSG) trained with next-token prediction alone exploits this to improve a frozen backbone.
The main results (LLaVA-1.5-7B) are reproduced by four stages, each a directory:
sink_id/ — identify V-sinks and L-sinks: per-dimension massive-activation
statistics → sink dimension + threshold τ per architecture. Input to every
later stage; precomputed results ship in modeling/sink_dicts/ and
configs/*.yaml.probing/ — do sink tokens encode global information? Layer-wise linear
probing on CLEVR scene attributes.sweep/ — oracle key-gating sweep: scale V-sink keys by a coefficient at
one layer, over a layer x coefficient x subtask grid (MMStar + CVBench).lsg/ — train and evaluate LSG gates (backbone frozen, NTP loss only):
single-layer and multi-layer stacking.Each directory has its own README with exact commands.
pip install -r requirements.txt
export LSG_PUBLIC_ROOT=$(pwd)
Then install the backbone package(s) and copy our modified files over them —
see modeling/README.md (one virtualenv per backbone
is recommended). Edit the paths in configs/*.yaml for your environment.
| dataset | used by | how to get it |
|---|---|---|
| MMStar, CVBench | sweep, lsg eval | downloaded automatically by lmms-eval |
| CLEVR v1.0 | probing | download, set clevr_images in configs/llava15.yaml; question CSVs are generated by probing/data_generation/ |
| Cambrian 10k subset | lsg training | conversations: JH-C-k/lsg_public (HF dataset); images from Cambrian-10M |
| Result | Command |
|---|---|
| Table 1 (single-layer LSG, L10) | bash lsg/run_train.sh → python lsg/select_best_checkpoint.py → bash lsg/run_eval.sh with lsg/configs/single_layer_L10.json |
| Table 2 (5-layer stack {3,6,7,10,19}) | python lsg/stack_gates.py --layers 3,6,7,10,19 ... → bash lsg/run_eval.sh with lsg/configs/stack_L3_L6_L7_L10_L19.json |
| Figure 4 (CLEVR probing bands) | python figures/make_fig4_probing_band.py |
| Figure 5 (layer-wise optimal gate coefficients, 4 panels) | python figures/make_fig5_gate_coefficients.py |
| Figure S4 (oracle vs. learned gate heatmaps) | python figures/make_figS4_oracle_heatmap.py |
| Figure 7 (learned gate heatmap) | python figures/make_fig7_learned_heatmap.py |
The figure scripts read the precomputed CSVs in data/ and need no GPU.
To regenerate those CSVs from your own runs: probing/export_probing_csv.py,
sweep/aggregate_results.py (32-layer sweep) and sweep/aggregate_stage2_ranges.py
(stage-2 range sweep behind Fig. 5d).
notebooks/sweep_analysis.ipynb — rendered
directly on GitHub: probing curves, the oracle per-layer sweep, where the same
intervention flips sign across subtasks, the broad-optimum property, and the
learned-gate comparison. All computed from data/.
data/sweep_results/ and data/probing_results/ — the aggregated source data
behind the paper's figures (raw per-sample logs are not shipped). Schemas are
documented in data/README.md.
experiments/llava_ov/ — Appendix E: sink taxonomy
replication and oracle key-gating sweep on LLaVA-OneVision-7B.experiments/cross_arch/ — Appendix F: oracle
sweep and LSG on two additional backbones (Phi-3+CLIP, Qwen2.5+SigLIP).@article{choi2026sinks,
title={When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models},
author={Choi, Jiho and Kim, Jaemin and Kim, Sanghwan and Hong, Seunghoon and Park, Jin-Hwi},
journal={arXiv preprint arXiv:2604.03316},
year={2026}
}
Apache-2.0. Files under modeling/ are derived from
LLaVA,
LLaVA-NeXT and
lmms-eval (all Apache-2.0);
original copyright notices are retained.
ECCV 2026 (Oral, Spotlight) · Paper (arXiv) · Project Page · Model checkpoints · Training subset
TL;DR — Attention-sink tokens in LVLMs can help or hurt downstream accuracy depending on the layer and the task; a tiny per-layer learned gate (Layer-wise Sink Gating, LSG) trained with next-token prediction alone exploits this to improve a frozen backbone.
The main results (LLaVA-1.5-7B) are reproduced by four stages, each a directory:
sink_id/ — identify V-sinks and L-sinks: per-dimension massive-activation
statistics → sink dimension + threshold τ per architecture. Input to every
later stage; precomputed results ship in modeling/sink_dicts/ and
configs/*.yaml.probing/ — do sink tokens encode global information? Layer-wise linear
probing on CLEVR scene attributes.sweep/ — oracle key-gating sweep: scale V-sink keys by a coefficient at
one layer, over a layer x coefficient x subtask grid (MMStar + CVBench).lsg/ — train and evaluate LSG gates (backbone frozen, NTP loss only):
single-layer and multi-layer stacking.Each directory has its own README with exact commands.
pip install -r requirements.txt
export LSG_PUBLIC_ROOT=$(pwd)
Then install the backbone package(s) and copy our modified files over them —
see modeling/README.md (one virtualenv per backbone
is recommended). Edit the paths in configs/*.yaml for your environment.
| dataset | used by | how to get it |
|---|---|---|
| MMStar, CVBench | sweep, lsg eval | downloaded automatically by lmms-eval |
| CLEVR v1.0 | probing | download, set clevr_images in configs/llava15.yaml; question CSVs are generated by probing/data_generation/ |
| Cambrian 10k subset | lsg training | conversations: JH-C-k/lsg_public (HF dataset); images from Cambrian-10M |
| Result | Command |
|---|---|
| Table 1 (single-layer LSG, L10) | bash lsg/run_train.sh → python lsg/select_best_checkpoint.py → bash lsg/run_eval.sh with lsg/configs/single_layer_L10.json |
| Table 2 (5-layer stack {3,6,7,10,19}) | python lsg/stack_gates.py --layers 3,6,7,10,19 ... → bash lsg/run_eval.sh with lsg/configs/stack_L3_L6_L7_L10_L19.json |
| Figure 4 (CLEVR probing bands) | python figures/make_fig4_probing_band.py |
| Figure 5 (layer-wise optimal gate coefficients, 4 panels) | python figures/make_fig5_gate_coefficients.py |
| Figure S4 (oracle vs. learned gate heatmaps) | python figures/make_figS4_oracle_heatmap.py |
| Figure 7 (learned gate heatmap) | python figures/make_fig7_learned_heatmap.py |
The figure scripts read the precomputed CSVs in data/ and need no GPU.
To regenerate those CSVs from your own runs: probing/export_probing_csv.py,
sweep/aggregate_results.py (32-layer sweep) and sweep/aggregate_stage2_ranges.py
(stage-2 range sweep behind Fig. 5d).
notebooks/sweep_analysis.ipynb — rendered
directly on GitHub: probing curves, the oracle per-layer sweep, where the same
intervention flips sign across subtasks, the broad-optimum property, and the
learned-gate comparison. All computed from data/.
data/sweep_results/ and data/probing_results/ — the aggregated source data
behind the paper's figures (raw per-sample logs are not shipped). Schemas are
documented in data/README.md.
experiments/llava_ov/ — Appendix E: sink taxonomy
replication and oracle key-gating sweep on LLaVA-OneVision-7B.experiments/cross_arch/ — Appendix F: oracle
sweep and LSG on two additional backbones (Phi-3+CLIP, Qwen2.5+SigLIP).@article{choi2026sinks,
title={When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models},
author={Choi, Jiho and Kim, Jaemin and Kim, Sanghwan and Hong, Seunghoon and Park, Jin-Hwi},
journal={arXiv preprint arXiv:2604.03316},
year={2026}
}
Apache-2.0. Files under modeling/ are derived from
LLaVA,
LLaVA-NeXT and
lmms-eval (all Apache-2.0);
original copyright notices are retained.