[ICLR 2026] Revisiting Multimodal Positional Encoding in Vision–Language Models

This repository is the official implementation of Revisiting Multimodal Positional Encoding in Vision–Language Models.

Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) by examining its two core components: position design and frequency allocation. Through extensive experiments, we identify three key guidelines: positional coherence, full frequency utilization, and preservation of textual priors—ensuring unambiguous layout, rich representation, and faithful transfer from the pre-trained LLM. Based on these insights, we propose Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and plug-and-play variants that require no architectural changes. Our methods consistently outperform existing approaches across diverse benchmarks, with significant improvements in both general and fine-grained multimodal understanding.

News

• 2026.03 This repo upgraded to Hugging Face transformers 5.3.0.
• 2026.02 Qwen3.5 released: Also adopts MRoPE-Interleave w/o spatial-reset and uses partial RoPE (e.g. partial_rotary_factor).
• 2026.01 This paper was accepted by ICLR 2026. 🚀
• 2025.10 All variants of Qwen3-VL now adopt MRoPE-Interleave w/o spatial-reset!

Todo List: Implementations of Multimodal RoPE Variants

To enhance usability and consistency, we are refactoring various multimodal RoPE implementations into a unified interface, similar to Qwen3VLTextRotaryEmbedding. This effort is expected to be completed within one week (target date: November 9, 2025).

• Vanilla RoPE: Apply vanilla RoPE directly to multimodal sequences, discarding the spatio-temporal structure of visual content, yet it remains a strong baseline.
• MRoPE: Introduced in Qwen2VL, upcasts 1D positions to three axes (t, h, w) and splits the feature dimension across different positional axes.
• Our MRoPE-I: Applies an interleaved frequency allocation strategy, preserving a more complete frequency band for each positional axis; spatial reset is incorporated into the positional design to enhance visual attention.
• Our MHRoPE: Employs head-wise frequency allocation to maximize utility across attention heads; spatial reset is also used.
• VideoRoPE: Optimizes MRoPE’s frequency allocation by assigning the temporal dimension to low-frequency bands and adopting a diagonal positional design. Different from the official implementation, we vectorize the computation for faster execution.
• HoPE: Built on VideoRoPE, applies positional scaling in the design and uses NoPE (no positional encoding) on the temporal dimension during frequency allocation. Different from the official implementation, we reset the temporal positions and vectorize the computation for faster execution.
• CircleRoPE: A novel positional design that maps image tokens onto a circular trajectory orthogonal to text token indices, effectively mitigating cross-modal positional bias during generation. While the original CircleRoPE was designed for static images, we extend it to support video inputs by stacking circular rings along the temporal dimension, see here. Note that this repo doesn't support AGE mode for simplicity.
• V2PE: Treats multimodal input as a flattened 1D sequence while assigning visual tokens a configurable smaller stride, following the official rope_pos_id_stride / num_image_token rule.
• ILRoPE / OmniRoPE: Supports visual spatial reset layouts with text spatial axes reset to zero; ILRoPE uses interleaved 3D frequency allocation, while OmniRoPE follows the OmniGen2-style shifted visual layout with chunked 3D allocation.
• MMRoPE: Implements distributed 2:3:3 meta-component frequency allocation (T,T,H,W,H,W,H,W) and optional 3D position scaling.
• GRAPE: Provides commuting 3D multiplicative GRAPE designs for multimodal coordinates, plus learned-plane multimodal variants (axis, mixed, block_mixed) that apply trainable rank-2 rotations directly to Q/K.
• More variants... Feel free to open an issue or pull request, and I will add them here. 🤗

Usage

We organize various multimodal RoPE implementations under the transformers by decoupling them into two components: position design and frequency allocation.

This repo currently targets Hugging Face transformers 5.3.0 (no backward-compat guarantees).

Installation

You can install the multimodal-ropes package directly from the repository:

git clone https://github.com/JJJYmmm/Multimodal-RoPEs.git
pip install -e .
# Successfully installed multimodal-ropes-0.1.0

Quick Test (Qwen3-VL)

By default, test.py will:

• load a local Qwen3-VL-2B-Instruct checkpoint from ../checkpoints/Qwen3-VL-2B-Instruct
• patch Qwen3-VL with each RoPE variant in SUPPORT_MM_ROPES
• run a deterministic captioning prompt on a generated demo image
• print one line per variant: rope_name: caption

python test.py
python test.py --max-new-tokens 32
python test.py --verbose        # include patch details
python test.py --raw            # do not fold newlines/spaces in the decoded caption
python test.py --ckpt-dir ../checkpoints/Qwen3-VL-2B-Instruct

Qwen3.5 (Partial RoPE)

If you have a local Qwen3.5 checkpoint (e.g. ../checkpoints/Qwen3.5-0.8B), the same script will auto-detect model_type=qwen3_5 and run a small multimodal caption smoke test, including a patched mrope-interleave run that respects partial_rotary_factor and mrope_section from Qwen3.5 configs.

python test.py --ckpt-dir ../checkpoints/Qwen3.5-0.8B

Integration with Vision–Language Models (e.g., Qwen3-VL)

The package provides a simple interface to plug in different multimodal RoPE variants. Below is an example of how to patch Qwen3-VL with your preferred RoPE configuration:

Note that adapting to the new positional encodings typically requires additional training or fine-tuning to ensure optimal performance.

For a compact reference of all variant-specific knobs, see docs/rope_hyperparameters.md.

from multimodal_ropes.entry import get_multimodal_rope

def rotate_half(x):
    """Rotates half the hidden dimensions of the input."""
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]
    return torch.cat((-x2, x1), dim=-1)

def apply_multihead_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
    """
    Applies multi-head rotary positional embeddings for MHA/GQA.
    
    Args:
        q: [bs, num_heads, seq_len, head_dim]
        k: [bs, num_key_value_heads, seq_len, head_dim]
        cos, sin: [bs, num_key_value_heads, seq_len, head_dim] → broadcast to num_heads
    """
    n_repeat = q.shape[1] // cos.shape[1]
    q_embed = (q * cos.repeat_interleave(n_repeat, dim=1)) + (rotate_half(q) * sin.repeat_interleave(n_repeat, dim=1))
    k_embed = (k * cos) + (rotate_half(k) * sin)
    return q_embed, k_embed

def monkey_patch_qwen3vl(rope_name, **kwargs):
    rope_index, rope_embed = get_multimodal_rope(rope_name, **kwargs)

    logging.info(f"Begin patching Qwen3VLModel with {rope_name}.")
    Qwen3VLModel.get_rope_index = rope_index
    transformers.models.qwen3_vl.modeling_qwen3_vl.Qwen3VLTextRotaryEmbedding = rope_embed

    if rope_name == "mhrope":
        # Special handling for MHRoPE: replace the rotary embedding application function
        transformers.models.qwen3_vl.modeling_qwen3_vl.apply_rotary_pos_emb = apply_multihead_rotary_pos_emb
        logging.info("MHRoPE: Replaced apply_rotary_pos_emb with multi-head version.")

    logging.info(f"Successfully patched Qwen3VLModel with {rope_name}.")
    test_forward()  # Optional: run a minimal forward pass to verify

# Common RoPE configuration
common_kwargs = dict(
    dim=128,
    base=5_000_000,
)

# Examples of patching with different RoPE variants
monkey_patch_qwen3vl("vanilla-rope", **common_kwargs)
monkey_patch_qwen3vl("mrope-interleave", mrope_section=[24, 20, 20], temporal_stride=1, spatial_reset=True, **common_kwargs)
monkey_patch_qwen3vl("mhrope", num_key_value_heads=8, mrope_section=[2, 3, 3], temporal_stride=1, spatial_reset=True, **common_kwargs)

For more comprehensive examples and testing utilities, please refer to test.py.

CI / Regression Tests

To avoid silent behavioral drift when upgrading dependencies (e.g., transformers), this repo includes a golden regression test:

• The golden file is generated using transformers 4.57.1 and the repo pre-upgrade state.
• The CI test re-computes position_ids, rope_deltas, and rotary cos/sin for a fixed multimodal token sequence and compares against the golden.

Files:

• tests/golden/qwen3vl_rope_hf4571.pt
• tests/test_rope_golden_hf4571.py
• .github/workflows/ci.yml

Citation

If you find this repository is useful in your research or applications, please consider giving us a star 🌟 and citing it by the following BibTeX entry.

@misc{huang2025revisitingmultimodalpositionalencoding,
      title={Revisiting Multimodal Positional Encoding in Vision-Language Models}, 
      author={Jie Huang and Xuejing Liu and Sibo Song and Ruibing Hou and Hong Chang and Junyang Lin and Shuai Bai},
      journal={arXiv preprint arXiv:2510.23095},
      year={2025}
}

License

The content of this project itself is licensed under LICENSE.