Revisiting Promptriever: Generalization and Robustness of Instruction-Following Ability

This repository contains the code and data for:

Revisiting Promptriever: Generalization and Robustness of Instruction-Following Ability

We revisit Promptriever to investigate whether its acquired instruction-following capability generalizes beyond the training distribution and remains robust to meaning-preserving input variations.

---

RQ0: Reproducibility

We reproduce the original Promptriever experiments using the original training code and settings.

For InstructIR, we modify the evaluation implementation to obtain the EOS token ID from each model's tokenizer instead of using a model-specific fixed value.

Included in RQ0_reproducibility/

• Modified InstructIR evaluation file

All other training and evaluation procedures follow the original Promptriever implementation.

---

RQ1.1: Generalization across Constraint Dimensions

We investigate whether instruction-following ability generalizes to constraint dimensions that are rare or absent in the Promptriever training data.

We consider six constraint dimensions: Audience, Keyword, Format, Language, Length, and Source.

Included in RQ1-1_constraint_dimensions/

• data/ — Constraint-dimension annotations of sampled training instructions
• InfoSearch/ — Modified version of InfoSearch with added support for Promptriever, as the original implementation does not support it

---

RQ1.2: Generalization under Constraint Accumulation

We examine whether instruction-following ability remains effective as multiple retrieval constraints are progressively accumulated.

Included in RQ1-2_constraint_accumulation/

• data/ — Atomic-constraint annotations of sampled training instructions
• MulticonIR/ — Modified version of MultiConIR with added support for Promptriever, as the original implementation does not support it

---

RQ2.1: Lexical Robustness

We evaluate whether retrieval performance remains consistent across semantically equivalent instruction variations.

For InstructIR, we directly use the publicly available lexical-robustness dataset. For FollowIR, we provide the constructed evaluation data together with the prompt used for its generation.

Included in RQ2-1_lexical_robustness/

• data - Evaluation data for InstructIR and FollowIR

---

RQ2.2: Positional Robustness

We investigate whether instruction-following performance remains consistent when the relative position of the query and instruction changes.

We consider three input arrangements:

• Head — query before the instruction
• Middle — query within the instruction
• Tail — query after the instruction

We further introduce multi-position training, which exposes the model to different query positions during training.

Included in RQ2-2_positional_robustness/

• training/ — Multi-position training implementation
• evaluation/ — Positional-robustness evaluation code & dataset

During multi-position training, the query position is sampled from Head, Middle, and Tail with probabilities of 0.6, 0.2, and 0.2, respectively.

The model architecture and training objective remain unchanged from the original Promptriever.

---

Acknowledgements

This work builds upon the original Promptriever implementation.