Is Mamba Capable of In-Context Learning?

This is the official code repository for our paper "Is Mamba Capable of In-Context Learning?" by Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, and Frank Hutter.

Experimental Setup

Our experiments are split into two parts - simple function approximation and natural language processing (NLP) tasks.

Section 3: Simple Function Approximation

The code from the experiments is based on the code by Garg et al. (2022) Please see the detailed readme under /simple_functions/README.md for details on how to reproduce our experiments

Section 4: Investigation of Simple NLP Tasks

The code from the experiments is based on the code by Hendel et al. (2023) Please see the detailed readme under /simple_nlp_tasks/README.md for details on how to reproduce our experiments

References:

Garg, Shivam, et al. "What can transformers learn in-context? a case study of simple function classes." Advances in Neural Information Processing Systems 35 (2022): 30583-30598.

Hendel, Roee, Mor Geva, and Amir Globerson. "In-Context Learning Creates Task Vectors." Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.

Cite Us:

@article{grazzi2024mamba,
  title={Is Mamba Capable of In-Context Learning?},
  author={Grazzi, Riccardo and Siems, Julien and Schrodi, Simon and Brox, Thomas and Hutter, Frank},
  journal={arXiv preprint arXiv:2402.03170},
  year={2024}
}