RUMBA

Dataset Summary

RUMBA is a long-term conversational memory benchmark for evaluating how language models and memory systems answer questions about long multi-session dialogues. The dataset contains long synthetic dialogue histories, session timestamps, and annotated question-answer pairs. It is designed to test whether a system can retrieve, integrate, and reason over evidence distributed across a long conversation history.

RUMBA is released in Russian and English versions. The Russian version is the primary version of the benchmark, while the English version enables cross-lingual comparison with models and memory systems that are stronger in English.

The Hugging Face release is stored as flat JSONL. Each JSONL row corresponds to one question-answer example and contains the full dialogue history needed to answer that question.

The benchmark contains:

• 85 dialogues with unique users;
• 4,031 dialogue sessions;
• 36,558 utterances;
• 1,543 annotated question-answer pairs;
• Russian and English versions with matching dialogue/question structure;
• questions annotated along semantic, quantitative/scope, and temporal dimensions.

The dataset is intended primarily for evaluation, not for model training. In particular, it can be used to compare full-context long-context models, retrieval-augmented generation systems, and explicit long-term memory frameworks.

Supported Tasks

RUMBA supports evaluation of long-term conversational memory, long-context question answering, retrieval-augmented memory systems, and temporal reasoning over multi-session dialogue histories.

Languages

The dataset is available in two language versions:

• Russian (ru)
• English (en)

The Russian version reflects Russian-language discourse and contains culturally grounded topics, names, social realities, and references. The English version preserves the same benchmark structure for cross-lingual evaluation.

Dataset Structure

The Hugging Face version uses a flat JSONL format. Each row is one QA example. The full conversation history is duplicated inside each row so that each example is self-contained.

A simplified example of one JSONL row is:

{
  "id": "0|q0",
  "conversation_idx": 0,
  "user_id": "user_0",
  "question": "...",
  "answer": "...",
  "category": "...",
  "subcategory": "...",
  "subsubcategory": "...",
  "text_temporal_expression": "...",
  "evidence": [1, 4, 7],
  "question_date": "2024-03-15 12:00:00",
  "conversation": [
    {
      "session_idx": 1,
      "session_datetime": "2024-01-10 18:30:00",
      "conversation": [
        {
          "role": "user",
          "message": "..."
        },
        {
          "role": "assistant",
          "message": "..."
        }
      ]
    },
    {
      "session_idx": 2,
      "session_datetime": "2024-01-15 09:10:00",
      "conversation": [
        {
          "role": "user",
          "message": "..."
        },
        {
          "role": "assistant",
          "message": "..."
        }
      ]
    }
  ]
}

Data Fields

Dialogue and session fields

These fields describe the source dialogue history and its session-level structure:

• conversation_idx: numeric identifier of the source dialogue.
• user_id: synthetic user identifier associated with the dialogue, for example user_0.
• conversation: full multi-session dialogue history for the corresponding synthetic user.

Each item in conversation represents one dialogue session:

• session_idx: integer session index.
• session_datetime: timestamp of the session.
• conversation: list of utterances in the session.

Each utterance contains:

• role: normalized speaker role, typically user or assistant.
• message: utterance text.

Question and annotation fields

These fields describe the benchmark question, its reference answer, and its annotations:

• id: unique QA example identifier in the format <conversation_idx>|q<question_idx>.

• question: benchmark question asked from the perspective of the user represented in the dialogue.

• question_date: timestamp from which the question is asked. It may be required to interpret relative dates, determine the currently valid fact, or perform temporal reasoning.

• answer: reference answer.

• category: comma-separated annotation containing three orthogonal components in the following fixed order:

  <semantic_type>, <session_scope>, <temporality>
  

  The components are:

  1. Semantic type — the operation or memory behavior required to answer the question.

     • Extraction types: StaticUser, UpdatingInfo, DeleteInfo, AsstQuery, OpenDomainType, and DateExtraction.
     • Reasoning types: SocialRelationship, Ordering, Arithmetic, Comparison, UserQA, CalendarUnderstanding, TemporalCommonsense, TemporallyModifiedGeneralReasoning, ComplexRelations, and OtherReasoning.
     • Abstention type: Abstention, used when the answer is not supported by the dialogue.

  2. Session scope — the number of evidence sessions required to answer the question.

     • single: the required evidence is contained in one session.
     • multi: evidence from multiple sessions must be combined.

  3. Temporality — whether temporal information is required to determine the answer.

     • atemporal: the question can be answered without considering when the relevant information was valid or when an event occurred.
     • temporal: answering requires temporal grounding, such as identifying a date, resolving a relative temporal expression, ordering events, or determining which fact was valid at a particular time.

  For example:

  {
    "question": "Что я делала 28 февраля?",
    "answer": "Убирались в квартире",
    "category": "DateExtraction, single, temporal",
    "subcategory": "ExplicitTEConditionQuestion",
    "subsubcategory": "date_question",
    "text_temporal_expression": "explicit_te",
    "evidence": [10]
  }
  

  In this example:

  • DateExtraction is the semantic question type;
  • single indicates that the answer is supported by one evidence session;
  • temporal indicates that answering depends on temporal information;
  • ExplicitTEConditionQuestion specifies that the temporal condition is stated explicitly in the question;
  • date_question further specifies that the question contains a calendar-date condition;
  • explicit_te indicates that the supporting evidence contains an explicit temporal expression.

• subcategory: optional category-specific subtype. Its interpretation depends on the semantic type, for example DateQA, RelativeTimeUnderstanding, EventDuration, Arithmetic, or UserQA.

• subsubcategory: optional additional level of category-specific annotation, for example date_question, date_answer, span_question, span_answer, or month/year_answer.

• text_temporal_expression: one or more tags describing temporal expressions in the supporting evidence:

  • explicit_te: the evidence explicitly states the relevant date or time;
  • implicit_te: the relevant time must be resolved from an implicit or relative temporal expression;
  • no_te: the evidence does not contain a textual temporal expression, and temporal information must instead be derived from timestamps or other context.

  Some examples may contain multiple comma-separated tags when more than one temporal-expression type is present.

• evidence: list of integer session indices containing the evidence required to answer the question.

Intended Use

RUMBA is intended for research on long-term memory in language models and AI assistants. Suitable uses include:

• evaluating long-context LLMs in a full-context setting;
• evaluating retrieval-augmented generation systems;
• evaluating explicit memory frameworks;
• comparing model performance across Russian and English;
• diagnosing failure modes by question category, temporal reasoning, and evidence scope;
• studying robustness of LLM-as-judge evaluation for memory benchmarks.

Loading the Dataset

Loading with LightEval

RUMBA can be used through a custom LightEval task configuration. In the original evaluation setup, the task is registered as follows:

memorybench_qa = LightevalTaskConfig(
    name="rumba",
    prompt_function=memorybench_prompt,
    hf_repo="ai-forever/RUMBA",
    hf_subset="ru",
    hf_avail_splits=["test"],
    evaluation_splits=["test"],
    few_shots_split=None,
    few_shots_select=None,
    metrics=[Metrics.f1_score, Metrics.rumba_metric],
    stop_sequence=["\n"],
    version=0
)

For the English version, set hf_subset="en". If the task is loaded directly from the Hugging Face Hub rather than from a local custom task, replace hf_repo="<n/a>" with the final dataset repository name.

Loading with datasets

The JSONL files can also be loaded directly with the HuggingFace datasets library:

from datasets import load_dataset

# Replace ORG_OR_USER/RUMBA with the final Hugging Face repository name.
ds_ru = load_dataset("ai-forever/RUMBA", "ru", split="test")
ds_en = load_dataset("ai-forever/RUMBA", "en", split="test")

If loading the JSONL files locally:

from datasets import load_dataset

ds_ru = load_dataset("json", data_files="rumba-ru.jsonl", split="train")
ds_en = load_dataset("json", data_files="rumba-en.jsonl", split="train")

Citation

If you use RUMBA, please cite the associated paper:

@misc{shevtsova2026rumbarussianusermemory,
      title={RUMBA: Russian User Memory Benchmark}, 
      author={Elizaveta Shevtsova and Inna Glebkina and Mark Baushenko and Pavel Gulyaev and Alena Fenogenova},
      year={2026},
      eprint={2607.21447},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.21447}, 
}