MobileMem

MobileMem is the dataset release for MobileMem: On-Device Memory for Continually Evolving Agents, a benchmark for evaluating long-term memory systems in realistic mobile-assistant scenarios.

MobileMem models a continually evolving personal assistant that receives heterogeneous, temporally ordered interactions from users, assistants, and mobile applications. The benchmark evaluates whether a memory system can preserve, update, retrieve, and reason over personal knowledge across long interaction horizons.

For details about the data synthesis pipeline, please refer to the MobileMem repository and the technical report.

Dataset Splits

MobileMem contains three complementary splits:

Text Split

The text split corresponds to the MobileMem textual setting described in the paper. Mobile applications are connected to a system-level assistant through predefined templates. When a user creates or updates a record, the application converts the interaction into a structured textual event and forwards it to the memory layer.

Files

text/
├── mobilemem_data.json
└── profiles/
    ├── user_01.json
    └── user_02.json

• mobilemem_data.json is a JSON array of post-processed trajectory records, with one element per synthesized user.
• profiles/*.json contains the initial persona inputs used to start trajectory synthesis. The person field in mobilemem_data.json is the final, trajectory-evolved profile and is not identical to the corresponding initial profile.

Data Format

Each trajectory record in mobilemem_data.json contains the following main fields:

Each session contains a list of messages:

{
  "id": "session_...",
  "event_id": "event_...",
  "messages": [
    {
      "id": "message_...",
      "name": "Calendar",
      "content": "The textual interaction or app event.",
      "role": "system",
      "timestamp": "2024-12-04 19:53:00",
      "metadata": {}
    }
  ],
  "started_at": "2024-12-04 19:53:00",
  "ended_at": "2024-12-04 21:53:00"
}

Application events use the system role, while ordinary conversations use user and assistant. The name field identifies the speaker or source application.

Each finalized QA pair under question_type_toolbook.question_types[*].qa_pairs follows this structure:

{
  "id": "qa_...",
  "question": "Which event happened first, and on what dates?",
  "question_type": "temporal-reasoning",
  "question_form": "open_ended",
  "golden_answers": [
    "Event A occurred first on December 4, followed by Event B on December 6."
  ],
  "difficulty": "hard",
  "num_hops": 2,
  "effective_timestamp": "2024-12-06 18:48:01",
  "source_evidences": [
    {
      "id": "message_...",
      "role": "system",
      "timestamp": "2024-12-04 19:53:00",
      "content": "..."
    }
  ]
}

For benchmark evaluation, please feed sessions to the memory system in chronological order and flatten question_type_toolbook.question_types[*].qa_pairs to obtain the evaluation questions and their golden_answers. The source_evidences field is provided for evidence analysis and reference-grounded judging.

Loading the Text Split

After cloning this dataset repository, run the following examples from the repository root: the directory containing this README and the text/ folder.

To load the raw JSON with Hugging Face Datasets:

from datasets import load_dataset

dataset = load_dataset(
    "json",
    data_files={"text": "text/mobilemem_data.json"},
)
text_split = dataset["text"]

To preserve the complete nested structure, load the file directly:

import json
from pathlib import Path

data_path = Path("text/mobilemem_data.json")
with data_path.open(encoding="utf-8") as data_file:
    trajectories = json.load(data_file)

For memory construction and benchmark evaluation, MobileMem can also be loaded through MemBase. After setting up MemBase, run the following from the root of the cloned MemBase repository and replace the dataset path as needed:

from membase import DATASET_MAPPING

mobilemem_dataset = DATASET_MAPPING["MobileMem"].read_raw_data(
    "/path/to/MobileMem/text/mobilemem_data.json"
)
print(repr(mobilemem_dataset))

The MemBase loader converts the raw records into its standardized trajectory, session, message, and QA data models.

Omni Split

The omni split extends MobileMem to multimodal mobile interactions. In addition to textual conversations and app events, each user trajectory includes mobile screenshots, camera photos, and other visual content.

Files

omni/
├── data.jsonl          # Multimodal trajectory data with image references
├── image.zip           # All images referenced in data.jsonl
└── questions.jsonl     # Evaluation questions with image grounding

• data.jsonl contains one JSON record per user, following the same session and message structure as the text split, with additional image_refs fields linking to visual content.
• image.zip contains all PNG/JPEG images (screenshots, photos, generated imagery) referenced in the trajectories.
• questions.jsonl contains the benchmark QA pairs for the omni split, including visual reasoning questions that require interpreting image content.

Data Format

Dialogue Structure contains:

Each question contains:

Loading the Omni Split

After downloading and extracting image.zip to the omni/ directory:

import json
from pathlib import Path

# Load trajectories
data_path = Path("omni/data.jsonl")
trajectories = []
with data_path.open(encoding="utf-8") as f:
    for line in f:
        trajectories.append(json.loads(line))

# Load questions
questions_path = Path("omni/questions.jsonl")
questions = []
with questions_path.open(encoding="utf-8") as f:
    for line in f:
        questions.append(json.loads(line))

For Hugging Face Datasets:

from datasets import load_dataset

dataset = load_dataset("json", data_files={
    "data": "omni/data.jsonl",
    "questions": "omni/questions.jsonl",
})

Struct Split

The struct split evaluates long-term memory in a device-native setting. It preserves information in separate app-like stores rather than flattening a user's history into a single conversation.

The complete split contains ten synthetic personas: a young professional, a young manager, a local business owner, a public-sector employee, a father, a mother, a truck driver, a delivery rider, a male university student, and a female university student.

Files

struct/
├── persona01/
├── persona02/
├── ...
└── persona10/

Every persona directory follows the same organization. Using persona01/ to illustrate one persona:

persona01/
    ├── bill/batch.json
    ├── calendar/batch.json
    ├── document/batch.json
    ├── note/batch.json
    ├── todo/batch.json
    ├── voice/batch.json
    ├── image.zip
    ├── screen/<evidence_id>.html
    ├── video/description.json
    ├── event/user.json
    └── case.csv

Data Format

Loading the Struct Split

Run the following from the dataset repository root, replacing persona01 with the desired persona directory:

import json
from pathlib import Path

persona_dir = Path("struct/persona01")

with (persona_dir / "event/user.json").open(encoding="utf-8") as f:
    persona_profile = json.load(f)

with (persona_dir / "calendar/batch.json").open(encoding="utf-8") as f:
    calendar_records = json.load(f)

Evaluation

MobileMem uses an end-to-end evaluation protocol. For the text and omni splits, a memory system processes trajectory messages in chronological order before answering the retained questions. For the struct split, it processes the app records and artifacts available by the query time, then answers the cases grounded in those sources. Predictions are compared with the corresponding reference answers.

Text-split evaluation code is maintained in the MemBase repository. Omni-split evaluation code is available in the MobileMem-Omni/eval directory. Struct-split reference cases and scoring rubrics are distributed with each persona directory.

Citation

If you use MobileMem, please cite:

@techreport{mobilemem,
      title={MobileMem: Learning from a Year of Mobile Experiences}, 
      author={Xinle Deng and Yida Xue and Xiangyuan Ru and Yijun Chen and Buqiang Xu and Mingjun Mao and Xinjie Liu and Haoming Xu and Shuofei Qiao and Mengru Wang and Chen Jiang and Yuchen Eleanor Jiang and Lizhong Wang and Jason Wang and Li Zeng and Haofen Wang and Guilin Qi and Huajun Chen and Ningyu Zhang},
      year={2026},
      institution={OPPO and OpenKG},
      eprint={2608.13606},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2608.13606}, 
}

License

This dataset is released under the MIT License.