WorldMemArena

WorldMemArena is a large-scale multimodal memory benchmark designed to evaluate how well AI systems retain, update, and recall information across extended multi-session interactions grounded in real-world scenarios.

Dataset Summary

Scale highlights:

• ~2.8M total words of dialogue across all samples
• Lifelong samples average 30 sessions and ~57K words each, comparable to a short novel
• Agent samples average 15~20 sessions with ~29 screenshots each
• 11 question types spanning factual recall, dynamic updates, visual reasoning, and cross-modal integration

Dataset Structure

WorldMemArena/
├── agent/
│   ├── gui/                              # 203 samples, 8 task types
│   │   ├── css/                          # CSS styling (20)
│   │   ├── excel/                        # Spreadsheet (24)
│   │   ├── file_mgmt/                    # File management (19)
│   │   ├── image_edit/                   # Image editing (30)
│   │   ├── mobile/                       # Mobile app (19)
│   │   ├── web/                          # Web browsing (27)
│   │   ├── webarena_lite/                # WebArena (39)
│   │   └── word_docs/                    # Document editing (25)
│   └── embodied/                          # 220 samples, 12 task types
│       ├── eb_alfred_base/                # ALFWorld basic (15)
│       ├── eb_alfred_common_sense/        # ALFWorld common-sense (14)
│       ├── eb_alfred_complex_instruction/ # ALFWorld complex (14)
│       ├── eb_alfred_long_horizon/        # ALFWorld long-horizon (14)
│       ├── eb_alfred_visual_appearance/   # ALFWorld visual (12)
│       ├── eb_nav_base/                   # Navigation basic (16)
│       ├── eb_nav_common_sense/           # Navigation common-sense (15)
│       ├── eb_nav_complex_instruction/    # Navigation complex (17)
│       ├── eb_nav_long_horizon/           # Navigation long-horizon (18)
│       ├── eb_nav_visual_appearance/      # Navigation visual (13)
│       ├── minecraft/                     # Minecraft (32)
│       └── omnigibson/                    # OmniGibson (40)
├── lifelong/
│   ├── project/                           # 18 samples, 6 domains
│   │   ├── academic/                      # Academic (3)
│   │   ├── education/                     # Education (3)
│   │   ├── finance/                       # Finance (3)
│   │   ├── health/                        # Health (3)
│   │   ├── software/                      # Software (3)
│   │   └── startup/                       # Startup (3)
│   └── personal/                          # 20 samples
└── small_ids.json                         # 150-sample subset ID list

Each subcategory folder contains:

• {sample_id}.json — the full sample including sessions, memory points, and QA checkpoints
• images/{sample_id}/ — all associated images for that sample (relative paths)

small_ids.json

small_ids.json is a JSON array of 150 sample IDs that form a balanced subset of the full benchmark, suitable for faster evaluation runs. It covers all domains and subcategories proportionally. To load only the small subset:

import json

with open("WorldMemArena/small_ids.json") as f:
    small_ids = set(json.load(f))

# Filter samples when iterating over the dataset
if sample["sample_id"] in small_ids:
    ...

Sample Format

Each JSON file follows a unified schema:

{
  "sample_id": "excel_01",
  "sessions": [
    {
      "_v2_session_id": "S00",
      "dialogue": [
        {
          "role": "user",
          "content": "...",
          "timestamp": "Jan 02, 2025, 09:00:00",
          "attachments": [
            {
              "caption": "description of the image",
              "type": "image_caption",
              "image_id": "excel_01_img_001",
              "file_path": "images/excel_01/excel_01_img_001.png"
            }
          ]
        },
        {
          "role": "assistant",
          "content": "...",
          "timestamp": "Jan 02, 2025, 09:02:00",
          "attachments": []
        }
      ]
    }
  ],
  "memory_points": [
    {
      "session_id": "S01",
      "memory_points": [
        {
          "index": 1,
          "memory_content": "The user switched from Plan A to Plan B due to ...",
          "memory_type": "Event Memory",
          "memory_source": "secondary",
          "is_update": "True",
          "update_type": "status_update",
          "original_memories": ["The user is following Plan A."],
          "timestamp": "2025-01-14 10:30:00",
          "importance": 1.0,
          "memory_id": "mp_S01_1"
        }
      ]
    }
  ],
  "qa_checkpoints": [
    {
      "checkpoint_id": "QA00",
      "covered_sessions": ["S00", "S01", "S02", "S03", "S04", "S05"],
      "questions": [
        {
          "question": "What caused the user to abandon the original plan?",
          "answer": "The original plan was abandoned because ...",
          "question_type": "Fact Recall",
          "question_type_abbrev": "FR",
          "difficulty": "easy",
          "evidence": [
            {"memory_id": "mp_S02_3"},
            {"image_id": "excel_01_img_005"}
          ]
        }
      ]
    }
  ]
}

Field Descriptions

Evaluation: QA Question Types

Each QA checkpoint covers a window of preceding sessions and contains questions that probe different aspects of memory. Questions are categorized into 11 types organized across three dimensions:

Textual Memory

Visual Memory

Reasoning

There are also 4,410 questions from the agent domain with the type field unset, distributed across easy (808), medium (1,830), and hard (1,772) difficulties.

Difficulty Distribution

Key Concepts

Sessions

Each sample consists of multiple chronological sessions simulating an ongoing interaction. Agent samples contain action-observation trajectories where each step includes a screenshot and the agent's action; lifelong samples contain multi-turn conversations spanning weeks to months of simulated time, with user-shared images interspersed throughout.

Memory Points

Structured ground-truth annotations of the information that should be retained after each session. Each memory point specifies:

• Content — what should be remembered
• Type — Event Memory, Semantic Memory, etc.
• Update metadata — whether it supersedes a previous memory, and if so, which one
• Timestamp and importance — when the information was established and how critical it is

Memory points serve as the gold-standard memory state for evaluating whether a system's internal memory is accurate and up-to-date.

QA Checkpoints

Checkpoints are inserted at regular intervals (typically every 5~6 sessions) throughout each sample's timeline. Each checkpoint specifies:

• Covered sessions — the range of sessions whose content the questions draw from
• Questions — a set of questions (with answers, type labels, difficulty, and evidence links)

This design enables both incremental evaluation (test after each checkpoint) and cumulative evaluation (test at the final checkpoint using the full history).

Image References

All file_path values in attachments are relative to the directory containing the JSON file. For example, in agent/gui/excel/excel_01.json, the path images/excel_01/excel_01_img_003.png resolves to:

agent/gui/excel/images/excel_01/excel_01_img_003.png

Quick Start

import json
import os
from PIL import Image

sample_path = "WorldMemArena/agent/gui/excel/excel_01.json"
with open(sample_path) as f:
    sample = json.load(f)

base_dir = os.path.dirname(sample_path)

# Walk through sessions
for session in sample["sessions"]:
    print(f"\n--- {session['_v2_session_id']} ---")
    for turn in session["dialogue"]:
        print(f"[{turn['role']}] {turn['content'][:100]}...")
        for att in turn.get("attachments", []):
            if "file_path" in att:
                img = Image.open(os.path.join(base_dir, att["file_path"]))
                print(f"  📎 {att['image_id']} ({img.size[0]}x{img.size[1]})")

# Inspect memory points
for mp_session in sample["memory_points"]:
    for mp in mp_session["memory_points"]:
        tag = "🔄" if mp.get("is_update") == "True" else "🆕"
        print(f"{tag} [{mp['memory_id']}] {mp['memory_content'][:80]}...")

# Run QA evaluation
for checkpoint in sample["qa_checkpoints"]:
    print(f"\n=== Checkpoint {checkpoint['checkpoint_id']} ===")
    print(f"Covers: {checkpoint['covered_sessions']}")
    for q in checkpoint["questions"]:
        print(f"  [{q['question_type_abbrev']}|{q['difficulty']}] {q['question'][:80]}...")
        print(f"  → {q['answer'][:80]}...")

License

This dataset is released under the CC BY-NC 4.0 license.