H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions

H2HMem evaluates multimodal LLM memory in realistic human-to-human interactions, spanning both dyadic and multi-party conversations with diverse multimodal memory challenges.

📄 Paper • 🤗 Hugging Face Dataset • 🏆 Leaderboard • 🌐 Project Page • 📦 Submission Template

⭐ If you find H2HMem useful, please consider starring the repository to help others discover the benchmark.

---

Unlike traditional human–assistant settings (left), H2HMem evaluates LLM agents as observers in human–human interactions (right), where agents must process multimodal conversations, resolve complex discourse phenomena (anaphora, deixis), and handle asynchronous or conflicting information from multiple participants.

---

✨ What is H2HMem?

Most memory benchmarks ask a simple question:

Can the agent remember information from past interactions?

H2HMem asks a more deployment-relevant one:

Can the agent observe, retain, and utilize multimodal information from complex human–human conversations, spanning multiple participants, sessions, and modalities?

This matters in emerging real-world applications where LLM agents act as observers in human–human interactions, such as:

• 🩺 Clinical documentation: Generating patient-centered notes from clinician–patient dialogues.
• 🏥 Medical board meetings: Processing multimodal inputs from AI-powered medical meeting assistants.
• 📋 Meeting summarization: Summarizing general multi-party meetings with cross-modal references.
• 🗣️ Multi-party discourse: Tracking information distributed across multiple participants over extended interactions.

These environments introduce three fundamental challenges:

H2HMem evaluates three core memory capabilities across nine task types:

---

📌 Benchmark at a Glance

---

🧠 Why H2HMem?

Existing memory benchmarks largely focus on single-user, text-only interactions, where an agent recalls information from its own past interactions with a single user. However, a new class of applications is emerging: LLM agents as observers in human–human interactions.

Unlike traditional human–assistant settings, these environments:

• Are inherently multimodal, interleaving text with visual content such as shared photographs and screenshots.
• Involve complex discourse phenomena such as anaphora and deixis, requiring agents to resolve references against evolving conversational memory.
• Contain asynchronous or conflicting information from multiple participants.
• Require tracking information across multiple participants, sessions, and modalities.

This makes H2HMem fundamentally different from prior benchmarks that focus on single-agent recall, personalization, or long-context understanding in isolation. Experiments with advanced agents reveal substantial limitations in constructing, retaining, and utilizing memories across modalities, participants, and sessions, highlighting substantial room for improvement in next-generation LLM agents.

---

🏗️ Benchmark Construction Pipeline

H2HMem is constructed from dyadic and multi-party human-human conversations with multimodal inputs, structured into multiple sessions with diverse QA tasks spanning recall, reasoning, and application.

---

📊 Main Results

Experiments reveal substantial limitations in current memory systems. The best weighted average LLM-as-Judge score reaches only 0.5757 (A-Mem). Four major bottlenecks are identified:

• Cross-modal alignment remains challenging, with consistent gaps between unimodal and cross-modal retrieval.
• Weak distractor filtering despite successful retrieval, showing large recall–precision gaps across methods.
• Limited causal reasoning and adaptation to human referential conventions, particularly in Multimodal Causal Reasoning (MCR) and Reference & Evolution Tracking (RET).
• Poor robustness to conflicting information, with near-zero lexical precision and recall in Conflict Detection (CD).

LLM-as-a-judge evaluation results across task types and methods.

---

📥 Dataset Download

Download the dataset from Hugging Face:

🤗 Hugging Face Dataset

Save the contents to the following local folders:

• ./dataset/dyadic
• ./dataset/multi-party

---

⚙️ Installation

git clone https://github.com/varib1/H2HMEM.git
cd H2HMEM
pip install -r requirements.txt

🧪 Inference

This repository supports both text-based and multimodal memory methods.

📝 Text-based Methods

Text-based methods first convert images into textual captions, then perform evaluation using the corresponding baseline.

Step 1 — Generate Image Captions

bash ./caption/run_caption.sh

Step 2 — Run Evaluation

🖼️ Multimodal Methods

Multimodal methods directly process image-text inputs and do not require caption generation.

---

📈 Metrics Evaluation

After inference, evaluate the generated responses using the following metrics.

🔤 Lexical Metrics

bash ./evaluate_metrics/Lexical_metrics/run_Lexical_metrics.sh

🤖 LLM-as-a-Judge

bash ./evaluate_metrics/LLM-as-judge/run_LLM_as_judge.sh

📌 Notes

• Text-based methods require an additional caption generation stage.
• Multimodal methods directly consume image-text inputs.
• All evaluation scripts automatically save outputs to their corresponding result directories.

---

🏆 Leaderboard Submission

We provide an online leaderboard for evaluating memory methods on H2HMem:

🏆 H2HMem Leaderboard

Submission Workflow

1. Run inference on the dyadic and multi-party datasets.
2. Format your predictions according to the submission template.
3. Upload the files to the leaderboard website.

📦 Submission Format

We provide a JSON template file that defines the required submission format:

• leaderboard/prediction_template.txt — Template for submission format

Your final submission should include two JSON files:

prediction_dyadic.json
prediction_multiparty.json

📤 Submit to Leaderboard

1. Open the 🏆 H2HMem Leaderboard.
2. Upload prediction_dyadic.json and prediction_multiparty.json.
3. Fill in method metadata: method name, organization / affiliation (optional), and additional description (optional).
4. Submit your results.

Your submission needs to be reviewed by an administrator before it appears on the public leaderboard.

📌 Submission Notes

• Please ensure the prediction files strictly follow the provided JSON format.
• Both dyadic and multi-party prediction files are required for complete evaluation.
• We recommend preserving original model outputs before post-processing for reproducibility.

---

📚 Citation

If you use H2HMem, please cite the accompanying paper.

@misc{zhu2026h2hmemmultimodalmemorybenchmark,
      title={H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions}, 
      author={Shiping Zhu and Yibo Yang and Zhengyang Wang and Tiancheng Shen and Dandan Guo and Ming-Hsuan Yang},
      year={2026},
      eprint={2606.09461},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.09461}, 
}