Conversational Memory Benchmark

A comprehensive benchmark for evaluating conversational memory in large language models, featuring 75,336 question-answer pairs across six evidence categories. This benchmark addresses the critical challenge of memory management in conversational AI systems, where models must retain, update, and utilize information across extended multi-turn dialogues.

📚 Resources

• Paper: ConvoMem Benchmark: Why Your First 150 Conversations Don't Need RAG
• Code: github.com/SalesforceAIResearch/ConvoMem

📊 Dataset Overview

This benchmark provides a systematic evaluation framework for testing how well language models handle conversational memory across various scenarios:

• 75,336 question-answer pairs across all categories and variations
• 100 diverse personas representing different professional roles and backgrounds
• 40,000 filler conversations (400 per persona) for creating realistic conversational contexts
• 6 evidence categories testing different memory capabilities
• Pre-mixed test cases at 15 context sizes (1, 2, 3, 4, 5, 6, 10, 20, 30, 50, 70, 100, 150, 200, 300 messages)

🎯 Key Features

Evidence Categories

The benchmark tests six distinct memory capabilities:

1. User Evidence (core_benchmark/evidence_questions/user_evidence/)

   • User states facts about themselves, later asks the assistant to recall
   • Tests basic memory retention of user-provided information
   • Example: User mentions their favorite color, later asks "What's my favorite color?"

2. Assistant Facts Evidence (core_benchmark/evidence_questions/assistant_facts_evidence/)

   • Assistant provides information, user later asks to recall it
   • Tests whether models remember their own statements
   • Example: Assistant explains a concept, user later asks about that concept

3. Changing Evidence (core_benchmark/evidence_questions/changing_evidence/)

   • Information that evolves over conversation
   • Tests ability to track state changes and retain only the latest information
   • Includes 2, 3, 4, 5, and 6 evidence variations
   • Example: User's project deadline changes multiple times

4. Abstention Evidence (core_benchmark/evidence_questions/abstention_evidence/)

   • Questions where no answer exists in the conversation
   • Tests model's ability to recognize when to say "I don't know"
   • Critical for avoiding hallucination

5. Preference Evidence (core_benchmark/evidence_questions/preference_evidence/)

   • User preferences for recommendations
   • Tests nuanced understanding and application of stated preferences
   • Uses scoring rubrics for evaluation

6. Implicit Connection Evidence (core_benchmark/evidence_questions/implicit_connection_evidence/)

   • Facts requiring multi-hop reasoning across messages
   • Tests ability to connect information from different parts of conversation
   • Example: Inferring relationships between mentioned entities

Personas

100 professionally diverse personas across 6 major categories:

• Sales & Business Development: Account Executives, Business Development Managers, Sales Directors, Field Sales Representatives
• Marketing & Lead Generation: Marketing Managers, Content Marketing Specialists, Demand Generation Managers, Brand Managers
• Customer Success & Post-Sales: Customer Success Managers, Account Managers, Adoption Specialists, Renewal Specialists
• Customer Service & Technical Support: Support Engineers, Customer Service Representatives, Call Center Managers, Technical Account Managers
• Internal Operations & Executive Leadership: CEOs, CFOs, CMOs, Business Analysts, Operations Managers, Data Scientists
• Customer & Prospect Personas: Various buyer personas including CTOs, E-commerce Managers, Financial Advisors, Consultants

Each persona includes a rich narrative background that encompasses their professional journey, personal experiences, and work style. These detailed backgrounds are designed to create realistic and diverse conversational scenarios that reflect authentic professional contexts from the CRM/sales domain.

📁 Dataset Structure

.
├── README.md                              (This file)
├── dataset_info.json                      (Metadata and configuration)
│
├── core_benchmark/                        (Main benchmark data)
│   ├── personas/
│   │   └── personas_default.json         (All 100 personas)
│   │
│   ├── evidence_questions/               (75,336 question-answer pairs)
│   │   ├── user_evidence/
│   │   │   ├── 1_evidence/              (Single evidence item)
│   │   │   ├── 2_evidence/              (Two evidence items)
│   │   │   ├── 3_evidence/              (Three evidence items)
│   │   │   ├── 4_evidence/              (Four evidence items)
│   │   │   ├── 5_evidence/              (Five evidence items)
│   │   │   └── 6_evidence/              (Six evidence items)
│   │   │
│   │   ├── assistant_facts_evidence/    (Similar structure)
│   │   ├── changing_evidence/           (2-6 evidence variations)
│   │   ├── abstention_evidence/         (3 subcategories)
│   │   ├── preference_evidence/         (2 subcategories)
│   │   └── implicit_connection_evidence/ (3 subcategories)
│   │
│   ├── filler_conversations/            (40,000 irrelevant conversations - 100 files, 400 per file)
│   │
│   └── pre_mixed_testcases/            (Pre-generated ready-to-use test cases)
│       ├── user_evidence/              (6 evidence counts: 1-6)
│       │   ├── 1_evidence/             (batched_*.json files)
│       │   ├── 2_evidence/             (batched_*.json files)
│       │   ├── 3_evidence/             (batched_*.json files)
│       │   ├── 4_evidence/             (batched_*.json files)
│       │   ├── 5_evidence/             (batched_*.json files)
│       │   └── 6_evidence/             (batched_*.json files)
│       ├── assistant_facts_evidence/   (6 evidence counts: 1-6)
│       ├── changing_evidence/          (5 evidence counts: 2-6)
│       ├── abstention_evidence/        (3 evidence counts: 1-3)
│       ├── preference_evidence/        (2 evidence counts: 1-2)
│       └── implicit_connection_evidence/ (3 evidence counts: 1-3)
│
└── legacy_benchmarks/                    (Converted external benchmarks)
    ├── longmemeval/                     (LongMemEval dataset)
    └── locomo/                          (LoCoMo dataset)

💡 Pre-Mixed Test Cases

The pre_mixed_testcases directory contains pre-mixed test cases combining evidence questions with filler conversations at various context sizes. These pre-generated test cases ensure reproducible evaluation across all experiments without requiring LLM calls, eliminate mixing overhead during evaluation, and guarantee that all models are benchmarked on identical inputs. They eliminate the need for users to:

• Mix evidence questions with filler conversations
• Handle complex batching logic
• Manage memory for large-scale test generation

Test Case Structure

Each batched_*.json file contains pre-mixed test cases with:

• Evidence items embedded at various positions in conversations
• Filler conversations mixed in to create realistic context
• Context sizes: 1, 2, 3, 4, 5, 6, 10, 20, 30, 50, 70, 100, 150, 200, 300 messages
• Multiple test variations for each context size

Customization Options

While we provide comprehensive test coverage, you have full flexibility to:

• Select specific subsets: Choose only the evidence types, counts, or context sizes relevant to your evaluation
• Sample randomly: Use a fraction of test cases for quick evaluations
• Skip categories: Omit entire evidence types if not needed for your use case
• Create custom mixes: Combine the raw evidence files with filler conversations using your own strategy

The pre-mixed test cases are a convenience, not a requirement. All source components (evidence questions and filler conversations) are available for custom evaluation pipelines.

🔧 Usage

Evidence File Format

Each evidence file contains:

{
  "evidence_items": [
    {
      "question": "What color do I use for hot leads in my personal spreadsheet?",
      "answer": "Green",
      "message_evidences": [
        {
          "speaker": "User",
          "text": "I use green for hot leads in my personal spreadsheet."
        }
      ],
      "conversations": [
        {
          "messages": [
            {
              "speaker": "User",
              "text": "Hey, Assistant! I've been trying to optimize my workflow..."
            },
            {
              "speaker": "Assistant",
              "text": "Hi! Sure, I'd be happy to help..."
            }
            // Full 100-message conversation containing the evidence
          ]
        }
      ]
    }
  ]
}

Filler Conversation Format

Filler conversations use the same structure as evidence files but contain irrelevant conversations:

{
  "evidence_items": [
    {
      "question": "Initial user request or topic",
      "answer": "Description of conversation flow",
      "message_evidences": [
        {
          "speaker": "User",
          "text": "Initial message"
        }
      ],
      "conversations": [
        {
          "messages": [
            {
              "speaker": "User",
              "text": "Conversation message..."
            },
            {
              "speaker": "Assistant",
              "text": "Assistant response..."
            }
            // 100-message conversation without evaluation evidence
          ]
        }
      ]
    }
  ]
}