Dataset Card for CAME-Bench

CAME-Bench (Context-aware Agent Memory Evaluation Benchmark) is a synthetic dataset designed to evaluate agentic memory in continuous, goal-oriented trajectories characterized by high contextual interference.

🌐 Project Website: https://contextual-intent.github.io

📦 Project GitHub: https://github.com/Seattleyrz/contextual-intent
The main repo provides complete, clear instructions to run retriever methods on CAME-Bench.

For more information about the research paper, interactive data explorer (sample of CAME-Bench), methodology (STITCH), and additional resources, please visit our project website.

Using This Hugging Face Repo

• example_data/ is a small, decoded sample used only for Hugging Face dataset viewer visualization (not the benchmark data).
• The full benchmark lives in encoded_benchmark_codec/ and must be decoded before use (see Section 5. Decoding the Dataset).
• For the recommended workflow, use the main GitHub repo: it includes scripts to download from Hugging Face and run/evaluate your retriever methods on CAME-Bench.

1. Dataset Overview

CAME-Bench constructs dense interaction histories where recurring entities and interleaved goals create significant ambiguity. It is specifically designed to evaluate an agent's ability to manage context over long horizons.

The benchmark consists of two distinct domains, each available in three length scales:

Domains

• Travel Planning: An LLM-simulated assistant collaborates with a simulated user to construct multi-day itineraries. This involves tracking constraints, bookings, and comparison across different days.

• Debate: Two LLM agents advocate opposing positions in evidence-grounded policy argumentation. This involves tracking attacks, defenses, and concessions.

Scales

To test scalability, the dataset is stratified into three subsets:

2. Why CAME-Bench?

Deploying agents in long-horizon interactions remains challenging because similar entities and facts recur under different latent goals. Standard Retrieval-Augmented Generation (RAG) often fails here due to Contextual Interference.

CAME-Bench is designed to break standard retrieval systems by introducing:

• The Disambiguation Bottleneck: Entities (e.g., "Hotel Apollo") appear repeatedly under different contexts (e.g., "Day 1" vs. "Day 2"). Keyword search cannot distinguish these.

• Dynamic Referential Ambiguity: The history is full of implicit references (e.g., "that hotel," "the previous argument") that require structural tracking rather than static retrieval.

• No Turn-Taking Guarantees: Unlike chat benchmarks, information is often interleaved, deferred, and resolved only after distracting steps.

3. Data Schema

After preprocessing, both Travel Planning and Debate domains use a unified data schema. The dataset is stored as two files:

• Turn file: One JSON object per line, each representing a turn
• Question file: One JSON object per line, each representing an evaluation question

{
  "id": "trip-uuid-turn-1",
  "role": "user",
  "content": "I'd like to book Hotel A for Day 1.",
  "timestamp_mapping": {
    "trip-uuid": "2024-01-15T10:30:00Z",
    "day-1": "2024-01-15T10:30:00Z"
  },
  "partition": ["trip-uuid", "day-1"],
  "action": "PROPOSE",
  "action_object": "Hotel A"
}

{
  "id": "trip-uuid-turn-1-type_1",
  "type": "type_1",
  "content": "What hotel was decided for Day 1?",
  "answer": {
    "free_form_answer": "[\"Hotel A\"]"
  },
  "date": "2024-01-15",
  "question_turn_ids": ["trip-uuid-turn-1", "trip-uuid-turn-8", ...],
  "answer_turn_ids": ["trip-uuid-turn-15"],
  "answer_type": "ANSWER_TYPE_FREEFORM"
}

4. Evaluation Question Types

We pose four categories of free-response questions to target complementary aspects of memory:

• Type 1: Incremental Memory Revision: Assesses the ability to maintain and update entity states as they evolve (e.g., tracking a restaurant as it is added, then rejected later).

• Type 2: Context-Aware Factual Recall: Tests precise retrieval amidst semantically similar content (e.g., retrieving a hotel price specifically for Day 2, distinguishing it from the price for Day 1).

• Type 3: Context-Aware Multi-Hop Reasoning: Requires resolving referential expressions before retrieval (e.g., answering a question about "the dinner reservation I mentioned earlier" by first identifying the referent).

• Type 4: Information Synthesis: Assesses the ability to aggregate information distributed across multiple steps (e.g., reconstructing a full itinerary from scattered bookings).

5. Decoding the Dataset

To prevent data contamination in LLM training, CAME-Bench files are published in an encoded format. The dataset uses XOR encryption with SHA256-derived keystreams and Base64 encoding to obfuscate content while preserving the ability to decode and use the benchmark.

Prerequisites

The decoding script (codec.py) is included in the repository. No additional dependencies are required beyond Python 3.6+.

Decoding Instructions

1. Download the encoded dataset from this Hugging Face repository. The encoded files are located in the encoded_benchmark_codec/ directory, which contains:

   • metadata.json: File records with SHA256 checksums and decryption keys
   • blobs/: Directory containing Base64-encoded encrypted files

2. Decode the dataset using the provided script:

   python3 codec.py decode --encoded_dir encoded_benchmark_codec --out_dir decoded_benchmark_codec
   

   This command will:

   • Read metadata.json to retrieve file structure and decryption keys
   • Decode each encrypted blob file using its unique canary (UUID)
   • Restore the original file structure in decoded_benchmark_codec/
   • Verify SHA256 hashes and file sizes to ensure integrity

Encoding Details

The encoding process uses:

• Per-file encryption: Each file is encrypted with a unique UUID (canary)
• XOR cipher: Plaintext is XORed with a SHA256-derived keystream
• Base64 encoding: Encrypted data is Base64-encoded for safe text storage
• Integrity checks: SHA256 hashes are stored in metadata for verification

File Structure After Decoding

After decoding, you will have the original benchmark structure:

decoded_benchmark_codec/
├── benchmark_meta.json
├── traj-0/
│   ├── questions.jsonl
│   └── turns.jsonl
├── traj-1/
├── ...
└── traj-13/

Troubleshooting

• Missing metadata.json: Ensure you downloaded the complete encoded_benchmark_codec/ directory
• Decoding errors: The script includes strict integrity checks by default. If you encounter errors, verify that all blob files are present and uncorrupted
• File size mismatches: The decoder verifies file sizes match expected values. If mismatches occur, re-download the encoded dataset

For more details on the encoding/decoding process, see CODEC_USAGE.md in the repository.

6. Grading Methodology

Evaluation is performed using an LLM-as-a-Judge approach; for end-to-end evaluation scripts (including prompts/signatures), please refer to the main GitHub repo.

• For Set-based Answers: We employ answer-set macro-averaged Precision, Recall, and F1 score. The LLMAnswerEvaluatorNumberSignature counts how many ground truth items appear in the model's output.