🧩 PRAGMA

PRAGMA is a benchmark for evaluating personalized guidance with memory alignment over lifelong conversation histories. It tests whether a model can use relevant past interactions while remaining aligned with a user's evolving experiences and trajectory.

PRAGMA accompanies the paper “PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations” (EMNLP 2026).

✨ Dataset Overview

📂 Files

🧭 Query Types

🚀 Usage

Download the conversation histories, benchmark queries, and evaluation rubrics:

import json
from huggingface_hub import hf_hub_download

def load_pragma_file(filename):
    path = hf_hub_download(
        repo_id="stellahj/PRAGMA",
        filename=f"data/{filename}",
        repo_type="dataset",
    )
    with open(path, encoding="utf-8") as file:
        return json.load(file)

full_sessions = load_pragma_file("full_sessions.json")
metadata = load_pragma_file("metadata.json")
response_metrics = load_pragma_file("response_metrics.json")

Use full_sessions.json as the conversation history and metadata.json as the evaluation queries. Join them by user_id; for each query, evidence_session_index identifies the gold evidence sessions in the corresponding user's sessions array. Match the rubrics in response_metrics.json by (user_id, query).

alignment_metric evaluates whether the response is aligned with the user's relevant experience or trajectory. grounding_metric evaluates whether the response explicitly uses the relevant history. The experiments reported in the paper used GPT-5 (gpt-5-2025-08-07) as the default LLM judge.

🔗 Source and License

PRAGMA is constructed from sampled records from NVIDIA Privasis-Zero.

The dataset is distributed under the NVIDIA License and is intended for non-commercial research and evaluation use. See LICENSE for details.

🛠️ Construction Code

The benchmark construction pipeline and documentation are available in the PRAGMA GitHub repository.

📜 Citation

@misc{yu2026pragmaevaluatingpersonalizedguidance,
  title={PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations},
  author={Hyojeong Yu and Hyukhun Koh and Minsung Kim and Yunah Jang and Kyomin Jung},
  year={2026},
  eprint={2609.09664},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2609.09664},
}