🧩 PRAGMA

This is the official repository for PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations (EMNLP 2026).

📂 Directory Layout

PRAGMA/
  config.yaml
  pipeline/
    config.py
    run_pipeline.py
    sample_users.py
    extract_schema.py
    create_events.py
    event_query.py
    create_traj.py
    traj_query.py
    evid_sessions.py
    filler_sessions.py
    full_sessions.py
  metrics/
    eval_criteria.py
  figures/
    pipeline.png
  requirements.txt

📦 Using the Released Benchmark

The released benchmark and evaluation rubrics are hosted in the PRAGMA dataset repository on Hugging Face. Download them into the expected local directory with:

python -m pip install -U huggingface_hub
hf download stellahj/PRAGMA \
  --repo-type dataset \
  --include "data/*.json" \
  --local-dir .

Most users only need:

• data/full_sessions.json: timestamp-ordered conversation histories for 100 users.
• data/metadata.json: 400 benchmark queries with query types and gold evidence-session indices.
• data/response_metrics.json: released alignment and grounding rubrics.

Join records by user_id. For each item in metadata.json, evidence_session_index selects the gold evidence sessions from that user's sessions array in full_sessions.json. Match rubric records in response_metrics.json by (user_id, query).

The experiments reported in the paper used GPT-5 (gpt-5-2025-08-07) as the default LLM judge for the released alignment and grounding rubrics.

🛠️ Reconstructing or Extending the Benchmark

The construction pipeline is provided for researchers who want to recreate PRAGMA or extend it with additional data.

Pipeline Overview

Installation

Python 3.10 or newer is recommended.

git clone https://github.com/yuhyojeong/PRAGMA.git
cd PRAGMA
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

On Windows PowerShell, activate the environment with .venv\Scripts\Activate.ps1 instead of source .venv/bin/activate.

Set an OpenAI API key before generating model-produced artifacts:

export OPENAI_API_KEY="your-api-key"

Configuration

Edit config.yaml to use other OpenAI models or change generation concurrency, retry limits, and user-sampling settings. Model IDs are configured separately for each construction task, so stages can be changed independently. Stages that request JSON-schema outputs require an OpenAI model that supports Structured Outputs.

evidence_requests and filler_requests directly limit the total number of concurrent API requests in their respective stages. Reduce these values if API rate limits are encountered. All configuration fields are required; missing or invalid values stop the pipeline with an explicit error.

Run the Pipeline

Run all construction stages:

python pipeline/run_pipeline.py

Include evaluation-rubric generation:

python pipeline/run_pipeline.py --with-metrics

python pipeline/run_pipeline.py --start create_events
python pipeline/run_pipeline.py --start event_query --end traj_query
python pipeline/run_pipeline.py --only event_query,traj_query
python pipeline/run_pipeline.py --skip sample_users
python pipeline/run_pipeline.py --dry-run

python metrics/eval_criteria.py

🧭 Query Types

📄 Data Reference

📝 Notes

• Generation scripts call OpenAI models and may take time.
• Keep queries, evidence fields, metadata, and evaluation rubrics synchronized when extending the benchmark.

📜 Citation

@misc{yu2026pragmaevaluatingpersonalizedguidance,
  title={PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations},
  author={Hyojeong Yu and Hyukhun Koh and Minsung Kim and Yunah Jang and Kyomin Jung},
  year={2026},
  eprint={2609.09664},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2609.09664},
}