ESL-bench

ESL-bench (Event-driven Synthetic Longitudinal Benchmark) is a virtual health user dataset for evaluating AI health assistants. Each virtual user contains a complete health profile, event timeline, clinical exam data, and knowledge-graph-grounded evaluation queries, designed for use with the Mirobody-Eval framework.

⚠️ Research use only. Outputs are synthetic and intended for benchmarking AI agents. They should not be used for diagnosis or treatment decisions.

News

• 2026-08-31 — New leaderboard dataset: sample320-20260830 (batch 202608, 320 questions over 20 virtual users, user5300_AT_demo ~ user5319_AT_demo). The five capability dimensions carry 40 items each (200 total) and the behavioural dimension grows from 80 to 120 items, so the mix is 40x5 + 120. Breaking change for tag-based parsers: the behavioural items are now tagged difficulty:Adversarial (they were difficulty:Distractor in 202607). Also published: full-20260830 (2000 questions) and per-user raw data for all 20 users. The previous dataset (sample280-20260730) remains accessible for historical comparison; note that scores are not directly comparable across the two, since the question mix differs.

• 2026-07-30 — New leaderboard dataset: sample280-20260730 (batch 202607, 280 questions over 20 virtual users). This batch introduces a sixth evaluation dimension, Inducement: 80 behavioural items that plant a premise in the conversation and check whether the agent resists fabricated premises without becoming over-skeptical of true ones. Scored as the F1 of resist-false-premise rate and accept-true-premise rate — a model that denies everything scores 0. The five capability dimensions carry 40 items each (200 total), so the question mix is 40x5 + 80. Also published: full-20260730 (2000), sample500, sample200 and sample50, all drawn from the same 20 users. The previous dataset (sample200-20260628) remains accessible for historical comparison.

• 2026-06-28 — New leaderboard dataset: sample200-20260628 (batch 202606, 200 questions over 40 virtual users). Mixed-language batch (115 Chinese items); scores are not directly comparable with the all-English 202605 release. The previous dataset (sample200-pro-0530) remains accessible for historical comparison.

• 2026-05-30 — New leaderboard dataset: sample200-pro-0530 (batch 202605, 200 questions over 20 virtual users). The questions are brand-new: an evolutionary algorithm takes items a target model already answers correctly and rewrites them into harder ones over the same knowledge graph and scoring protocol, with every answer still deterministically computable. The previous dataset (sample200-20260512) remains accessible for historical comparison.

Paper

For the full technical description of the dataset generation pipeline, evaluation taxonomy, and experimental results, see our paper:

Event-Driven Synthetic Longitudinal Benchmark for Health Agents (PDF)

The paper presents:

• An event-driven agentic synthesis pipeline that generates realistic longitudinal health trajectories with temporal response kernels (sigmoid onset + exponential fade-out)
• A five-dimension, three-tier evaluation taxonomy (Lookup, Trend, Comparison, Anomaly, Explanation x Easy/Medium/Hard) with deterministic ground truth derived from the event-indicator knowledge graph
• A two-stage scoring protocol combining programmatic checks with LLM-based rubric evaluation
• Experimental results comparing structured-access agents, text-based retrieval, and RAG approaches across all dimensions

Dataset Summary

Each patient trajectory is generated by an event-driven agentic pipeline that:

1. Samples a patient profile (demographics, lifestyle, conditions)
2. Selects a personalized set of health indicators (device-style + exam-style)
3. Synthesizes daily wearable measurements driven by discrete events (exercise changes, diet shifts, medication, acute infections, etc.)
4. Generates sparse clinical exam records
5. Records a structured event log explaining why indicators change
6. Produces a knowledge-graph-grounded objective evaluation query set with deterministic ground truth

Indicator dynamics follow a temporal response model: sigmoid onset + exponential fade-out, with multi-event superposition and physiological range constraints.

Dataset Structure

├── manifest.json                          # Change-detection entry point (root)
├── README.md
└── data/
    └── <YYYYMM>/                          # Monthly batch directory
        ├── sample50-YYYYMMDD.json         # Sampled 50 benchmark questions (questions and difficulty infomation for all virtual users)
        ├── full-YYYYMMDD.json             # Benchmark questions (questions and difficulty infomation for all virtual users)
        └── <email_id>/                    # Single virtual user, e.g.: user110_AT_demo
            ├── profile.json               # User profile (demographics, history, family history)
            ├── timeline.json              # Events and indicators timeline
            ├── kg_evaluation_queries.json           # Knowledge-graph evaluation questions and groundtruth (available after the competition closes)
            ├── kg_evaluation_queries_YYYYMMDD.json  # Versioned evaluation queries for different batch dates (same user, different question batches)
            ├── exam_data.json             # Clinical data (lab tests, imaging)
            └── plausibility_report.json   # Data plausibility report

manifest.json Format

{
  "version": 1,                           // Monotonically increasing integer; +1 on any change
  "updated_at": "2026-03-06T10:00:00Z",   // Last update timestamp
  "latest": "data/202608/sample320-20260830.jsonl",  // Path to the latest evaluation dataset file
  "batches": {
    "202603": {
      "created_at": "2026-03-04T00:00:00Z",
      "checksum": "0f3bc6c60fb5c4b2",    // First 16 hex chars of batch SHA-256
      "user_count": 120,
      "users": ["user110_AT_demo", "user111_AT_demo", "..."]
    }
  }
}

Field Reference

Data Access

Fetch Latest Batch

from huggingface_hub import hf_hub_download, snapshot_download
import json

REPO_ID = "healthmemoryarena/ESL-Bench"

# 1. Read manifest to get latest batch
manifest_path = hf_hub_download(repo_id=REPO_ID, filename="manifest.json", repo_type="dataset")
manifest = json.loads(open(manifest_path).read())
latest_file = manifest["latest"]  # e.g. "data/202604/sample50-20260331.jsonl"
latest_batch = latest_file.split("/")[1]  # e.g. "202604"

# 2. Download only the latest batch
snapshot_download(
    repo_id=REPO_ID,
    repo_type="dataset",
    allow_patterns=[f"data/{latest_batch}/*"],
)

Incremental Update

import json
from pathlib import Path
from huggingface_hub import hf_hub_download, snapshot_download

REPO_ID = "healthmemoryarena/ESL-Bench"
LOCAL_DIR = Path("./eslbench_data")

# 1. Fetch remote manifest (single file, fast)
remote_manifest_path = hf_hub_download(repo_id=REPO_ID, filename="manifest.json", repo_type="dataset")
remote_manifest = json.loads(open(remote_manifest_path).read())

# 2. Load local manifest (empty dict on first run)
local_manifest_path = LOCAL_DIR / "manifest.json"
local_manifest = json.loads(local_manifest_path.read_text()) if local_manifest_path.exists() else {}

# 3. Fast check: same version means no changes
if remote_manifest["version"] == local_manifest.get("version"):
    print("No changes.")
else:
    # 4. Compare per-batch checksums; download only changed batches
    for batch_id, batch_info in remote_manifest["batches"].items():
        local_batch = local_manifest.get("batches", {}).get(batch_id)
        if local_batch and local_batch["checksum"] == batch_info["checksum"]:
            continue  # unchanged, skip
        print(f"Downloading batch {batch_id} ({batch_info['user_count']} users)...")
        snapshot_download(
            repo_id=REPO_ID,
            repo_type="dataset",
            allow_patterns=[f"data/{batch_id}/*"],
            local_dir=str(LOCAL_DIR),
        )
    # 5. Save updated manifest locally
    local_manifest_path.parent.mkdir(parents=True, exist_ok=True)
    local_manifest_path.write_text(json.dumps(remote_manifest, indent=2))
    print(f"Updated to version {remote_manifest['version']}")

Change Detection

Batch Naming

• Format: YYYYMM (e.g., 202603 = March 2026 batch)
• New users each month are added as a new batch; existing batch directories are not modified
• If a batch is revised, update its files and increment checksum and version in the manifest

Current Batches

Released Answers

Once a batch is retired (a newer batch becomes the active evaluation set), its ground-truth answers are released as separate *-with-answers.jsonl files alongside the original stripped versions, which remain unchanged for backward compatibility.

The -with-answers.jsonl files restore the four answer fields (eval.evaluator, eval.expected_value, eval.key_points, eval.source_data) on top of the public schema. They omit user.target_overrides (internal routing config) — answer fields only.

import json
from huggingface_hub import hf_hub_download

p = hf_hub_download(
    repo_id="healthmemoryarena/ESL-Bench",
    filename="data/202605/full-20260520-with-answers.jsonl",
    repo_type="dataset",
)
with open(p) as f:
    for line in f:
        item = json.loads(line)
        # item["eval"]["expected_value"], item["eval"]["key_points"], ...

File Schemas

profile.json

{
  "metadata": { "profile_id": "user110_AT_demo", "language": "en", ... },
  "demographics": { "age": 34, "gender": "Male", "occupation": "...", ... },
  "personality": { ... },
  "health_profile": { "baseline_indicators": {...}, "conditions": [...], ... }
}

timeline.json

A unified chronological timeline merging events, device measurements, and clinical exam results. This is the primary data source that health agents should ingest for each virtual user.

{
  "user_id": "user110_AT_demo",
  "generated_at": "2026-03-23T15:43:11.475119",
  "entry_count": 58727,
  "entries": [
    {
      "time": "2023-06-02T00:00:00",
      "end_time": "2026-03-21T00:00:00",
      "entry_type": "event",
      "event": {
        "event_id": "long_term_habit_1",
        "event_type": "long_term_habit",
        "event_name": "Regular morning jogging 3 days/week",
        "start_date": "2023-06-02",
        "duration_days": 1023,
        "interrupted": false
      }
    },
    {
      "time": "2023-06-02T13:55:34",
      "entry_type": "device_indicator",
      "indicator": "24-HourAverageDiastolicBP",
      "device_type": "bp_monitor",
      "value": 79.2,
      "unit": "mmHg"
    },
    {
      "time": "2023-06-02T09:00:00",
      "entry_type": "exam_indicator",
      "exam_type": "routine",
      "exam_location": "hospital",
      "indicator": "AlanineAminotransferase-ALT",
      "value": 24.0,
      "unit": "U/L"
    }
  ]
}

Each entry has one of three entry_type values:

kg_evaluation_queries.json / kg_evaluation_queries_YYYYMMDD.json

Each file contains 100 objective questions and answers constructed from the patient's event-indicator knowledge graph (KG). Answers are computed deterministically from the local JSON artifacts — no external medical knowledge is required for scoring.

The KG links each event instance to affected indicators and time windows, enabling objective questions at multiple reasoning depths: direct lookup, single-hop relation traversal, multi-hop aggregation, and temporal window reasoning.

A single user may have multiple versioned query files (kg_evaluation_queries_YYYYMMDD.json), each representing a different batch of generated questions. The base kg_evaluation_queries.json is the default set; dated files contain additional or refreshed question batches for the same user.

{
  "user_id": "user110_AT_demo",
  "evaluation_queries": [
    {
      "query_id": "user110_AT_demo_Q001",
      "difficulty": "direct",
      "query_text": "Is the overall health effect of ... positive, negative, or neutral?",
      "ground_truth": {
        "answer_type": "text",
        "expected_value": "negative",
        "key_points": [...],
        "source_data": { "data_type": "event_health_effect", ... }
      }
    }
  ]
}

Evaluation JSONL Format

Starting from batch 202604, pre-built evaluation JSONL files are available under data/{batch}/:

Each line is a JSON object with the following structure:

{
  "id": "user5022_AT_demo_Q082",
  "title": "[Explanation] Did 'event_name' have a positive or negative health effect?",
  "description": "Full question text with instructions",
  "user": {
    "type": "manual",
    "strict_inputs": ["The exact question sent to the agent"],
    "target_overrides": {
      "theta_api": { "email": "user5022@demo" },
      "llm_api": { "tool_context": { "user_email": "user5022@demo" } }
    }
  },
  "eval": {
    "evaluator": "kg_qa",
    "answer_type": "text | numeric_value | list | boolean",
    "expected_value": "ground-truth answer",
    "key_points": ["key fact 1", "key fact 2"],
    "difficulty": "Lookup | Trend | Comparison | Anomaly | Explanation",
    "source_data": {
      "data_type": "query category identifier",
      "...": "additional fields depending on query type"
    }
  }
}

Field Reference

Answer Types

Evaluation Taxonomy

ESL-bench organizes evaluation queries along two axes: 5 user-centric capability dimensions and 3 difficulty tiers, yielding a 5x3 matrix that isolates distinct failure modes and reasoning depths. From batch 202607 onward a sixth, behavioural dimension — Inducement — is evaluated alongside them; it is not part of the 5x3 matrix and is scored by its own protocol (see News).

Five Dimensions

Three Difficulty Tiers

Queries are distributed equally across the five dimensions (20 per dimension per user). All answers are computed deterministically from the event-indicator-time knowledge graph; Explanation queries additionally use a rubric that checks evidence grounding and baseline consistency.

Generation Pipeline

Data is generated by the ESL-bench agentic pipeline using:

• LLM: Gemini 3 Pro (profile generation, event synthesis)
• Temporal model: sigmoid onset + exponential fade-out per event-indicator pair
• Physiological constraints: per-indicator plausibles [L_k, U_k]
• KG-based query generator: deterministic rule engine over the event-indicator knowledge graph

Citation

If you use this dataset, please cite:

@article{eslbench2026,
  title     = {{ESL-bench}: An Event-Driven Synthetic Longitudinal Benchmark
               for Health Agent},
  author    = {Chao Li and Cailiang Liu and Ang Gao and Kexin Deng and Shu Zhang and Langping Xu and Xiaotong Shi and Xionghao Ding and Jian Pei and Xun Jiang},
  year      = {2026},
  eprint    = {2604.02834},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url       = {https://arxiv.org/abs/2604.02834}
}

License

Apache 2.0