CogEval-Bench

A structural diagnostic benchmark for evaluating whether an agent's memory substrate forms cognitive structure under continuous event streams — not just whether it can retrieve facts. CogEval-Bench is the structural-evaluation half of the two-layer evaluation in CogniFold: Always-On Proactive Memory via Cognitive Folding (Wang et al., 2026).

Why structural diagnostics?

Conventional memory benchmarks (LoCoMo, LongMemEval, MuSiQue, …) measure end-to-end retrieval accuracy — they reward a system that returns the right fact, but they cannot tell whether the underlying memory has formed cognitive structure or whether it is just keyword-matching. CogEval-Bench inverts the setup:

1. Gold first. For each scenario, a hand-specified gold concept graph $\mathcal{G}^* = (\mathcal{C}^, \mathcal{R}^, \mathcal{H}^, \mathcal{I}^)$ defines the concepts, inter-concept relationships, hierarchy, expected emergent intents, and planted multi-hop chains a competent memory should form.
2. Events grounded from gold. First-person events are generated from the gold graph, then ~12% distractor events are added and the stream is temporally shuffled.
3. Score the topology, not the answer. A memory system ingests the stream; we then score the formed graph against $\mathcal{G}^*$ on three tracks (concept quality, topology, compression × proactivity).

Files

gold_graphs/
├── academic_research.json
├── customer_support.json
├── health_journey.json
├── news_stream.json
├── software_engineer.json
└── team_project.json

generated/
├── academic_research_small.json    academic_research_medium.json
├── customer_support_small.json     customer_support_medium.json
├── health_journey_small.json       health_journey_medium.json
├── news_stream_small.json          news_stream_medium.json
├── software_engineer_small.json    software_engineer_medium.json
└── team_project_small.json         team_project_medium.json

gold_graphs/<scenario>.json

The reference cognitive structure for one scenario:

generated/<scenario>_<scale>.json

A self-contained generated stream + QA bundle:

Evaluation tracks

Standalone Python evaluators are bundled in this dataset under scripts/ — concept_evaluator.py (Track A + B) and compression_evaluator.py (Track C). The same scripts plus the LLM-assisted generator generate_dataset.py let you reproduce or extend the dataset; see scripts/README.md. Full evaluation harness lives in the CogniFold repo at benchmarks/cogeval/.

Quick start

from datasets import load_dataset
import json
from huggingface_hub import hf_hub_download

# Load one generated stream
path = hf_hub_download(
    repo_id="OpenNorve/CogEval-Bench",
    repo_type="dataset",
    filename="generated/academic_research_small.json",
)
stream = json.loads(open(path).read())
print(f"{stream['statistics']['total_events']} events, "
      f"{stream['statistics']['total_questions']} questions")

# Load the matching gold graph
path = hf_hub_download(
    repo_id="OpenNorve/CogEval-Bench",
    repo_type="dataset",
    filename="gold_graphs/academic_research.json",
)
gold = json.loads(open(path).read())
print(f"{len(gold['concepts'])} gold concepts, "
      f"{len(gold['relationships'])} relationships, "
      f"{len(gold['expected_intents'])} expected intents")

To run the full benchmark on the CogniFold substrate:

git clone https://github.com/OpenNorve/CogniFold.git && cd CogniFold
bash scripts/reproduce.sh cogeval

Citation

@article{wang2026cognifold,
  title   = {CogniFold: Always-On Proactive Memory via Cognitive Folding},
  author  = {Wang, Suli and Duan, Yiqun and Deng, Yu and Zhao, Rundong and Shi, Dai and Zhou, Xinliang},
  journal = {arXiv preprint arXiv:2605.13438},
  year    = {2026},
  url     = {https://arxiv.org/abs/2605.13438}
}

License

Apache-2.0 — see LICENSE.