README
Substream Recollection
A controlled benchmark for membership recall, designed to test properties of memory beyond accuracy in LLMs.
Each row is a (stream, probe, label) tuple: the model sees a long input stream and a short candidate, and answers whether the candidate (or in the case of natural video, the referenced action) occurred inside the stream.
| config | rows | content |
|---|---|---|
text | 7,640 | synthetic substream questions, text modality, L=8…4096 |
synthetic_video | 6,065 | the same streams rendered as video |
natural_video | 1,813 | 1,028 EPIC-Kitchens-100 + SoccerNet clips, L=8…1024, with two question sets over them — see question_set below |
easyhuman | 672 | 3-belt EasyHuman: 224 video rows (L=256) and 448 text rows (224 at L=256, 224 at L=1024). modality distinguishes them |
Loading
from datasets import load_dataset
ds = load_dataset("anonstreammem/substream-recollection", "natural_video")["train"]
Each config also ships questions.json (NDJSON) and manifest.json (nested, for
direct main.py ingestion).
video_path and clip_path are relative to the config that holds the media,
given by the media_config column, present on every config:
from huggingface_hub import snapshot_download
root = snapshot_download("anonstreammem/substream-recollection", repo_type="dataset")
path = f"{root}/{row['media_config']}/{row['video_path']}" # works for every config
Check media_included first: false means the mp4 is NDA-gated and absent, null
means the row has no media. Filter with media_included != False, not
df[df.media_included], or the null rows are dropped too.
Columns worth knowing
answer is the ground truth, "yes" / "no".
The key is (question_id, length_L, entropy_band, question_variant) — plus
modality in easyhuman, where each question appears once as text and once as video.
natural_video holds two question sets over the same 1,028 clips, told apart by
question_set:
question_set | rows | what it asks |
|---|---|---|
designed | 1,028 | the one question the clip was built for, at every length L=8…1024 |
density_topup | 785 | extra mined questions about other classes, added for the density analysis, on the 403 codec-banded clips (L=64…1024) |
They are disjoint: no shared question_id, and no clip is asked the same question
twice. Filter on question_set to recover either subset.
Natural-video-specific columns:
| column | meaning |
|---|---|
density_band | low / high, codec distortion at a fixed 32 kbit/frame budget, one global cutoff. null on the 625 designed rows at L=8/16 and on unbanded clips — those buckets predate the codec protocol |
event_start_frame, event_end_frame | integer frame indices of the clip's designed event — index the decoded clip directly |
event_clip_start, event_clip_end | the same instants in seconds. Clips are 1 fps with frame count = duration, so the frame is floor(seconds). Populated on 882 of 1,813 rows; null where the clip has no localised event |
event_timing | measured (EPIC: extent from narration) or centered_by_construction (SoccerNet: labels give an instant, so the clip is centred on it with a nominal 2 s window) |
clip_sha256 | sha256 of the exact mp4 published at media_config/video_path. null only where the media is NDA-gated |
label_source | how the label was derived: designed for the designed questions, one of narration_margin / whole_video_absence / labels_v2_window / labels_v2_window_absence for the mined ones |
query_class | the class this question asks about |
question_time | when the question is put, in clip seconds. Always length_L: the whole stream is shown first |
source_provenance | JSON, on every row: source_video_id, clip_start/end_in_source_seconds, event_position/end_in_source_seconds, event_type, slowdown_factor, anomaly_label, plus SoccerNet match/half/path/entry-id |
Event columns localise the clip's designed event, which is what this question asks
about only when question_set == "designed". No density_topup question asks about
its clip's designed class, so for those rows compare event_type with query_class
before reading the event columns as the answer's location.
89 groups of clips are byte-identical under different names — a clip built as
one class's positive is reused as another's negative. Group on clip_sha256 to
dedupe or check independence; contains_anomaly and the event columns describe the
named clip, not the file.
Licences
Synthetic and EasyHuman content is CC BY 4.0. EPIC-Kitchens-100 derived clips are
CC BY-NC 4.0. SoccerNet rows are provenance-only under the SoccerNet NDA. Per-row
license column; full texts in LICENSES/.
Citation
Anonymous, "Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy," 2026 (anonymized for review).
The original sources for the natural videos:
- Damen, D. et al. "Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100." IJCV, 2022.
- Giancola, S., Amine, M., Dghaily, T., Ghanem, B. "SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos." CVPR Workshops, 2018.
Contributors
22 commits
Similar repos
README
Substream Recollection
A controlled benchmark for membership recall, designed to test properties of memory beyond accuracy in LLMs.
Each row is a (stream, probe, label) tuple: the model sees a long input stream and a short candidate, and answers whether the candidate (or in the case of natural video, the referenced action) occurred inside the stream.
| config | rows | content |
|---|---|---|
text | 7,640 | synthetic substream questions, text modality, L=8…4096 |
synthetic_video | 6,065 | the same streams rendered as video |
natural_video | 1,813 | 1,028 EPIC-Kitchens-100 + SoccerNet clips, L=8…1024, with two question sets over them — see question_set below |
easyhuman | 672 | 3-belt EasyHuman: 224 video rows (L=256) and 448 text rows (224 at L=256, 224 at L=1024). modality distinguishes them |
Loading
from datasets import load_dataset
ds = load_dataset("anonstreammem/substream-recollection", "natural_video")["train"]
Each config also ships questions.json (NDJSON) and manifest.json (nested, for
direct main.py ingestion).
video_path and clip_path are relative to the config that holds the media,
given by the media_config column, present on every config:
from huggingface_hub import snapshot_download
root = snapshot_download("anonstreammem/substream-recollection", repo_type="dataset")
path = f"{root}/{row['media_config']}/{row['video_path']}" # works for every config
Check media_included first: false means the mp4 is NDA-gated and absent, null
means the row has no media. Filter with media_included != False, not
df[df.media_included], or the null rows are dropped too.
Columns worth knowing
answer is the ground truth, "yes" / "no".
The key is (question_id, length_L, entropy_band, question_variant) — plus
modality in easyhuman, where each question appears once as text and once as video.
natural_video holds two question sets over the same 1,028 clips, told apart by
question_set:
question_set | rows | what it asks |
|---|---|---|
designed | 1,028 | the one question the clip was built for, at every length L=8…1024 |
density_topup | 785 | extra mined questions about other classes, added for the density analysis, on the 403 codec-banded clips (L=64…1024) |
They are disjoint: no shared question_id, and no clip is asked the same question
twice. Filter on question_set to recover either subset.
Natural-video-specific columns:
| column | meaning |
|---|---|
density_band | low / high, codec distortion at a fixed 32 kbit/frame budget, one global cutoff. null on the 625 designed rows at L=8/16 and on unbanded clips — those buckets predate the codec protocol |
event_start_frame, event_end_frame | integer frame indices of the clip's designed event — index the decoded clip directly |
event_clip_start, event_clip_end | the same instants in seconds. Clips are 1 fps with frame count = duration, so the frame is floor(seconds). Populated on 882 of 1,813 rows; null where the clip has no localised event |
event_timing | measured (EPIC: extent from narration) or centered_by_construction (SoccerNet: labels give an instant, so the clip is centred on it with a nominal 2 s window) |
clip_sha256 | sha256 of the exact mp4 published at media_config/video_path. null only where the media is NDA-gated |
label_source | how the label was derived: designed for the designed questions, one of narration_margin / whole_video_absence / labels_v2_window / labels_v2_window_absence for the mined ones |
query_class | the class this question asks about |
question_time | when the question is put, in clip seconds. Always length_L: the whole stream is shown first |
source_provenance | JSON, on every row: source_video_id, clip_start/end_in_source_seconds, event_position/end_in_source_seconds, event_type, slowdown_factor, anomaly_label, plus SoccerNet match/half/path/entry-id |
Event columns localise the clip's designed event, which is what this question asks
about only when question_set == "designed". No density_topup question asks about
its clip's designed class, so for those rows compare event_type with query_class
before reading the event columns as the answer's location.
89 groups of clips are byte-identical under different names — a clip built as
one class's positive is reused as another's negative. Group on clip_sha256 to
dedupe or check independence; contains_anomaly and the event columns describe the
named clip, not the file.
Licences
Synthetic and EasyHuman content is CC BY 4.0. EPIC-Kitchens-100 derived clips are
CC BY-NC 4.0. SoccerNet rows are provenance-only under the SoccerNet NDA. Per-row
license column; full texts in LICENSES/.
Citation
Anonymous, "Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy," 2026 (anonymized for review).
The original sources for the natural videos:
- Damen, D. et al. "Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100." IJCV, 2022.
- Giancola, S., Amine, M., Dghaily, T., Ghanem, B. "SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos." CVPR Workshops, 2018.
Similar repos
Linked in READMEs
Contributors
22 commits