Chrisyichuan/screenshot-training-natural-filtered-v2

Wikipedia screenshot retrieval training dataset exported from local hard-negative mining.

Contents

• train.jsonl / train_hn.jsonl
• eval.jsonl / eval_hn.jsonl
• test.jsonl / test_hn.jsonl
• train_hn_with_answer.jsonl / eval_hn_with_answer.jsonl / test_hn_with_answer.jsonl
• lite-query-v2-full-filtered-hn-with-answer.jsonl
• images/

Each metadata row has the form:

{
  "query": "...",
  "chunk_path": "images/shard_123/shard_00001/123456.png.tiles/chunk_0000_00.png",
  "neg_chunk_paths": [
    "images/shard_234/shard_00002/234567.png.tiles/chunk_0000_01.png"
  ],
  "split": "train"
}

The answer-enriched metadata adds one more field:

{
  "query": "...",
  "chunk_path": "images/shard_123/shard_00001/123456.png.tiles/chunk_0000_00.png",
  "neg_chunk_paths": [
    "images/shard_234/shard_00002/234567.png.tiles/chunk_0000_01.png"
  ],
  "answer": "...",
  "split": "train"
}

Split sizes

• train: 104033
• eval: 5779
• test: 5781

Notes

• Image paths are stored relative to the dataset root.
• The source images were deduplicated before export so repeated hard negatives only upload once.
• The primary split files are query-filtered hard-negative metadata without answers.
• Additional *_with_answer.jsonl files were joined back to the original lite-query-v2-full-filtered.jsonl source via (query, chunk_path) with 100.0% match rate for this cleaned subset.

Image Storage

The images are stored as 1000 tar shards under image_shards/ to keep the repository file count low and make uploads/downloads more reliable.

To materialize the images locally after download:

python extract_hf_image_shards.py --dataset-dir .