Valen-Eval-General-5k

GitHub · 中文 README · Preview model · Technical notes

✨ Introduction

Valen brings visual perception to System One decision-making: text, images or video in, candidate probabilities out. This dataset provides 5,000 image-based decision records for held-out evaluation, spanning visual question answering, interfaces, games and documents.

Each record contains one decision question, a target probability distribution, local image references and provenance metadata. All records are marked as English in the supplied metadata.

Keep these records out of training and report evaluation results over all 5,000 records. This is a static image-question test, not an interactive end-to-end game benchmark. Use Valen-Training-General-100k for companion training.

📥 Dataset downloads

Images are packaged in assets.zip. Extract it before loading records; image paths and data_manifest.json refer to the extracted files. Despite its filename, train.jsonl contains evaluation data here: the Hub split is test, while record metadata is eval.

📊 Dataset composition

• Records / questions: 5,000 / 5,000.
• Referenced image files: 2,500; unique image SHA-256 hashes: 2,390.
• Distinct groups: 2,390. Multiple questions can share an image or group.
• Domains: vqa: 1,666, ui: 1,428, game: 953, document: 953.
• Local split metadata: meta.split = "eval".

noul is a binary judgment with implicit candidate keys true and false. choice maps candidate IDs to answer descriptions in criteria. score uses an ordered list of descriptions; target keys are the zero-based index strings ("0", "1", ...). All supplied targets are one-hot distributions; these are finite-candidate decisions rather than free-form text targets.

🚀 Quick start

Download the dataset and extract its image archive:

hf download Valen-Team/Valen-Eval-General-5k \
  --repo-type dataset --local-dir data/Valen-Eval-General-5k
unzip -n data/Valen-Eval-General-5k/assets.zip -d data/Valen-Eval-General-5k

After downloading the repository with its image files, use Python's JSON reader. This preserves the heterogeneous criteria field, which is a mapping for choice questions, a list for score questions and absent for binary questions. Automatic Arrow/JSON loading may require normalization of that field.

import json
from pathlib import Path
from PIL import Image

root = Path("data/Valen-Eval-General-5k")
with (root / "train.jsonl").open(encoding="utf-8") as stream:
    record = json.loads(next(stream))

request = record["request"]
question = request["questions"]["decision"]
probabilities = record["targets"]["decision"]["probabilities"]
correct_candidate = max(probabilities, key=probabilities.get)
with Image.open(root / record["assets"][0]["path"]) as image:
    print(image.size, question["type"], correct_candidate)

For candidate-selection evaluation, compare the predicted candidate ID against the target distribution, and report overall accuracy together with per-source and per-question-type accuracy. Score questions also allow ordered-error metrics when explicitly defined; noul outputs may be evaluated as probabilities when calibration is relevant.

From a configured Valen checkout, evaluate a trained checkpoint with:

python -m valen.evaluate \
  --checkpoint /path/to/checkpoint \
  --data data/Valen-Eval-General-5k/train.jsonl \
  --output output/general_5k_eval

The checkpoint requires its matching base model. See the model downloads and keep config.json pointed at the local base weights. Published General results use General-trained checkpoints; the Sokoban-trained Preview is a different checkpoint.

Record format

Only request is model input. Keep targets, grouping identifiers and provenance outside the model prompt.

Repository layout

train.jsonl           # Complete 5,000-record evaluation file
<source>.jsonl        # Ten per-source subsets of train.jsonl
assets/<source>/...   # Images referenced by relative path
dataset_info.json     # Counts, source metadata and validation results
data_manifest.json    # SHA-256 and byte size of every supplied data file
README.md
card-assets/         # Dataset-card branding and community QR code

The original filename train.jsonl is preserved for compatibility. In this repository it contains evaluation data, despite its name; the Hub split is named test and record metadata remains eval. The complete train.jsonl is the canonical entry point: its group IDs have been normalized, while some per-source files retain original grouping prefixes. The number of such differences is recorded in dataset_info.json; all other record fields match. Per-source files contain the same examples as the complete file: read either train.jsonl or the source subsets, not both.

All image paths are relative to the repository root. The image files are stored in the ZIP archives listed below; extract them before using these relative paths. The dataset payload occupies approximately 0.31 GiB, including both the complete JSONL and its per-source copies.

Sources and provenance

The table reproduces source identifiers and license strings from the supplied record metadata; it is not an independent verification of upstream licensing or label quality.

Upstream identifiers, revisions, source splits and label provenance are retained in each record's meta fields where provided. meta.source_split describes the upstream source partition; meta.split describes this assembled dataset partition. They need not have the same name.

Historical identifiers such as visionjev/programmatic and Visual-Jev-Eval-General-5k remain in the original records and metadata. The project and Hub repository are now named Valen; the dataset payload is unchanged.

Integrity and split isolation

Packaging validated all record schemas, unique record IDs, target distributions, referenced image paths and image SHA-256 hashes. Each per-source subset was checked against the corresponding records in the complete JSONL, allowing only the documented group-ID normalization difference. Across the 100k training and 5k evaluation repositories, overlap is 0 record IDs, 0 group IDs and 0 exact image SHA-256 hashes. This does not establish semantic or near-duplicate independence.

The complete file SHA-256 is:

8505dc5a4cb4db9bdc193eface2abb005b7c963d264f83d7994ed0bb57565a15

Counts and audit details are in dataset_info.json; data-file checksums are in data_manifest.json. The original JSONL records and images are preserved without relabeling or rewriting.

Image archives

Images are distributed as ZIP archives. Each archive preserves its folder prefix and all nested paths. The original image directories have been removed after verifying their contents against the archives. Extract the ZIP files to restore them. Record paths are unchanged; after downloading archives, extract them from the repository root before loading the JSONL data.

unzip -n assets.zip -d .

unzip -n preserves existing extracted files. image_archives.json records archive SHA-256 hashes, source sizes and file counts. Every member was decompressed and checked against the original data manifest by SHA-256; ZIP CRC checks also passed. ZIP64 is enabled for large archives.

🤝 Contributions

Contributions to Valen are welcome. Open an issue to report a problem, share a use case or discuss experimental results. Submit a pull request to improve the code or documentation, contribute training data or add evaluation tasks.

Scan the QR code below to join the Valen WeChat group, discuss the project and share your experiments.

📄 Annotation status and terms

The supplied metadata marks 4,953 records as human_audit: pending and 47 as not_required. Packaging checks establish structural and file integrity, not independent semantic verification of every answer.

The repository retains its existing Apache-2.0 metadata tag. This does not replace the source-specific terms recorded above for third-party images and annotations. Source attribution and license metadata remain attached to the records.

Valen is built on Qwen3.5, with its decision interface inspired by TypeSafe's Jev. Project documentation and evaluation results are available in the GitHub README.