Open-Jev v1.1 data

This release publishes 326,619 typed decision records, including 147,139 training records, from the frozen community-hard-mix-v2-final mixture used for the new Open-Jev 27B training stage. It is a redistributable projection, not the exact complete training or evaluation dataset: 2,053 Wikispeedia records are omitted because a separate redistribution license for the archived graph/path data has not been verified.

Project website · Code · Previous data release · Collection

Exact split counts

The complete mixture manifest SHA-256 is 7b26f948d2ae11f20d7a18be437d1ada616fc479e1596ae94be7596787fe4e54. Retained rows preserve their original bytes, order, IDs, target distributions and split membership. No Wiki task text, options, graph or paths are included. The exact original manifest and exclusion positions with content hashes are provided for reconstruction with separately obtained Wiki records.

Training and evaluation scope: the 27B stage trained on the complete 148,639-row training split for 37,160 global-batch-four optimizer steps. The final step wraps one training row. Published projection counts must not replace those training counts. The 512-row internal Test/OOD panels and the complete internal splits include their original Wiki rows; published metrics are not measurements on this filtered projection. Accuracy excludes soft-target rows, while distribution metrics can use all rows. This v1.1 name identifies the data release and does not imply that every model size has completed training on it.

What changed

The complete mixture adds 51,200 original controlled decision rows across eight authored domains: support, information retrieval, contracts, RAG, browser tools, shell history, games and rubric judging. These cover 80 authored policy families and 6,400 underlying scenarios; related views and counterfactuals are correlated. They are not 80 independently collected external sources or proof of broad real-world coverage.

It also contains 101,207 WANLI-derived English natural-language-inference records: 82,045 Train, 4,707 Calibration, 4,830 Validation, 4,998 Test and 4,627 OOD. The remaining records replay existing Open-Jev controls. Replay training is capped at 1,500 rows per source, with hash-ordered round-robin selection across parent groups; retained new-source training rows are used once per pass. Older held-out records retain their existing split assignments.

WANLI is one English NLI source. Its OOD split holds out connected seed/premise components within that source; it is not a new-language or new-domain benchmark. Other OOD definitions depend on each corpus and often reserve templates, rule families, layouts or goals. Test/OOD composition differs from the earlier release, so aggregate scores across versions are not a paired comparison.

Load and recover the original record

import json
from datasets import load_dataset

ds = load_dataset("ZefanCai/Open-Jev-v1.1", "community-hard-mix-v2-redistributable")
record = json.loads(ds["train"][0]["record_json"])

Use a pinned dataset commit for reproducible experiments. The Parquet schema contains id, group_id, split, source, kind, question, options, target, state_json, metadata_json, record_json, and original_line_number. JSON string columns preserve heterogeneous state/metadata without forcing a shared nested Arrow schema. record_json decodes to the original training record; original_line_number refers to its position in the complete mixture, so gaps identify omitted rows.

kind is choice, noul or score. Targets are distributions over the supplied options; they are not automatically empirical uncertainty estimates. Read metadata.target_basis: labels may come from exact synthetic rules, a known stochastic construction, human annotations or uniform sets of optimal actions. For Noul, the distribution is [P(no), P(yes)]. Dataset metadata, control latents and target distributions are supervision/provenance fields, not model input. The model input comprises state, question, kind and options.

The deterministic gzip files under raw/ preserve the exact retained JSONL bytes. Parquet/raw round-trip verification and original/public split hashes are recorded in export-manifest.json. To restore the full original mixture, supply separately obtained wikispeedia-v1 split files:

python reproduce/restore_original_mixture.py \
  --release-root /path/to/downloaded/Open-Jev-v1.1 \
  --wiki-dir /path/to/separately-obtained/wikiracing \
  --output /path/to/new/full-original-mixture

The utility selects the exact retained Wiki subset by canonical record hashes, including the 1,500-row training replay subset, restores row order and verifies every original split checksum. It does not download the Wiki archive or grant rights to it.

Licensing and attribution

This is a source-dependent release. WANLI text and derivatives remain CC BY 4.0. Original generated Open-Jev content is dedicated under CC0 1.0; source code is MIT. These terms do not relicense external wording, assets, model weights or datasets. Customer-control provenance retains its existing qualification: generated conversations are CC0, while short upstream question descriptions are from TypeSafe documentation whose source license was not verified. See LICENSE-DATA.md and THIRD_PARTY_NOTICES.md.

WANLI attribution: Alisa Liu, Swabha Swayamdipta, Noah A. Smith and Yejin Choi (2022), WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation. Source repository, pinned revision 61c95318fd71c55b6ba355d76253254615f387ec; CC BY 4.0. The source dataset card is retained in provenance/WANLI-README.md.

Modifications: premise/hypothesis pairs become a typed Choice over entailment, contradiction and neutral; premise text, hypothesis text and retained human gold are preserved. Options are deterministically reordered. Connected MNLI-seed/normalized-premise components are kept within splits, official Test components are excluded from Train/Calibration/Validation/OOD, and duplicate or conflicting pairs are removed. Every WANLI record retains original IDs, source split, revision, source-file hash, attribution and conversion notes. Human-reviewed labels may still be wrong.

Evaluation separation and limits

No JevBench or Frontier-100 task payload is published here. The recorded lexical screen compared the complete 148,639 training rows against 531 frozen evaluation requests using visible state/question/options and found no matching rows under that screen. It did not use benchmark answers or rationales. This does not prove absence of paraphrases, semantic overlap or foundation-model pretraining exposure. See the copied screening evidence in provenance/benchmark-overlap-screen.json.

Export validation checks unique row IDs, cross-split parent-group separation, source hashes, labels, omission counts and exact Parquet/raw round trips. Such checks establish data integrity, not semantic correctness or benchmark performance. No evaluation payload, API credentials, private mailbox contents, game ROMs, media assets or model weights are included in this dataset release.