BixBench Dataset

Contains the dataset file BixBench.jsonl and each corresponding data capsule as a .zip file. Capsules are named CapsuleFolder-{uuid}.zip

IMPORTANT UPDATE 2025/09/23: In ongoing work, we found that many questions in the original dataset had insufficient detail to be answerable, especially in the preferred open-answer setting. To address this, we've extensively re-reviewed and revised a substantial portion of the benchmark. We have also updated the format of the dataset by flattening it to one question per row. This is a significant change and you should expect changes in performance (generally improved.) We will be updating our preprint soon with more details and our own updated results, as well as updating the public evaluation harness on GitHub shortly. We've left the original question set accessible under the v1.0 tag. The current updated set is at main and v1.5.