The International Olympiad in Informatics (IOI) is one of five international science olympiads (if you are familiar with AIME, IOI is the programming equivalent of IMO, for which the very best students who take part in AIME are invited) and tests a very select group of high school students (4 per country) in complex algorithmic problems.
The problems are extremely challenging, and the full test sets are available and released under a permissive (CC-BY) license. This means that IOI is the perfect dataset to test a model's code reasoning capabilities.
See Problem statements dataset for more info.
To load and parse tests for a given year (e.g., 2021):
from datasets import load_dataset
from collections import defaultdict
tests_dataset = load_dataset("open-r1/ioi-test-cases", name="2021", split="train")
test_cases = defaultdict(dict)
for test_case in tests_dataset:
test_cases[test_case['problem_id']][test_case['test_name']] = test_case['test_input'], test_case['test_output']
You can then index test cases by their test_name value from open-r1/ioi
open-r1/ioiopen-r1/ioi-test-cases (you are here!)open-r1/ioi-sample-solutionsopen-r1/ioi-2024-model-solutionsTo have models generate solutions to IOI problems, follow the instructions here.
To run tests on generated solutions, follow the instructions here.
All data was extracted, parsed, and adapted from the official IOI Tasks, which are released under CC-BY:
Similarly to the original problem data, this and accompanying datasets are released under CC-BY.
The International Olympiad in Informatics (IOI) is one of five international science olympiads (if you are familiar with AIME, IOI is the programming equivalent of IMO, for which the very best students who take part in AIME are invited) and tests a very select group of high school students (4 per country) in complex algorithmic problems.
The problems are extremely challenging, and the full test sets are available and released under a permissive (CC-BY) license. This means that IOI is the perfect dataset to test a model's code reasoning capabilities.
See Problem statements dataset for more info.
To load and parse tests for a given year (e.g., 2021):
from datasets import load_dataset
from collections import defaultdict
tests_dataset = load_dataset("open-r1/ioi-test-cases", name="2021", split="train")
test_cases = defaultdict(dict)
for test_case in tests_dataset:
test_cases[test_case['problem_id']][test_case['test_name']] = test_case['test_input'], test_case['test_output']
You can then index test cases by their test_name value from open-r1/ioi
open-r1/ioiopen-r1/ioi-test-cases (you are here!)open-r1/ioi-sample-solutionsopen-r1/ioi-2024-model-solutionsTo have models generate solutions to IOI problems, follow the instructions here.
To run tests on generated solutions, follow the instructions here.
All data was extracted, parsed, and adapted from the official IOI Tasks, which are released under CC-BY:
Similarly to the original problem data, this and accompanying datasets are released under CC-BY.