A continuous, long-horizon benchmark for ambitious agents
Rust
1
3 commits
updated Oct 4, 2026
See what people are saying
README
BinkBench - a continuous, long-horizon benchmark for ambitious agents
How good are agents at video engineering?
Intro
This benchmark is based off an idea which originated from my Blogspot article, although it's more centred on codec engineering, rather than reverse-engineering.
BinkBench is a proof-of-concept benchmark for coding agents based on four open metrics - VMAF (as primary), PSNR and SSIM for measuring video quality, as well as bits per pixel (bpp) for measuring efficiency. The former three metrics can also be comprised into one quality metric (via a geometric mean).
BinkBench is also contamination-resistant, since there is no public encoder for the KB2i revision of Bink 2 for agents to find. The agent must derive one from the decoder alone, not retrieve a known solution. This also allows us to have internet access on for any research into compression techniques for encoding, because no matter how hard they try, they won't be able to find a reference encoder. It's worth putting here that the most likely way the agent gets access to an encoder is through someone uploading a previous run or the agent navigating to HuggingFace, which will get sorted out in the future (or compromising Epic Games).
What's different about it?
The majority of popular benchmarks today score based on the amount of tasks completed, instead of how good you can get at tasks specifically - collapsing the ability to see how well an agent did on something specific to a score of zero or one. For example, simplified:
- DeepSWE - does the agent pass or fail this specific test?
- FrontierCode - should this be merged?
While this is a stable standard, it doesn't mean that every other benchmark should work this way. For example, Vending-Bench was created in a manner similar to BinkBench.
What are the agents supposed to do?
The agents are set up in an environment where they have access to NihAV and tools to call RAD's Bink 2 decoder for Linux. They're tasked with building an encoder from scratch to satisfy the decoder, focusing on getting as efficient and standard a video output as possible.
Running BinkBench
BinkBench requires the Harbor framework. To run it, try:
git clone https://github.com/MaskNinjaSquared/BinkBench
uv tool install harbor
export API_KEY=...
harbor run -p BinkBench/tasks/binkbench-classic --agent mini-swe-agent --model provider/model
Results
Agent scores will in future be plotted on a graph of VMAF against bits per pixel. Currently, scores are passed as a reward (made up of the geometric mean × efficiency) which also factors in completion of available clips for validation.
Running this at scale against multiple SOTA models requires an inference budget I don't have yet. However, if you're interested in running BinkBench against a model and sharing results, feel free to reach out to me.
Credits
- HikingFex for sample videos used in both agent testing and external verification of the encoders
- Playground Games for their cutscenes
- NihAV for their implementation of the Bink 2 decoder
- FFmpeg and John Van Sickle for the static ffmpeg 7.0.2 build (GPLv3, redistributed unmodified in
tools/), and Netflix VMAF (libvmaf) for the VMAF metric - RAD Game Tools for creating Bink 2!
Notes
Please note that BinkBench is still a proof-of-concept, and I haven't been able to test this with models yet. I've tried to make the scripts as robust as possible, but if you find anything, reporting it would be a huge help.
The sample clips are genuine KB2i .bk2 files stored in the repository (via Git LFS) under tasks/binkbench-classic/environment/samples/. Agents decode them on demand with evaluation.py.
License
BinkBench is licensed under the Apache 2.0 License.
Similar repos
A continuous, long-horizon benchmark for ambitious agents
Rust
1
3 commits
updated Oct 4, 2026
See what people are saying
README
BinkBench - a continuous, long-horizon benchmark for ambitious agents
How good are agents at video engineering?
Intro
This benchmark is based off an idea which originated from my Blogspot article, although it's more centred on codec engineering, rather than reverse-engineering.
BinkBench is a proof-of-concept benchmark for coding agents based on four open metrics - VMAF (as primary), PSNR and SSIM for measuring video quality, as well as bits per pixel (bpp) for measuring efficiency. The former three metrics can also be comprised into one quality metric (via a geometric mean).
BinkBench is also contamination-resistant, since there is no public encoder for the KB2i revision of Bink 2 for agents to find. The agent must derive one from the decoder alone, not retrieve a known solution. This also allows us to have internet access on for any research into compression techniques for encoding, because no matter how hard they try, they won't be able to find a reference encoder. It's worth putting here that the most likely way the agent gets access to an encoder is through someone uploading a previous run or the agent navigating to HuggingFace, which will get sorted out in the future (or compromising Epic Games).
What's different about it?
The majority of popular benchmarks today score based on the amount of tasks completed, instead of how good you can get at tasks specifically - collapsing the ability to see how well an agent did on something specific to a score of zero or one. For example, simplified:
- DeepSWE - does the agent pass or fail this specific test?
- FrontierCode - should this be merged?
While this is a stable standard, it doesn't mean that every other benchmark should work this way. For example, Vending-Bench was created in a manner similar to BinkBench.
What are the agents supposed to do?
The agents are set up in an environment where they have access to NihAV and tools to call RAD's Bink 2 decoder for Linux. They're tasked with building an encoder from scratch to satisfy the decoder, focusing on getting as efficient and standard a video output as possible.
Running BinkBench
BinkBench requires the Harbor framework. To run it, try:
git clone https://github.com/MaskNinjaSquared/BinkBench
uv tool install harbor
export API_KEY=...
harbor run -p BinkBench/tasks/binkbench-classic --agent mini-swe-agent --model provider/model
Results
Agent scores will in future be plotted on a graph of VMAF against bits per pixel. Currently, scores are passed as a reward (made up of the geometric mean × efficiency) which also factors in completion of available clips for validation.
Running this at scale against multiple SOTA models requires an inference budget I don't have yet. However, if you're interested in running BinkBench against a model and sharing results, feel free to reach out to me.
Credits
- HikingFex for sample videos used in both agent testing and external verification of the encoders
- Playground Games for their cutscenes
- NihAV for their implementation of the Bink 2 decoder
- FFmpeg and John Van Sickle for the static ffmpeg 7.0.2 build (GPLv3, redistributed unmodified in
tools/), and Netflix VMAF (libvmaf) for the VMAF metric - RAD Game Tools for creating Bink 2!
Notes
Please note that BinkBench is still a proof-of-concept, and I haven't been able to test this with models yet. I've tried to make the scripts as robust as possible, but if you find anything, reporting it would be a huge help.
The sample clips are genuine KB2i .bk2 files stored in the repository (via Git LFS) under tasks/binkbench-classic/environment/samples/. Agents decode them on demand with evaluation.py.
License
BinkBench is licensed under the Apache 2.0 License.