README
Image & Video Codec Evaluation
ChanganBench is a configurable benchmark toolkit for large-scale image and video codec evaluation. It provides a unified framework for evaluating traditional codecs, learned codecs, generative codecs and tokenizer-based codecs using distributed execution and configurable benchmarking pipelines.
Image Codec Evaluation
Supported Image Codecs
| Perceptual Image Codec | Image Tokenizer | ||
|---|---|---|---|
| Traditional | JPEG, HM, VTM | GAN-based |
FSQ, BSQ, VAR, TA-Tok Infinity, Cosmos, IBQ |
| Learning-based | ELIC, TCM, MLIC++ | ||
| GAN-based | HiFiC, MS-ILLM | Diffusion-based | FlowMo, SSDD |
| Diffusion-based | PerCo, DiffEIC, StableCodec |
Supported Metrics
| Category | Metrics |
|---|---|
| Pixel Fidelity | PSNR, SSIM, MS-SSIM |
| Perceptual Quality | LPIPS, DISTS |
| Distribution Quality | FID |
| Compression Efficiency | BPP |
Supported Datasets
| Dataset | Description |
|---|---|
| Imagenet/ CLIC2020/ Kodak | Built-in benchmark dataset |
| Custom Dataset | Any image directory or dataset implementation |
New codecs can be integrated by implementing a codec class and registering it through the configuration system.
reconstruction, bpp = codec(image)
Evaluation Pipeline
Image Evaluation
Dataset
↓
Codec Inference
↓
Reconstruction + BPP
↓
Metric Evaluation
↓
Distributed Aggregation
↓
Final Benchmark Results
Video Evaluation
Supported Video Codecs
| Traditional | H.264, H.265, H.266 | Neural Codec |
DCVC, DCVC-TCM, DCVC-HEM DCVC-DC, DCVC-FM, DCVC-RT |
Supported Video Metrics
| Category | Metrics |
|---|---|
| Pixel Fidelity | PSNR, SSIM, MS-SSIM |
| Perceptual Quality | LPIPS, DISTS |
| Distribution Quality | FID, FVD |
| 3D/Geometry Sensitivity | VGGT camera center error, camera rotation error, depth AbsRel, point L2 |
| Compression Efficiency | BPP |
Configuration-Driven Benchmarking
Each benchmark experiment is defined by three configurable components:
- Dataset
- Codec
- Metric
All components are configured through YAML files, enabling arbitrary combinations of datasets, codecs, and evaluation criteria without modifying benchmark code.
The framework supports:
- PyTorch Distributed Data Parallel (DDP)
- Extensible codec registration
- Extensible metric registration
- Extensible dataset registration
- Unified image and video benchmarking workflows
Contributors
Similar repos
Languages
Python
92.6%
C++
5.2%
Cuda
1.4%
README
Image & Video Codec Evaluation
ChanganBench is a configurable benchmark toolkit for large-scale image and video codec evaluation. It provides a unified framework for evaluating traditional codecs, learned codecs, generative codecs and tokenizer-based codecs using distributed execution and configurable benchmarking pipelines.
Image Codec Evaluation
Supported Image Codecs
| Perceptual Image Codec | Image Tokenizer | ||
|---|---|---|---|
| Traditional | JPEG, HM, VTM | GAN-based |
FSQ, BSQ, VAR, TA-Tok Infinity, Cosmos, IBQ |
| Learning-based | ELIC, TCM, MLIC++ | ||
| GAN-based | HiFiC, MS-ILLM | Diffusion-based | FlowMo, SSDD |
| Diffusion-based | PerCo, DiffEIC, StableCodec |
Supported Metrics
| Category | Metrics |
|---|---|
| Pixel Fidelity | PSNR, SSIM, MS-SSIM |
| Perceptual Quality | LPIPS, DISTS |
| Distribution Quality | FID |
| Compression Efficiency | BPP |
Supported Datasets
| Dataset | Description |
|---|---|
| Imagenet/ CLIC2020/ Kodak | Built-in benchmark dataset |
| Custom Dataset | Any image directory or dataset implementation |
New codecs can be integrated by implementing a codec class and registering it through the configuration system.
reconstruction, bpp = codec(image)
Evaluation Pipeline
Image Evaluation
Dataset
↓
Codec Inference
↓
Reconstruction + BPP
↓
Metric Evaluation
↓
Distributed Aggregation
↓
Final Benchmark Results
Video Evaluation
Supported Video Codecs
| Traditional | H.264, H.265, H.266 | Neural Codec |
DCVC, DCVC-TCM, DCVC-HEM DCVC-DC, DCVC-FM, DCVC-RT |
Supported Video Metrics
| Category | Metrics |
|---|---|
| Pixel Fidelity | PSNR, SSIM, MS-SSIM |
| Perceptual Quality | LPIPS, DISTS |
| Distribution Quality | FID, FVD |
| 3D/Geometry Sensitivity | VGGT camera center error, camera rotation error, depth AbsRel, point L2 |
| Compression Efficiency | BPP |
Configuration-Driven Benchmarking
Each benchmark experiment is defined by three configurable components:
- Dataset
- Codec
- Metric
All components are configured through YAML files, enabling arbitrary combinations of datasets, codecs, and evaluation criteria without modifying benchmark code.
The framework supports:
- PyTorch Distributed Data Parallel (DDP)
- Extensible codec registration
- Extensible metric registration
- Extensible dataset registration
- Unified image and video benchmarking workflows
Similar repos
Contributors
Languages
Python
92.6%
C++
5.2%
Cuda
1.4%