Composition Helper

Composition Helper evaluates photographic composition using the Composition Assessment DataBase (CADB). The project reproduces SAMP-Net and compares it against three DINOv3 architectures that predict the same five-bin quality-score distribution:

• frozen DINOv3 with spatial-pyramid pooling;
• frozen DINOv3 with learned attention pooling;
• partially unfrozen DINOv3 with spatial-pyramid pooling.

The quality target is the distribution of five professional ratings from 1 to 5. The repository also supports CADB's separate 14-label composition classification target, although the reported experiments focus on quality assessment.

This work uses the dataset released with the CADB paper.

Final results

Only metrics from the locked 950-image official_test split are reported here. Each DINOv3 value is the mean ± sample standard deviation over seeds 2026, 2027, and 2028 after retraining from scratch on all 8,547 official_train images.

The partially unfrozen architecture improved MSE, SRCC, and LCC over SAMP-Net. Its EMD was statistically indistinguishable from SAMP-Net for every seed. The supported conclusion is therefore better ranking and absolute error with EMD parity—not across-the-board superiority.

See the experiment-specific reports for SAMP-Net, frozen spatial-pyramid DINOv3, frozen attention-pooling DINOv3, and partially unfrozen DINOv3 for final-test provenance. Training-loop and development-validation metrics are not reported as final results.

Architectures

The released samp_net/ checkpoint is the controlled baseline, not a fourth experimental architecture.

Setup

1. Install the command-line tools

The project uses uv for Python dependencies and Just for repeatable commands. On macOS:

brew install uv just

2. Initialize the repository

just setup

This initializes the CADB source submodule and installs the locked Python environment from uv.lock.

3. Install the CADB data

Download CADB using the instructions in the upstream repository and place the release at:

data/cadb/
├── images/
├── split.json
└── ...annotation files

The checked-in manifests under manifests/cadb/v1 define the exact development and official partitions. Do not create ad hoc splits.

4. Install model weights

• Download the released SAMP-Net checkpoint from the authors' official Dropbox release and place it at samp_net/checkpoints/samp_net.pth. The link is maintained in the official CADB/SAMP-Net repository.
• Request access to facebook/dinov3-vits16plus-pretrain-lvd1689m, accept its license, and cache the model through Hugging Face.

The experiment recipes run in offline mode, so missing weights or dependencies fail immediately rather than downloading during a run.

5. Verify the installation

just test
just smoke-frozen
just smoke-attention
just smoke-unfrozen

Every smoke test loads a real CADB batch and performs an optimization step. The unfrozen smoke test additionally verifies that gradients reach the selected backbone blocks but not the frozen prefix. Pass a device explicitly when needed, for example just smoke-unfrozen mps or just smoke-frozen cuda.

Running experiments

Run just --list at any time to see every command and its arguments.

Development training

Development training uses train.json, selects the best checkpoint using only validation.json, and never reads official_test.json.

just train-frozen
just train-attention
just train-unfrozen

All three commands share the same positional interface:

just <recipe> [device] [batch_size] [workers] [output_dir]

For example:

just train-frozen cuda 32 8 outputs/dinov3-frozen
just train-attention mps 16 0 outputs/dinov3-attn
just train-unfrozen cpu 8 4 outputs/dinov3-unfrozen

Each run writes:

OUTPUT_DIR/
├── seed-2026/
│   ├── best.pt
│   ├── metadata.json
│   └── validation.jsonl
├── seed-2027/
├── seed-2028/
└── validation-summary.json

The trainers are unguarded and may be rerun into an existing directory. Frozen spatial-pyramid runs can reuse completed seed checkpoints with just resume-frozen.

Re-aggregate development validation

Recompute all four metrics and mean ± sample SD directly from saved predictions:

just aggregate-frozen-validation
just aggregate-attention-validation
just aggregate-unfrozen-validation

For a custom directory:

just aggregate-validation outputs/my-experiment

Aggregation validates the prediction schema, exact manifest membership and order, file hashes, and seed layout. It does not load a model or run inference. Development aggregation is explicitly marked as non-reportable.

Final retraining and evaluation

After freezing a configuration, every architecture follows the same two-step workflow. Training uses all of official_train and writes checkpoints without reading official_test. Evaluation loads those checkpoints and writes ordered test predictions plus final-report.json.

just train-frozen-final cuda 8 outputs/dinov3-frozen-official-train
just evaluate-frozen-final cuda 8 outputs/dinov3-frozen-official-train outputs/dinov3-frozen-official-test

just train-attention-final mps 0 outputs/dinov3-attn-official-train
just evaluate-attention-final mps 0 outputs/dinov3-attn-official-train outputs/dinov3-attn-official-test

just train-unfrozen-final cpu 4 outputs/dinov3-unfrozen-official-train
just evaluate-unfrozen-final cpu 4 outputs/dinov3-unfrozen-official-train outputs/dinov3-unfrozen-official-test

Training arguments are [device] [workers] [output_dir]; evaluation arguments are [device] [workers] [checkpoint_root] [output_dir]. Both commands support CPU, MPS, and CUDA. Final-test results must never guide model selection.

Re-aggregate final test predictions

These commands only recompute metrics from predictions already on disk; they do not access images or repeat locked-test inference:

just aggregate-frozen-final
just aggregate-attention-final
just aggregate-unfrozen-final

For a custom output directory:

just aggregate-final outputs/my-final-run

Reproduce the SAMP-Net baseline

just evaluate-samp-net cpu 16 0

The local checkpoint reproduces the paper closely:

Evaluation protocol

Quality models predict five probabilities for scores 1–5. The predicted mean is the expectation of that distribution. Every reported run saves ordered per-image predictions, and the shared evaluator computes MSE, normalized EMD (r=2), SRCC, and LCC from the same prediction file.

The official test set must never be used for training, early stopping, checkpoint selection, threshold selection, debugging, or design decisions.

Acknowledgments and citations

This project builds on the following research and software:

• SAMP-Net and CADB: Bo Zhang, Li Niu, and Liqing Zhang, “Image Composition Assessment with Saliency-augmented Multi-pattern Pooling,” BMVC 2021 (paper, code and dataset). Portions of the SAMP-Net compatibility implementation are adapted from the authors' MIT-licensed code.
• AADB: Shu Kong, Xiaohui Shen, Zhe Lin, Radomír Měch, and Charless Fowlkes, “Photo Aesthetics Ranking Network with Attributes and Content Adaptation,” ECCV 2016 (project). CADB was constructed from AADB images and attributes.
• DINOv3: Oriane Siméoni et al., “DINOv3,” 2025 (paper, code, model card). DINOv3 was developed by Meta AI and is governed by the DINOv3 License.

CADB images, the released SAMP-Net checkpoint, and DINOv3 weights are not included in this repository. Users must obtain them from their official sources and comply with their respective terms. See THIRD_PARTY_NOTICES.md for license boundaries and source-level attribution.

Project layout

• shared/: CADB loading, manifests, prediction schema, metrics, and result aggregation.
• samp_net/: reproduced SAMP-Net model, preprocessing, and inference.
• dinov3/: frozen spatial-pyramid architecture.
• dinov3_attn/: frozen attention-pooling architecture.
• dinov3_unfrozen/: partially unfrozen spatial-pyramid architecture.
• tests/: unit, parity, ordering, caching, and checkpoint tests.
• manifests/: authoritative versioned CADB partitions.

Additional commands

just train-frozen-cached   # Faster frozen-feature development condition
just train-frozen-tuned    # No-dropout frozen candidate
just mlflow                # Start MLflow at http://127.0.0.1:50000

Cached-token training is valid only while the backbone is frozen. Cached and uncached runs are numerically distinct and must not be mixed in one comparison.

License

Original work in this repository is available under the MIT License. Third-party software, datasets, papers, model weights, and derivative model materials remain subject to their respective terms; see THIRD_PARTY_NOTICES.md.

If you use Composition Helper in research, please cite it using CITATION.cff and cite the underlying research listed in the acknowledgments.