Atopic Dermatitis Allergen Predictor

Predicts later food allergy in infants from parent-captured eczema photographs plus questionnaire data, in the CANDO birth cohort.

The design is a concept bottleneck, not an end-to-end CNN. A frozen pretrained encoder scores each photo on the four EASI signs (erythema, papulation, excoriation, lichenification); those scores are aggregated per infant by body site and fed, with tabular predictors, into a penalized logistic regression. The reason is sample size: at ~2000 infants and ~10% outcome prevalence the Riley et al. criteria support roughly 11-17 predictor parameters, so the model capacity has to sit in a frozen encoder rather than in fitted weights. See docs/ for the full architecture and the data request that unblocks it.

Project Structure

• data/: Raw, interim, and processed datasets. (Not tracked in Git)
• src/: Source code for the project.
  • src/data/: REDCap export parsing, image normalization, QC. Implemented.
  • src/models/: Aggregation and the outcome model. Not yet built.
  • src/cli/: Command-line interface.
• docs/: Architecture notes and the data request spec.
• outputs/: Model artifacts, logs, and figures. (Not tracked in Git)
• config.yaml: Configuration settings for the pipeline.

Handing this project to someone else

docs/HANDOFF.md is the complete context document: goal, verified literature, data state, architecture and its justification, every module built, measured results, the remaining phases, and the traps already hit. Start there.

Getting Started

1. Activate the pre-commit safety hook (do this first, every clone): git config core.hooksPath .githooks This repo is public and this project handles identifiable patient photos. .gitignore keeps data//outputs/ out of accidental commits, but the hook is the real backstop -- it hard-blocks any commit containing a path under data//outputs/, an image/media file anywhere in the repo, or an unusually large file, even if force-added. See .githooks/pre-commit.
2. Create a virtual environment (this project uses uv): uv venv
3. Install dependencies: uv sync (dependencies live in pyproject.toml; uv.lock pins the exact graph). The frozen-encoder model stack is a separate extra, so the parsing pipeline and test suite install without a multi-GB torch download -- add it later with uv sync --extra models.
4. Point data.source in config.yaml at your REDCap export folder (root directory plus the export's base filename, shared by its .csv/.r/report-folder trio).
5. Run the parsing pipeline: python -m src.cli.main --step parse
6. Embed the canonical images: python -m src.cli.main --step embed (needs uv sync --extra models; downloads the encoder named in config.yaml on first run)
7. Optionally check the encoder is working: python -m src.cli.main --step validate-encoder
8. Score the EASI sign concepts: python -m src.cli.main --step concepts
9. Test those scores against lighting change: python -m src.cli.main --step robustness

This reads the REDCap CSV export and its .r data dictionary and file-upload manifest (never modifying the source), and writes a cleaned, model-agnostic dataset under data/interim/ and data/processed/ (both git-ignored):

• data/processed/canonical/: EXIF-oriented, colour-normalized (sRGB) images
• data/processed/images.parquet: one row per photo -- body site, laterality, age-at-photo, visit index, QC metrics
• data/processed/clinical.parquet: decoded, tidied clinical data
• data/processed/dictionary.json, site_ontology.json: machine-readable data dictionary and body-site adjacency graph
• data/processed/qc_report.html: flagged images/records plus a contact sheet
• data/interim/exif/: the complete original EXIF for every photo, verbatim

The embed step adds data/processed/embeddings.parquet: one row per photo holding a frozen-encoder vector (MONET by default, 768-d), keyed by image_id and stamped with the encoder name and the canonical file's content hash. Re-runs only re-embed images whose pixels actually changed. These vectors are the input to the concept layer.

--step validate-encoder is an opt-in sanity check that the embeddings carry real signal rather than merely the right shape: it compares within- vs between-group cosine similarity for patients and body regions, and tests whether text prompts can recover the body-site labels the parser already derived. Its output is aggregate-only by construction -- no image_id, record_id, or filename ever reaches the report, which tests/features/test_validate.py enforces. The JSON lands under outputs/, which is git-ignored and blocked by the pre-commit hook.

--step concepts writes data/processed/concepts.parquet: one row per photo with a raw and a within-cohort-percentile score for each of the four EASI signs (erythema, papulation, excoriation, lichenification) plus scale and crust. These are not EASI grades. Photographs cannot supply EASI's area score, and a CLIP similarity difference has no natural zero, so mapping these onto 0-3 requires clinician annotation. Until then they are an ordering, not a measurement.

--step robustness is the gate that decides whether those scores are usable at all. It re-embeds photometrically perturbed copies of the real images and reports the Spearman correlation between each perturbed ordering and the unperturbed one. Rank stability is the metric because the ordering is the entire predictive content passed downstream. The report also compares each concept's stable band against the cohort's observed brightness spread, so a failure at sweep extremes can be judged against the lighting that actually occurs.

Re-running the parse step is incremental: unchanged photos are skipped, and any REDCap schema drift (added/removed CSV columns) between runs is reported, not silently absorbed. Run pytest to run the test suite (all tests use synthetic fixtures -- no patient data required).