Funko Scanner

Point your phone at a Funko Pop and it tells you which one it is. Everything runs on the phone, so it also works in airplane mode.

Two small models do the work. A detector (RF-DETR Nano, fine-tuned for one class) finds the figure in the camera frame. An embedding model (CLIP ViT-B/32, fine-tuned) turns the crop into a vector, and the app looks up the nearest vector in a gallery of 818 figures. A new figure needs one photo and one more vector, with no retraining.

The app is built with React Native ExecuTorch 0.10 and VisionCamera 5. Both models were trained and exported mostly by a coding agent; ml/ is that pipeline.

Run the app

You need a physical phone (the scanner uses the camera), Node 20+, and Xcode or the Android SDK.

./scripts/download-models.sh      # ~520 MB from the GitHub release into app/assets/funko/
cd app
npm install
npx expo run:ios --device         # or: npx expo run:android --device

The app is an Expo dev client; ios/ and android/ are generated on the first run.

Where to look

Train and export the models

Everything below runs on a Mac with Apple silicon (training uses MPS). The detector trains in about 35 minutes.

cd ml
uv sync

1. Data

None of the images are redistributed here; the scripts download them.

# Catalog: one studio photo per figure, from a Kaggle scrape of the Funko shop (needs ~/.kaggle/kaggle.json)
uv run kaggle datasets download victorsoeiro/funko-pop-dataset -p data/raw/kaggle --unzip
uv run python -m funko.download_catalog     # resumable
uv run python -m funko.build_catalog        # manifest + background-removed cutouts
uv run python -m funko.splits               # 80/10/10 by figure

# Detector boxes: Roboflow "funko-v2", version 9, COCO format (needs ROBOFLOW_API_KEY in ml/.env)
uv run python -c "import os, dotenv; dotenv.load_dotenv(); from roboflow import Roboflow; \
  Roboflow(api_key=os.environ['ROBOFLOW_API_KEY']).workspace('cd-1qlyv').project('funko-v2').version(9) \
  .download('coco', location='data/raw/roboflow_funko_v2_coco')"

# Real backgrounds for the synthetic detector scenes: COCO val2017
mkdir -p data/raw/coco && curl -L http://images.cocodataset.org/zips/val2017.zip -o /tmp/val2017.zip \
  && unzip -q /tmp/val2017.zip -d data/raw/coco

uv run python -m funko.prep_detection       # de-leaked Roboflow boxes + synthetic scenes + negatives

2. Train

uv run python -m funko.train_embed --backbone clip_vit_b32 --epochs 12   # weights/clip_vit_b32_ft.pt
uv run python -m funko.train_rfdetr                                      # runs/rfdetr_nano/

augment.py is the "make it look like a phone photo" step: cut-out figures on real backgrounds with blur, noise and bad light. evaluate.py holds the honest test: queries generated by different code than the training images.

3. Export

uv run python -m funko.export_all                    # embedder: XNNPACK fp32, Core ML fp16, int8; parity vs PyTorch
uv run python -m funko.export_rfdetr --checkpoint runs/rfdetr_nano/checkpoint_best_ema.pth   # detector: XNNPACK, Core ML
uv run python -m funko.export_vulkan detector        # Android GPU, 320 px input
uv run python -m funko.export_vulkan embedder
uv run python -m funko.build_gallery --embedder xnnpack_fp32 --detector rfdetr_coreml_fp32
uv run python -m funko.ship                          # copies everything into app/assets/funko/

Every export is checked the same way: the same photo goes through PyTorch and through the exported file, and the answers have to match. export_vulkan.py --check runs the rewritten Vulkan graph on the Mac's CPU against PyTorch, because the Vulkan files only run on the phone. Its docstring lists what it takes to get RF-DETR and CLIP onto the GPU with the Vulkan library that React Native ExecuTorch ships.

The released gallery differs from a fresh build_gallery run in two ways: it adds three figures from photos (Ekko and Champion Viktor from Arcane, Josh Rosen), and its unknown-figure threshold is 0.65 instead of the calibrated 0.71.

Testing on real frames

pull_debug.py copies the debug captures off an iPhone (--device <udid>), and device_frames_eval.py scores detector checkpoints on them. A handful of real photos, never used for training, catch problems no synthetic test set will.

Credits

• Detection data: funko-v2 by Roboflow user cd-1qlyv, CC BY 4.0.
• Catalog: Funko Pop dataset by Victor Soeiro (product metadata and image links).
• Backgrounds: COCO val2017, CC BY 4.0.
• Models: RF-DETR (Apache 2.0), CLIP (MIT).

Funko and Pop! are trademarks of Funko, LLC. This project is not affiliated with Funko.

License

MIT, see LICENSE. The model weights inherit the licenses of the models and data above.