BoldSearch

A local video-frame retrieval workspace for HCM AI Challenge tasks, helping an operator find, inspect, and record KIS, VQA, TRAKE, and image-reference answers.

BoldSearch pairs a React operator interface with a FastAPI retrieval API. It creates FG-CLIP text or image embeddings, searches a Zilliz/Milvus collection, enriches results with object-detection metadata, and serves the local keyframe corpus for visual verification.

The in-app Submit action validates and accepts a payload locally. Official BTC delivery is a separate CSV-and-ZIP workflow; see the submission guide.

---

Capabilities

Competition knowledge

• General submission guide

---

Quick start

Prerequisites

1. Prepare local runtime data and configuration

Keep the media corpus outside the Vite application bundle. Directories fall into four categories — see the tracked data/README.md contract for the full table:

data/                 # machine-local; populated from team storage or BTC packages
├── keyframes/        # served corpus: keyframe images
├── map-keyframes/    # served corpus: per-video frame maps
├── aic2026-downloads/    # raw BTC packages
├── aic2026-<round>/      # per-round working dirs (ZIP, results, picks)
└── frames|metadata|vectors/  # per-video derived artifacts

app/backend/
└── detections.csv    # tracked frame object-detection metadata

Create an ignored app/backend/.env file and provide the connection settings for your environment. At minimum, configure ZILLIZ_URI, ZILLIZ_TOKEN, and MILVUS_COLLECTION; use OBJECTS_CSV_PATH, DATA_DIR, or FG_CLIP_DEVICE only when their defaults do not fit your machine or file layout.

Do not commit credentials, local data, model artifacts, or generated evaluation output.

2. Start the API

cd app/backend
uv sync --locked --group dev
uv run python main.py

The API starts on http://localhost:8000 by default. Confirm the process is responding before starting the UI:

curl http://localhost:8000/api/health

3. Start the operator UI

cd app/frontend
npm ci
npm run dev

Open http://localhost:5173. During development, Vite proxies /api, /keyframes, and /map-keyframes to http://localhost:8000 unless VITE_API_URL overrides the backend origin.

---

How it works

1. The operator composes text queries, optional object hints, or an image cue in the React UI and chooses a task mode.
2. The UI calls POST /api/search/query for text/object retrieval or POST /api/search/visual_query for image-reference retrieval.
3. FastAPI validates the request. FG-CLIP encodes the query, then the search workflow executes hybrid retrieval against Zilliz/Milvus.
4. Multiple text queries narrow subsequent stages to the current frame context. Retrieved rows are enriched from the in-memory detections.csv index.
5. The API returns normalized frame results. FastAPI serves keyframes and frame-map CSVs from data/ so the UI can display and inspect the evidence.
6. The UI sends selected KIS, VQA, or TRAKE answers to the local submission endpoints. Package official results separately as described in the submission guide.

API surface

The backend also serves /keyframes and /map-keyframes from the configured local data directories.

---

Project structure

.
├── app/
│   ├── backend/              # FastAPI API, FG-CLIP encoder, Milvus access, and evaluation runner
│   └── frontend/             # Vite React operator interface
├── docs/                     # architecture, diagrams, operational guides, and competition knowledge
├── okf/                      # OKF knowledge bundle with provenance (start at okf/index.md)
├── data/                     # machine-local data; tracked layout contract in data/README.md
├── src/                      # exploratory notebooks and analysis spikes
├── .github/                  # Copilot instructions and CI workflows
├── AGENTS.md                 # coding-agent guidance
├── CLAUDE.md                 # Claude Code entry (imports AGENTS.md)
├── GIT_CONVENTION.md         # branch and commit rules
├── PROGRESS.md               # implementation history and team handoff
├── BLOCKERS.md               # current blockers
└── README.md                 # project entry point

app/backend/main.py is the composition root. At startup it loads the detection index, opens the Milvus client, and optionally loads FG-CLIP once into application state. The search module owns the public HTTP contract, request orchestration, result shaping, and local submission validation.

For the current runtime boundaries and planned evolution, read Architecture. For backend data contracts and offline evaluation, see the backend README and evaluation README.

Configuration

Backend settings read environment variables and app/backend/.env. Relative backend paths resolve from app/backend.

ZILLIZ_TOKEN and HF_TOKEN are secrets. Keep them server-side in the backend environment; values prefixed with VITE_ are browser-visible and must not contain secrets.

Common settings are listed below; the full field list lives in app/backend/.env.example.

---

Development commands

Verification

Run the narrowest check that proves your change. Before sharing a cross-stack change, run:

(cd app/backend && uv run pytest)
(cd app/frontend && npm test && npm run build)

For retrieval-quality changes, export rankings and use the offline evaluation runner rather than treating a successful API start as relevance evidence. The runner contract and command are documented in app/backend/evaluation/README.md.

Operational notes

• A Milvus collection must match the query encoder's model, checkpoint, preprocessing version, and embedding dimension.
• Translation is best-effort: a translation failure preserves the original text query rather than making search unavailable.
• Preserve both snake_case and compatibility camelCase frame fields in API responses; the frontend relies on both forms.
• The official corpus remains under ignored data/; do not copy it into Vite public or dist.
• Follow GIT_CONVENTION.md for branches and commits. See PROGRESS.md and BLOCKERS.md for the current team handoff state.