AIC26 Retrieval

An operator-first video retrieval system built for the AI Challenge 2026 (AIC) — interactive Known-Item Search (KIS), question answering (QA), and temporal action search (TRAKE) over a large broadcast-video corpus.

The repository contains two parts:

• A multi-channel retrieval app (backend/ + frontend/) — the live console an operator uses during the competition to search, inspect evidence, and submit answers to DRES.
• An offline indexing pipeline (root scripts + notebooks) — the tooling that turns raw videos/keyframes into the OCR / speech / audio / image-vector indices the app reads from.

Design principle: UI > Method > Model — minimise wrong submits and navigation time for a human operator under a clock.

---

Highlights

• Multi-channel retrieval fused with Reciprocal Rank Fusion (RRF, k=60):

  • image_pe — PE-Core-G14 text→image vectors in Milvus (cosine)
  • image_qwen — Qwen3-VL-Embedding-8B native 4096-d vectors for InfoShot++; the operator may search PE, Qwen, or both in parallel. A two-model search combines ranks with RRF and never adds their incomparable cosine scores.
  • ocr / speech / audio — full-text + filters in Elastic
  • audio vector — GLAP audio↔text embeddings in Milvus, fused with the Elastic audio signal

• Query understanding via DeepSeek V4.1 Flash (deepseek-flash, thinking off, JSON output, ~2 s) with a prompt written for the final-round collection and task types, validated field by field, and a deterministic heuristic fallback; VI→EN translation of the visual query. The same model writes the Expand paraphrases.

• Qwen3-VL visual reranker (opt-in, off by default) — a Rerank tick box in the query panel widens EVERY selected image index to QWEN_RERANKER_CANDIDATES, merges their candidates into one pool (round-robin over the two rank lists, so a frame only Qwen retrieved is judged too), and has Qwen3-VL-Reranker-8B rescore each (query, keyframe) pair at frame level. The result is a single image_visual ranking that meets OCR/speech/audio in the RRF, so a low visual score cannot overrule the channels whose evidence is not visual at all. Fail-open: if the worker is unreachable the union order (RRF over the indices, never a sum of their cosines) is kept and the panel says so. Serve it with aic26_qwen3vl_reranker8b_colab_server.ipynb; the box is hidden when no worker answers.

• Search scope — a checkbox filter over the dataset folders (BTC: L21–L30 + K01–K20; InfoShot++: L21–L30 + batch 2 M01–M10 news, N001–N100 traffic cameras, S01 cycling), pushed down into the Milvus/Elastic queries. Each folder is one programme, so a topic heuristic reads the folders off the query the same way the parser reads channels ("đầu bếp" → L26), while never excluding the programmes that carry every subject (the 60-second bulletins and L30).

• PE text-window meter — PE-Core reads 70 text tokens and silently drops the rest; about a third of the organisers' queries are longer than that even in English. The search box counts the query (plus hints) as the operator types — exactly with the PE tokenizer, or estimated when it will be translated — and says which part PE will ignore; each search reports the exact count of what it sent.

• Traffic-camera / race filter (InfoShot++) — every N camera prints its junction, date and clock in a banner that batch-2 OCR reads. Street names in the query ("Nguyễn Trãi – Cống Quỳnh", NTMK, CMT8), a date ("15/6") or a time ("19:05", "7 giờ tối") narrow the traffic cameras to the matching keyframes, and "chặng 6" narrows S01 to that stage's video. It is pushed into Milvus/Elastic like the scope, never excludes another folder, and drops a cue no recording matches with a warning; the console shows what it applied and can switch it off. Camera frames carry their banner reading, and OCR ranks the news ticker, banner and HUD below scene text.

• TRAKE is ranked by video, not by keyframe: after pass 1 the per-event results become one video_id -> per-event candidates map that drives pass-2 targeting, the DP, the heatmap and the ranking. A temporal NMS keeps distinct moments (not twelve frames of one burst) before an exact dynamic-programming maximum-weight increasing chain assembles the sequence — globally optimal, not greedy. trake_video_score folds (confident coverage, coverage, chain quality) into one coverage-dominant number, and each result card shows one representative per event over a heat row of every moment that event fires; clicking one seeks the player there and arms that event's slot. Before retrieval, shared_context_vi / shared_context_en_visual retain the statement's common subject, clothing and scene. Every event variant combines that context with its action; PE/Qwen, TARA and pass 2 consume the same composed queries. The context is never searched as a separate MAX-fused alternative. The heuristic keeps the literal preamble; the LLM is instructed to resolve shared references across events. This preserves query conditions in the text, rather than imposing a hard visual filter on embedding results.

• QA copilot — top-video-first candidate blocks read by DeepSeek V4.1 Flash itself (image input, thinking on): answer-bearing frames and exact-format answer alternatives (as printed, bare names, the question's own format rules), then DeepSeek deepseek-v4-pro web search for names and facts the frames cannot show, then a Flash visual-consistency pass before explicit human verification. ~15–20 s for all three passes; the NVILA-8B Colab worker remains selectable (QA_VISION_BACKEND=nvila).

• 100-answer generator — the preliminary round scores Final = (R@1 + R@5 + R@20 + R@50 + R@100) / 5 over up to 100 answers per query, so the answer list is a rank-budget problem, not a top-100 dump. Each position goes to whichever candidate adds the most new chance of a hit (U = π_B(video) × evidence × novelty), with temporal NMS turning the fused frames into distinct hypotheses and a data-calibrated ±ε ladder covering the case where the retrieved keyframe sits just outside the accepted window. One button in the Submission tab fills the whole question pack.

• CAD-VR adaptive agents (AGENT button, on by default) — retrieval feeds a query-local Evidence Board. Jev 1.13 on OpenRouter verifies constraints and chooses when to invoke Codex Scout (gpt-6.1-sol, high, fast), Claude Investigator (claude-opus-5-5, high, fast), or gather more evidence. Both retain all search capabilities. The default policy is full; fixed baselines and ablations are available for benchmarking. Agents search through a narrow MCP bridge (backend/app/agent/):

  • search: the same retrieval engine, in hybrid, visual, OCR or speech mode.

  • list_videos: the programme guide of a folder or series. Traffic cameras show junction, date and clock range; S01 shows the race stage; every other folder shows sample lines of what is said.

  • folder_frames: one frame per video in one grid (N gives one tile per camera).

  • video_outline: a video's table of contents, one line per minute of what is said and shown.

    These three browsing tools let an agent find a video without the search engine.

  • video_frames: labelled contact sheets, including center_time + before/after + density.

  • view_frames: a large view of specific keyframes.

  • video_text: the speech and OCR text in a time window.

  • compare_candidates: aligned before/center/after sheets for 2–4 moments.

  • constraint_probe: local frames/text for one missing constraint.

  • report_candidate: shows a candidate to the operator.

  Candidates appear in their own Agents strip at the top of the results column as soon as they are reported, and the strip flags a moment both agents found on their own. Their frames are submitted like any other frame:

  • KIS / QA / AVS: each frame card has Submit, which opens the usual guard with DRES preview and dedup, plus + Sticky and Play. QA can also take the agent's answer.
  • TRAKE: frames are grouped into one E1..En row per agent and video, with Submit sequence, Load into E1..En and + Sticky.
  • Paused frames: the strip's player captures an exact paused frame to submit, or to assign to an event slot for TRAKE.

  The operator's folder filter (manual, or what auto resolved from the query) goes into the prompt as a strong hint: the agents look there first. A frame from outside it is labelled "Outside the folder filter". The main /api/search never waits for the agents and its ranking is never changed by them. The run publishes its combined ranking and orders agent cards by it. A new search cancels the previous run. AGENT_TIMEOUT_SECONDS now bounds the entire controller (retrieval, decisions, queued agents and tool work). The CLIs have no shell and receive none of the backend's credentials. Each run is a fresh process, so an account switched in the meantime is picked up by the next search.

• Submit guard — duplicate detection, evidence preview, and the exact DRES v2 answerSets body previewed before it is sent (media item + ms window for KIS/TRAKE, text for QA); the open task name and evaluation run come live from DRES.

• Keyboard-first single-screen UI with a multi-track timeline (keyframes, speech, OCR, audio, heatmap) and BTC keyframe snapping for frame-accurate picks.

• Mock mode — deterministic fixtures so the UI and tests run with zero live services.

---

Architecture

OFFLINE INDEXING (scripts + notebooks at repo root)
  Videos / BTC keyframes
        │  Whisper ASR · OCR · audio tagging · PE-G14 / GLAP embeddings
        ▼
  ┌───────────────┬───────────────┬───────────────┐
  │ Elastic Cloud │    Milvus     │ Cloudflare R2 │
  │ ocr / speech  │ image vectors │ keyframes +   │
  │ audio / kfmap │ audio vectors │ videos (media)│
  └───────┬───────┴───────┬───────┴───────┬───────┘
          │               │               │
ONLINE QUERY              │               │
          ▼               ▼               ▼
  backend/  FastAPI ── parse → retrieve (parallel channels) → RRF fuse
            → group-by-video / TRAKE DP → DeepSeek visual QA → DeepSeek web grounding
            → DeepSeek visual verification → human choice → submit guard → DRES
          │
          ▼
  frontend/ Vite + React + TS operator console (keyboard-first, timeline)

The detailed, load-bearing contract (identity rules, API shape, scoring, fusion, TRAKE) lives in RETRIEVAL_APP_HANDOFF.md. The broader design/strategy is in AIC26_Pipeline.md.

---

Repository layout

Not tracked in git (see .gitignore): secrets (*.txt keys, .env), large data/model artifacts (peG14.pkl, *.jsonl, speech_out/, elastic_staging/, results_audio_event/, keyframe maps), virtualenvs, and build caches. These are regenerated by the pipeline scripts/notebooks above.

---

Quickstart

0. Just run it (Docker — no Python, no Node, no build)

docker compose up -d          # http://localhost:8000

One container serves the console and the API on one port. It ships with no credentials: open it, and the settings screen (⚙) asks you to import the team's .env. The imported file lands in ./config/.env and the submit history in ./data/, both outside the image, so a rebuild never loses them.

To hand the app to someone who has neither this repo nor a toolchain:

./scripts/package.sh          # → dist-app.zip (~470 MB, image included)

They unzip it and run ./run.sh (Linux/macOS) or double-click run.bat (Windows, Docker Desktop) — it loads the image, starts the container and opens the browser. Setup guide for them: scripts/dist/HUONG_DAN.md.

The compose file binds to 127.0.0.1 on purpose: the running app can rewrite its own credentials and submit under your DRES account, so it is not exposed to the LAN unless you change that line deliberately.

Prerequisites (development from source)

• Python 3.10+ and Node.js 18+
• (Optional for live mode) Elastic Cloud, Milvus/Zilliz, PE-Core-G14 and/or Qwen3-VL-Embedding-8B encoder endpoints, and a Cloudflare R2 media base URL.

1. Backend

cd backend
python -m venv .venv && source .venv/bin/activate   # or reuse an existing venv
pip install -r requirements.txt

# Dev / UI work without any live services (deterministic fixtures):
AIC26_MOCK_MODE=true PYTHONPATH=. python -m uvicorn app.main:app --reload --port 8000

# Live mode (reads ELASTIC_* / MILVUS_* / PE_ENCODER_URL / MEDIA_BASE_URL from .env):
PYTHONPATH=. python -m uvicorn app.main:app --port 8000

2. Frontend

cd frontend
npm install
npm run dev          # http://localhost:5173, proxies /api -> http://localhost:8000
# point elsewhere:   VITE_API_TARGET=http://host:8000 npm run dev

Open http://localhost:5173. In mock mode the whole console is usable immediately.

---

Configuration & secrets

Backend config is read from environment variables. Copy the template and fill in real values:

cp backend/.env.example backend/.env

In the packaged app there is no shell to export variables in, so the same file is imported through the UI instead: ⚙ → drag in a .env → Import. The backend writes it to AIC26_CONFIG_DIR/.env (mode 600) and rebuilds every service in place — no restart. Variables set on the container itself still win over the file, and the settings screen tags those env so it is clear an import will not change them.

Key variables (full list in backend/.env.example):

TARA artifact verification, Milvus upload, Colab worker setup, and fusion details are documented in VideoRetrieval/TARA_INTEGRATION.md.

⚠️ Secrets are never committed and never reach the frontend. The browser talks only to the backend. Do not paste real keys into tracked files.

---

Tests

cd backend  && PYTHONPATH=. python -m pytest -q   # backend unit/integration tests
cd frontend && npm run test                       # vitest component tests

---

Documentation map

---

Tech stack

Backend: Python · FastAPI · Uvicorn · Elasticsearch · Milvus/pymilvus · DeepSeek (query LLM, web grounding) · Whisper (voice input). Frontend: TypeScript · React 18 · Vite · Vitest. Infra: Elastic Cloud · Zilliz/Milvus · Cloudflare R2 · Kaggle GPU encoders.

---

Status

Implemented: multi-channel retrieval + RRF fusion, group-by-video, TRAKE DP, query parser (LLM + heuristic), DeepSeek-assisted QA, the 100-answer generator and its bulk button in the Submission tab, submit guard, mock mode, audio vector search. Planned (clean seams left in place — see RETRIEVAL_APP_HANDOFF.md §8): general-purpose VLM reranking, object–scene–action tag filters, and a custom Vietnamese analyzer.