RFP assistant (입찰메이트)

Internal assistant for historical Korean RFPs: search projects, select one document, ask a scoped question, receive one metered grounded answer, and open the original evidence. The plan lives in docs/plan/end-to-end; this README covers setup and launch for what is implemented (phases 1–4). Operations (access, stop/restart, billing recovery, reconciliation, evaluation, sealed run, backup/restore, index rollback) are in the runbook; screen layout and request ownership in DESIGN.md. Phase outcomes: phase 2, phase 3, phase 4 and the release report.

Where things live

Environment

The tested environment is the conda env rfp-assistant (Python 3.12, Windows). requirements.txt holds the curated direct pins; requirements-lock.txt records every installed version from that environment.

conda create -n rfp-assistant python=3.12
conda activate rfp-assistant
python -m pip install -r requirements.txt
python -m pip install -e . --no-deps

The HWP converter (hwp5proc from pyhwp 0.1b15) is installed into the same environment and resolved automatically from Scripts\hwp5proc.exe. To use an isolated converter environment instead, set RFP_HWP_CONVERTER to the absolute path of its hwp5proc.exe. The converter prints a harmless warning when xmllint is missing.

Database: PostgreSQL only

The application runs only on PostgreSQL 18.6 with pgvector 0.8.6. Every record lives there: documents, extractions, chunks, requests, traces, the billing ledger, audit events and the embedding vectors: the serving set and each compared model's own set, every one tagged with its model, dimensions and prefix policy. There is no other database and no fallback. Start Docker Desktop (Linux containers), then run ./tools/infra/start-postgresql.ps1. It starts the pinned server on loopback port 55432 and leaves RFP_DATABASE_DSN unchanged; it never selects the empty bidmate_rehearsal database. The application database is bidmate_app. Point RFP_DATABASE_DSN at it in the process environment, never in a file the repository tracks.

Startup refuses to run with a clear message when RFP_DATABASE_DSN is missing, or when the database has no validated import (the migration_import / migration_validation marker plus unchanged artifact hashes, index payload files included). A failed validation or an open recovery fence also closes paid admission. Paid dispatch needs both the ledger switch and PostgreSQL paid admission, which paid on sets together, plus the database-wide gateway advisory lock that the serving process holds.

The owner selected dragonkue/BGE-m3-ko at its native 1,024 dimensions as the default embedding baseline on 2026-10-07, after answer comparison E-9b785d7cd167. Serving uses its measured hybrid row H-5b2565f606; further retrieval tuning belongs to the retriever owner. GPT-5-mini remains the answer model. BGE-m3-ko tied KURE at 80% answer pass, with no significant difference from K1; this is the owner's practical selection rather than a demonstrated quality advantage.

The cutover ran on 2026-10-04. With the UI and writes stopped, a final consistent snapshot of the previous database was imported and validated: 27 tables, 674,049 rows, 18,983 active chunks, every referenced artifact hash matching, the ledger and every historical attempt unchanged, and no pending or unknown billing. All 18,983 active vectors are in pgvector, byte-identical to the verified set. Keyword index 29f261abafeb1f8c is active. The cutover's previous serving row was H-0fffb2a6ec. The selected BGE-m3-ko row retains its hybrid keyword-first geometry over exact pgvector search (activation refuses a run measured under another corpus-routing rule):

• The BM25 top 6 stay in BM25 order, and the rest come from weighted RRF (k 60, dense weight 1.0).
• Each channel retrieves 50 candidates, fusion keeps 50, and 10 evidence units go to the model.
• This is the best setting that passed the fusion gate against keyword-only K1 at the same limits. Among the alternatives measured, 20–30 units and the local reranker failed.
• HNSW did not reach recall@20 0.99 against exact search on scoped questions, so exact search serves.

Ask has an All documents scope across all 98 active sources. A question naming a project is narrowed to that project's documents; generic title words alone, such as 대학교 or 사업, narrow nothing. A question that only names the project gets its overview passages, ranked by meaning. The needle set finds the target passage in the top 5 for 32 of 33 questions (Wilson 95% 0.85–0.99). Measurements, the paid end-to-end check, the archive and the rollback are in the PostgreSQL handover.

Ask is a conversation within one scope. A follow-up is first rewritten into a standalone retrieval query by a paid call through the same gateway, and the trace records it. Its answer also sees the conversation and the evidence the previous answer cited, so a restatement keeps the earlier facts. The answer then streams in as provisional text until validation finishes. Each sentence ends with numbered citation markers that open the evidence pane (DESIGN.md §12).

The importer and the previous database's backup/rollback paths were deleted after the final import validated. Verified PostgreSQL custom-format dumps are kept under .runtime/archive/ as cold archives that no code references; the previous database's files were deleted on 2026-10-05. The Phase 4 pilot runtime (gold candidates, sealed set and pilot runs behind the release report) lives in its own PostgreSQL database, bidmate_pilot_archive, kept apart from the live ledger with paid admission disabled. Rollback means restoring the newest verified dump, taken after the last paid write (ledger revision 1712), into an empty database with restore-check. That leaves paid admission off. Records written after the dump's watermark are reconciled before paid on (runbook §12). The owner approved paid work and a $10 operating cap on 2026-10-03. The Settings page can change the shared cumulative limit while preserving spending and reservations; it uses the existing no-login attribution model.

Configuration

Settings resolve from the repository location, never the working directory. Optional overrides must be absolute paths.

Without a key the app still runs: search, filters, evidence browsing and retrieval traces are free; paid generation reports that the key is missing until one is entered on the 설정 page. provider: "fake" in the config file never builds a real SDK client.

Langfuse tracing

A Langfuse v4 stack can record every /api/ask request and every drafting run. A request trace shows retrieval (chunk IDs, scores and text), the assembled evidence, the answer model's generation with its usage and ledger cost, and the answer validation. It carries three free, deterministic scores from the existing checks: citation_valid, insufficient_evidence and evidence_tokens. Embeddings, ingestion, OCR and evaluation runs are not traced. Tracing is observability only: admission, reservations and settlement never read it, and an absent, stopped or failing Langfuse changes neither answers nor the ledger.

Every trace names where it came from. Its user, also kept as metadata member, is the signed-in account on the team host and BIDMATE_LOCAL_MEMBER (environment, then .env) on a member's PC. Its environment is server where JupyterHub sign-in is configured and local everywhere else, also added as a tag. Its release is the checkout's <branch>@<commit>, also kept as metadata git_ref. The trace list filters on all three.

The team's shared stack. One Langfuse on codeit holds every member's traces in one project. It also receives the live server's traces, and it keeps the owner's earlier local traces. It listens only on the VM's loopback, so you reach it through an SSH tunnel at http://127.0.0.3:3300, and each member signs in with their own account (runbook 3.7). To send your local runs there, open the tunnel and run ./tools/infra/use-shared-langfuse.ps1 -User <your username> -Identity "$HOME/.ssh/<your key>". It copies the project's keys from the VM into your .env, together with the tunnel address and BIDMATE_LOCAL_MEMBER. With the tunnel closed, traces are dropped with a warning and answers carry on.

A stack of your own, alone. Start Docker Desktop, then run ./tools/infra/start-langfuse.ps1. The first run generates the stack's secrets into the gitignored .runtime/langfuse.env. It creates the organization, the project and its API keys headlessly, and starts compose.langfuse.yaml detached (Compose project bidmate-langfuse, with its own Postgres, ClickHouse, Redis and MinIO). Later runs reuse the same secrets and volumes. The script also writes the three LANGFUSE_* lines into .env and removes any LANGFUSE_BASE_URL there, which would point SDK tools elsewhere. Tracing starts with the next application start.

The UI is at http://127.0.0.2:3100 (MinIO media at port 9190). Both listen only on 127.0.0.2, a loopback address of their own: browsers share cookies across ports, so a second local Langfuse on 127.0.0.1 would otherwise break this one's sign-in. Use that address rather than localhost, which a browser may resolve to ::1. Sign in with LANGFUSE_INIT_USER_EMAIL and LANGFUSE_INIT_USER_PASSWORD from .runtime/langfuse.env. A request's trace ID is derived from its request ID (Langfuse.create_trace_id(seed=request_id)). Text is sent in full, except that one mask on the client redacts API keys, tokens and other secret-shaped values on export.

To turn tracing off, remove the three LANGFUSE_* lines from .env (and from the process environment) and restart the application: no Langfuse client is created. Stop the stack with docker compose --env-file .runtime/langfuse.env -f compose.langfuse.yaml down; the secrets live only in that file, so Compose cannot read the stack definition without it. Add -v only to delete its data, and then delete .runtime/langfuse.env as well.

Answers use gpt-5-mini by default (settings.ALLOWED_GENERATION_MODELS: gpt-5-mini, gpt-5-nano; the team key reaches both). They go through Chat Completions with strict structured output, reasoning_effort: low and a default maximum of 4,000 output tokens, reasoning included: the phase-4 pilot's cap, restored on 2026-10-05 after a 2,000-token cap left a comparison no room past its reasoning. A config naming any other answer model, gpt-6-luna included, is refused at load. gpt-6-luna is not an answer model since 2026-10-07: it remains only the drafting, judge and AI-review model (drafting.MODEL, judges.MODEL), whatever the answer model is. The phase-4 pilot's results and limitations, measured with gpt-6-luna answers, are in the release report. Rates are in settings.DEFAULT_RATES (standard tier, per 1M tokens): gpt-5-mini $0.25 input, $0.025 cached, $2.00 output; gpt-6-luna $0.10 input, $0.01 cached, $0.125 cache write, $0.50 output. Evidence is capped per document by the serving run's limits (4,800 tokens), far below the 272K long-context tier.

Commands

Run from any directory with the environment's interpreter:

Every command except check, load-check and restore-check runs against RFP_DATABASE_DSN and refuses a database without a validated import. Every startup refuses an embedding model outside the compared registry (models.EMBEDDINGS) or at dimensions other than the ones it produces; serving then uses whatever embedding the activated run evaluated. Every connection, pooled, the paid gateway's or restore-check's, refuses a server other than PostgreSQL 18.6, and pooled connections refuse pgvector other than 0.8.6. check and load-check use temporary databases on the server named by RFP_POSTGRES_TEST_DSN (default: the local server from tools/infra/start-postgresql.ps1), and restore-check uses RFP_RESTORE_DATABASE_DSN. init creates the schema in a database the application owns; it never resets spending.

python -m rfp_assistant.cli init --paid-disabled          # schema + allowance row; never resets spending
python -m rfp_assistant.cli manifest                      # all 100 CSV associations, hashes, duplicates, conflicts
python -m rfp_assistant.cli ingest                        # every original; failures are quarantined with a reason
python -m rfp_assistant.cli build-keyword --include-unreviewed   # operating index over every parsed source
python -m rfp_assistant.cli check --phase 1 --provider fake   # automated invariants in temporary state, no key
python -m rfp_assistant.cli check --phase 3 --provider fake   # service, budget and request-state gate
python -m rfp_assistant.cli load-check --users 6 --provider fake   # six concurrent members, temporary ledger
python -m rfp_assistant.cli check --phase 4 --provider fake   # gold, evaluation, sealed run, backup and report gate
python -m rfp_assistant.cli check --phase all --provider fake --save   # every test; recorded for release-report

The screens are a Next.js app in web/ over a FastAPI wrapper of service (src/rfp_assistant/api.py). A build is a static export that the API serves itself, so one process on one port serves both. Node.js is needed only to build:

cd web; npm ci; npm run build; cd ..                     # writes web/out (types come from web/openapi.json)
$env:RFP_CONFIG_FILE = (Resolve-Path docs/history/postgresql-migration/config.example.json).Path   # RFP_DATABASE_DSN set
$env:BIDMATE_LOCAL_MEMBER = "<your name>"   # local only: no hub to sign in with; never set on codeit
python -m uvicorn rfp_assistant.api:app --host 127.0.0.1 --port 8501 --workers 1

The live shared instance runs on the team's GCP VM codeit, not on any member's machine. Members open http://35.255.64.243:8501 and sign in with their JupyterHub account (see Access and paid use). The host, the sign-in settings, the Linux start path (tools/infra/start-postgresql.sh, tools/infra/bidmate.service) and the member list are in runbook §3.

Exactly one worker: paid requests run on the process's own executor and the process holds a database-wide gateway advisory lock. For screen work, run the API on 8511 (with BIDMATE_LOCAL_MEMBER) and npm run dev in web/ (port 8510, /api/* forwarded to RFP_API_URL, default http://127.0.0.1:8511; only with BIDMATE_LOCAL_MEMBER does the API accept writes from that :8510 origin). After changing a route or its shapes, regenerate the schema the screens are typed from with python tools/openapi.py; tests/test_api.py fails while it is stale. npm run lint and npm run typecheck check web/; npm test runs its characterization tests (vitest, snapshots in web/test/__snapshots__/; rewrite them with npx vitest run -u only for an intended screen change).

For local work, use a branch in the original checkout rather than copying the repository. The original corpus, .env and runtime are ignored by Git and do not appear in a linked worktree. If a task already requires a worktree, set RFP_SOURCE_DIR and RFP_DATA_DIR to the existing corpus and shared runtime before starting the server, with permission to write to that runtime. Keep the API key in the server environment. Never copy the budget DB to create a second independently spendable ledger. The shared runtime still has exactly one gateway owner.

A fixture server with FakeTransport is for isolated UI or invariant checks. It is not the running application for user acceptance and cannot establish real retrieval or model-answer quality.

The operating index includes sources whose extraction has not been compared with the original yet; the consultant screen and every trace label them. build-keyword --reviewed-only builds the stricter index from sources that passed the automatic check (auto_verified) or a human review; the operating index also keeps unreviewed and auto_flagged sources, labeled.

Fidelity check against the original

Nobody reads whole documents. For each HWP, fidelity run has Hancom Viewer print the original to PDF and answers the save dialog itself (the Windows default printer must be "Microsoft Print to PDF"). The printed PDF's text layer comes from Hancom's own renderer, not from pyhwp or the HWP loader, so it is an independent witness. It is never parsed as a source of text: an HWP pyhwp cannot read is parsed by the loader during ingest and judged here like every other HWP. The check then compares both ways:

• every extracted paragraph and table-cell line must appear in the rendering (6-character shingles, so a single wrong character is reported);
• every rendered line and ruled-table cell must appear in the extraction;
• runs of 3+ digits must match exactly.

The verdict is auto_verified (자동 대조 통과) or auto_flagged (자동 대조: 확인 필요), with the exact element, table cell and page of every difference. On the verification screen, 원문 대조 (HWP) lists the documents; a person opens only the reported places next to the printed page and can mark the document sample_checked. A human review status is never overwritten by an automatic one. Text inside images is in neither side, so pages with images are listed separately.

python -m rfp_assistant.cli fidelity run                  # every parsed HWP; reuses existing prints (--reprint to redo)
python -m rfp_assistant.cli fidelity run --doc-id <id>
python -m rfp_assistant.cli fidelity show --doc-id <id>   # metrics and findings
python -m rfp_assistant.cli review render --doc-id <id>   # one PNG per page under .runtime/reviews/rendered/<hash>/
python -m rfp_assistant.cli review record --doc-id <id> --reviewer <name> --status sample_checked --locations loc.json --findings findings.json

Printing takes 20–30 s per document and shows viewer windows. Do not print anything else while it runs. The prints are kept in .runtime/reviews/printed/<hash16>.pdf, and the originals are never modified.

Text inside images

ocr reads with PaddleOCR-VL on the GPU the pictures embedded in each HWP file (WMF/EMF rendered to bitmaps, on Windows) and the raster images of each PDF original. The Hancom print is not used. Only regions that fail the fallback test (loop, low confidence, or output length far from what the ink suggests) are re-read by gpt-5-mini, paid with OPENAI_API_KEY in .env (the owner's personal key) through the budget gateway into the shared ledger, from the ocr envelope, which starts empty: the owner funds it with set-envelopes. Results are cached per region under .runtime/ocr/<hash>/, and a rerun resumes. The next ingest places an HWP picture's text as an image_text element where pyhwp met the picture (a picture in a table cell after its table), and a PDF image's next to the text around it. OLE objects, drawings and images that cannot be decoded are counted per document as ocr_unavailable. The text is chunked apart and cited as OCR, and the fidelity check skips it. Design and calibration: docs/rag/preprocessing.md.

python -m rfp_assistant.cli ocr --local-only              # free pass: local reads, flagged regions stay unresolved
python -m rfp_assistant.cli ocr sample                    # paid, reference only: gpt-5-mini on 50 Gemini-read print regions
python -m rfp_assistant.cli ocr                           # gpt-5-mini for the flagged regions, from the ocr envelope
python -m rfp_assistant.cli ingest                        # merge OCR text; then fidelity run and build-keyword again
python -m rfp_assistant.cli ocr coverage                  # embedded images read against the 463 print images

Dataset questions

Drafted questions wait in a review queue. On the 데이터셋 만들기 page a verifier picks development documents and source passages, has gpt-6-luna draft source-bound questions within a consented maximum (charged to gold_eval), and sends the valid drafts to the queue. A reviewer whose typed name differs from both the drafter and whoever started the drafting run then approves each one into the dataset or rejects it into the rejection wiki, always with a note, with the draft shown next to its original passages. The drafting procedure for agents, including exact file locations, is .wiki/gold-drafting.md.

python -m rfp_assistant.cli gold status                   # counts, rejection wiki path, rejections awaiting an inferred reason
python -m rfp_assistant.cli gold submit --file batch.jsonl --batch <id> --dataset dev-pilot --drafted-by <agent>
python -m rfp_assistant.cli gold infer --candidate-id <id> --by <agent> --file inference.json
python -m rfp_assistant.cli gold check                    # dataset file and wiki pages equal their database rendering
python -m rfp_assistant.cli gold repin --batch <new-id>   # after a parser revision: move pending rows to the new extraction by element path
python -m rfp_assistant.cli validate-gold --dataset dev-pilot   # approved rows only

Phase 2: corpus coverage and retrieval selection

Plan: 2-corpus-and-retrieval.md. The order below goes from free work to the one paid maintenance step. Angle-bracket values come from the previous command's output.

python -m rfp_assistant.cli ingest --profile all             # every unique original once; unchanged inputs are reused
python -m rfp_assistant.cli import-reviews --file C:\abs\reviews.jsonl
python -m rfp_assistant.cli recover-source --doc-id <id> --converted-file C:\abs\converted.pdf --review-file C:\abs\recovery.json
python -m rfp_assistant.cli identity                         # CSV institution/title versus the original's own cues
python -m rfp_assistant.cli resolve-metadata --doc-id <id> --field institution --value '"기관명"' --rationale "..." --evidence "공고문 1쪽" --actor <name>
python -m rfp_assistant.cli build-keyword --reviewed-only                         # structural profile
python -m rfp_assistant.cli build-keyword --reviewed-only --profile fixed-512-64  # comparison profile, never serves users
python -m rfp_assistant.cli evaluate-retrieval --dataset dev-pilot --runs K0,K1   # free lexical comparison
python -m rfp_assistant.cli evaluate-retrieval --dataset dev-pilot --runs K1 --index <fixed index version>
python -m rfp_assistant.cli plan-embeddings --index <keyword index version>       # tokens and maximum spend, no call
python -m rfp_assistant.cli build-dense --index <keyword index version> --estimate-id <id>   # paid, stop the UI first
python -m rfp_assistant.cli evaluate-retrieval --dataset dev-pilot --runs D,H --allow-paid-queries
python -m rfp_assistant.cli trial-reranker --dataset dev-pilot --candidate-counts 10,20
python -m rfp_assistant.cli compare-runs --run-id <K0 run> --run-id <K1 run> --run-id <H run>   # table + K1-default recommendation
python -m rfp_assistant.cli draft-activation --runs <run>,<run>,<run> --out C:\abs\decision.json  # inert until the owner fills it
python -m rfp_assistant.cli activate-run --run-id <run id> --decided-by <name> --note "..."   # note optional; or --decision-file C:\abs\decision.json
python -m rfp_assistant.cli report --phase 2                 # report.md, manifest.json (exit gates, dataset revalidated now) and source-map.json
python -m rfp_assistant.cli check --phase 2 --provider fake  # every automated invariant, temporary state, no key

• Ingest. HWP goes through pyhwp's XML walker; when pyhwp fails, the original is parsed by HwpHwpxLoader instead (location format hwp_loader, its own parser fingerprint, warning pyhwp_failed), and only a loader failure quarantines the source, with the loader's reason (hwp_loader_failed, hwp_loader_result_invalid, hwp_empty_output). A source is parsed again only when its original, parser revision (walker or loader), native print, OCR cache or HWP converter changed (--force overrides). The extraction revision is bound to the actual output: a converter that produces different text under the same parser fingerprint creates a new revision, resets review and leaves the old artifact and elements in place for pinned gold rows and citations. Every result, with timing and automatic diagnostics (short_output, no_tables, thin_tail, blank_pages, replacement_characters, parser_warnings, start/middle/end probes), is written to .runtime/reports/ingest-*.json. The diagnostics point at problems; they never mark a source reviewed. One crashing file is recorded as error and the run continues (exit code 1).
• Reviews. import-reviews takes JSON or JSONL records: doc_id (or source_hash), extraction_id (must be the active revision), reviewer, status, locations ([{"element_id": ...}] from that extraction), checks, limitations (a list, empty when none were found) and, for reviewed, coverage.sections. One invalid record imports nothing.
• Recovery. For a quarantined original, recover-source takes an approved PDF conversion and a review JSON with reviewer, method, compared_locations, fidelity_passed and mapping_limitations. A failed comparison keeps the quarantine. A passed one becomes a new extraction revision (pages refer to the converted PDF) that stays unreviewed until import-reviews checks it against its own element IDs. The original and its failure stay recorded. HWPX recovery is refused until a real artifact needs a parser.
• Identity. Byte-identical associations share one extraction and one vector per payload but keep their own metadata and scope. A conflicting field keeps both values visible. resolve-metadata records a canonical value with a written rationale; search filters, ranking and displayed fields then use it, and results keep the CSV values in csv_metadata. Evaluation families group byte-identical files and related revisions (same notice number).
• Chunk profiles. structural (default) and the fixed baselines fixed-256-32, fixed-512-64, fixed-800-96 use the same element order and source-span maps. A piece of an oversized element or row is linked to its siblings: packing keeps the condition with its fact or reports linked_evidence_missing. The structural chunker version changed in phase 2, so rebuild the keyword index once. The phase-1 index stays on disk for rollback and issued citations.
• Analyzer and query terms (kiwi-bm25-2). The analysis copy of indexed and query text collapses letter-spaced headings (사 업 비 → 사업비; 2–6 single syllables whose neighbours are not plain Hangul words) and applies a short observed alias list (지체 상금 → 지체상금, 부가세 → 부가가치세, …); stored evidence and offsets keep the source spelling. Inside the selected scope, K1 drops query terms that cannot discriminate there: terms present in more than half of the scope's chunks (non-positive scope-local IDF), and the selected project's own title/institution terms when the question restates that name. Codes, digits and negations are never dropped, nothing is dropped unless a kept term still matches the scope, and the trace lists what was dropped (scope_redundant_terms:). Each document's title/institution terms are frozen into the keyword index (scope-terms.json, part of its manifest and version), so evaluation and serving reproduce the same behaviour; a later resolve-metadata correction applies after the next build-keyword. An index built with another analyzer/query policy, or without that snapshot, is refused by evaluate-retrieval, trial-reranker, run validation and activation with a rebuild instruction; serving still answers from it but marks every result index_outdated:. Rebuilding reuses every cached vector and keeps the old index for issued citations. Policy retrieval-eval-4 retires runs recorded before this rule. The analyzer version changed: rebuild the keyword indexes; the dense matrix for the new index reuses every cached vector (no provider call).
• Runs. K0 is whitespace BM25, K1 Kiwi BM25, D dense only, H is K1 and D fused by RRF (1/(60+rank)), and HR is H plus the local reranker. Runs are retrieval only, frozen by dataset hash, the evaluated population (each eligible row's question, scope and pinned evidence, plus skipped rows), index manifest, analyzer, dense version and limits, and stored under .runtime/runs/<run_id>/ (config.json, traces.jsonl, scores.json, report.md). Metrics are graded against source spans: hit@k, recall and complete coverage, nDCG@5, MRR, packed-context coverage, qualifier losses, code checks, wrong-scope candidates, latency and Wilson intervals. A configuration that already has scores is reused (--force reruns it). Only independently reviewed dev rows are scored. Skipped rows are listed with their reason.
• Embeddings. plan-embeddings counts unique payloads with cl100k_base, subtracts verified cache hits and stores an estimate bound to the index, model, dimensions, payload set, rates and batch limits (24 h). build-dense refuses a stale or over-envelope estimate. It reserves and settles each bounded batch in the embedding envelope and keeps a failure's settled batches cached. It never resends an unknown outcome: while any embedding attempt's billing is unknown (a timeout, or vectors returned without usage), build-dense refuses and paid evaluation queries stop until the attempt is reconciled. The matrix is published only after every row is verified. Query vectors are cached by text, model and dimensions. Free retrieval uses only cached ones. A paid answer embeds an uncached query through the gateway, and evaluation does so only with --allow-paid-queries (gold_eval envelope).
• Reranker. Optional: pip install -e .[reranker] (or the pinned requirements.txt). Set reranker_revision to the model card's commit hash in the RFP_CONFIG_FILE JSON. reranker_precision (fp32 default, fp16 on CUDA) and reranker_max_length are measured settings: each is recorded in the HR run and applied when it serves; an HR run that recorded no precision serves fp32. reranker_max_concurrency must stay 1: the shared tokenizer is used only under the model lock, because parallel preprocessing crashed on the real GPU. An inference error falls back to H in serving and turns a trial into a blocked bypass artifact. Traces carry rerank_queue/rerank_infer milliseconds and truncated pairs, and the trial reports queue and inference p95 under load separately. An unpinned, missing or failing model records the bypass. The trial scores the frozen H pool at each depth and measures warm latency alone and under six concurrent users. Retrieval runs with the frozen H run's limits, embedding and analyzer, so only the reranker changes. The measured concurrency bound is part of the HR run and is what serving uses. Its gate (nDCG@5 +0.03, no new critical failure, at most +1 s p95) is reported as a column; it does not block activation, because the person picks from the table. Critical failures are a code check failing, a candidate outside the scope, or a numeric_qualifier row (or a row marked critical) losing part or all of its evidence from the packed context.
• Selection. compare-runs writes .runtime/runs/comparisons/<key>.md/.json from recorded scores only: dataset, index/profile, policy, scored denominator, Wilson intervals, nDCG@5, packed completeness, critical/code/scope failures, latency, query cost and whatever would block activation. Runs over different evaluated populations are never compared or recommended against each other, and a run whose population has since changed (a source revision, review or dataset edit) is blocked until rerun. Its recommendation keeps K1 unless another current-policy mode on the same population has no new critical failure and at least +0.03 nDCG@5; the runner-up becomes the phase 4 finalist. draft-activation turns that into a decision file with a rationale draft and the remaining blocking checks; decided_by and rationale stay empty so it cannot activate anything until the owner completes it.
• Gold drafting inputs. gold excerpts --out <abs new dir> writes excerpts.jsonl (a few numeric/qualifier, late-table, repeated-code and deadline elements per dev-family document; each excerpt is a bounded window around its triggering fact, with the table header or neighbouring sentences, the trigger, raw-text offsets and context_clipped) and drafting-context.json (queue counts, rejections with reasons and inferences, existing question keys). It contains source text: keep it in local inputs unless the owner shares it. gold status lists pending_invalid candidates that no longer pass the shared checks (for example a converter case whose document has since been recovered); the review screen shows the same errors. validate-gold requires the converter_unavailable case only while a source is actually quarantined, and reports required_types and operational_cases.
• Activation. A person picks a row; nothing activates on its own. activate-run --decided-by <name> [--note ...], the 실험 비교 view's activate action, or a decision JSON (run_id, mode, decided_by, optional rationale and finalist_run_id) go through the same check: the index and matrix verify, the run is current-policy, and a non-OpenAI embedding still matches its pinned revision and prefixes. A reranker gate result is shown, not required. It switches serving in one transaction and keeps the history in activations. Runs and gates carry the evaluation policy version (EVAL_VERSION). Only runs scored under the current policy can be activated. An HR selection from an older policy keeps hybrid retrieval without the reranker until a current trial passes. Cached corpus and query vectors make that rerun free. Serving reuses the run's evaluated embedding model/dimensions, depths, RRF constant, evidence limits and reranker input length, whatever the process configuration says. Until then the keyword default serves. If the matrix fails verification or a query vector is unavailable, serving falls back to K1 and the trace records why.

Comparison runner: pipelines run every variant, a person picks

Plan rule (0-overview.md): pipelines run every variant, AI reviewers approve gold rows, and a person chooses from comparison tables and activates. Nothing runs on a schedule, and nothing changes what serves until the person clicks.

python -m rfp_assistant.cli compare --matrix lexical      # K0 versus K1
python -m rfp_assistant.cli compare --matrix chunking     # structural and the three fixed profiles, non-serving indexes
python -m rfp_assistant.cli compare --matrix embedding    # 13 models x dense/hybrid; paid rows stop at a priced estimate
python -m rfp_assistant.cli compare --matrix reranker     # 8 local rerankers x whole-list/below-the-BM25-head
python -m rfp_assistant.cli compare --matrix embedding --only Qwen/Qwen3-Embedding-0.6B   # one model's rows
python -m rfp_assistant.cli compare --approve <estimate id> --approved-by <name>        # then rerun the matrix
python -m rfp_assistant.cli compare-cap --usd 0.50 --actor <name> --reason "..."          # Gemini cap, its own ledger
python -m rfp_assistant.cli golden-counts                 # rows of every gold set by status, 50-versus-55 explained
python -m rfp_assistant.cli maintain --actor <name>       # backup → restore check → ingest → HWP fidelity → keyword → embedding → regression → report
python -m rfp_assistant.cli judge-set                     # free: mutated held-out positives, counted apart from the development set

• Maintenance. maintain, or 검증 → 유지보수 → 유지보수 실행, runs the whole DB and vector sequence in order and ends with the regression table (K1 and the serving configuration on the served and the rebuilt index) and one report under .runtime/maintenance/reports/. Every step reuses what is unchanged, so a rerun with no changed input makes no call. A paid embedding stops at its estimate, and a failed step stops the sequence with its name and reason. A run whose reparse cost the rebuilt rows any question ends unverified and names those questions. Each reparse runs in a child process with a per-file time limit (parse_timeout_seconds, default 120 s). A file that runs past it keeps its previous extraction and is retried on the next run, and the run ends unverified naming it. The screen shows per-file progress with elapsed times, and its 취소 button kills a running parse at once, stops any other step at its end, and ends the run cancelled. None of these changes what serves (runbook §13).

• Matrix. A matrix declares its axes (chunk profile, analyzer, embedding model, retrieval, fusion, reranker and mode, evidence units, depth) and its fixed values; compare.MATRICES holds the four named ones, and an absolute JSON path runs a custom one. Every row is scored on the same three populations: the development set inside each question's own document, the development set over the whole corpus, and the needle set over the whole corpus. Only the declared axis changes per row. A row keeps the serving run's recorded limits for every axis it does not vary (fusion; depth with channel depth; units with evidence tokens), and always its search settings (dense_search, hnsw_ef_search). Every effective limit is part of a cached row's identity. The embedding matrix keeps fusion keyword_first:60:1.0:6, 50 candidates and 10 evidence units, and the reranker matrix reranks the serving hybrid's frozen pool.

• Caches. Keyword indexes, corpus and query vectors, reranker scores (rerank_scores, keyed by model and pair) and finished rows (.runtime/compare/cells/) are reused, so a rerun only does what is missing. Every row with a run is written as an activatable run under .runtime/runs/.

• Models. models.EMBEDDINGS and models.RERANKERS pin each model's revision, licence, dimensions or input length, precision and documented query/document prefixes. Local models run on the GPU and embed every active chunk into their own vector set. They never run on the CPU: without CUDA they refuse to load. On the first load, each process caps its PyTorch allocator at the dedicated VRAM free at that moment, minus 512 MB. Past dedicated memory, the Windows driver would otherwise spill into shared system memory and slow the GPU severalfold. An out-of-memory error halves the batch, and the smaller batch is kept; a model that does not fit at batch 1 becomes a failure row. Causal-LM rerankers compute only the last position's logits, without a KV cache. The table records size, peak VRAM, cold load and truncation. A model that fails to load, runs out of memory or crashes is a row with its reason.

• Paid steps. OpenAI rows reuse cached vectors; an uncached OpenAI or Gemini corpus build stops as a needs_approval row with a priced estimate (.runtime/compare/estimates/). It runs only after compare --approve records the person's approval: OpenAI spending goes through the shared ledger, and Gemini through its own cap (compare-cap, refused below committed spend) and ledger (external_ledger). An API model's query vectors are cached after their first paid call, so its query p50/p95 and corpus embedding time come from the dispatch and finish times of its settled ledger calls, at no further cost.

• Output. Each matrix writes .runtime/compare/tables/<matrix>.json and .md. 검증 → 실험 비교 shows the same tables: sortable by any column, the best value per column and the serving row marked. A row opens its missed questions and the activate action, which calls activate-run with the person's name and an optional note. Activation of a non-OpenAI embedding serves its corpus and query vectors from the same local model inside the server process; the earlier text-embedding-3-large run stays activatable for rollback.

Stage review app: baseline ref against candidate refs

A coding agent delivers a retriever, chunking, generation or OCR change as a branch; the person compares it with the baseline on identical frozen inputs and records a decision the agent reads. The CLI runs; the app (review/, a Next.js static export served by review/server.py on its own loopback port, apart from web/) only shows runs and writes decisions. It never runs a pipeline, loads a model or pays.

python -m rfp_assistant.cli stage-review run retriever --base main --cand my-branch --cand .   # '.' = working tree
python -m rfp_assistant.cli stage-review run chunking --base main --cand my-branch
python -m rfp_assistant.cli stage-review run chunking --base main --cand . --inputs-from <run-id>   # no database
python -m rfp_assistant.cli stage-review run generation --base main --cand my-branch    # stops at a priced estimate
python -m rfp_assistant.cli stage-review run ocr --base main --cand my-branch --ledger-dsn-env CODEIT_DATABASE_DSN
python -m rfp_assistant.cli stage-review approve --run <run-id> --approved-by <name>
python -m rfp_assistant.cli stage-review resume --run <run-id>                           # the only step that pays
python -m rfp_assistant.cli stage-review rescore --run <run-id>                          # re-score stored outputs, no calls
python -m rfp_assistant.cli stage-review reprice --run <run-id>                          # new estimate at today's rates, for what is left
python -m rfp_assistant.cli stage-review run traffic --since 2026-10-02 --sample 30 --source-dsn-env CODEIT_DATABASE_DSN   # real answers, read-only
.\review.cmd          # Windows; ./review.command on macOS (double-click in Finder)

• Runs. Each ref is checked out in its own git worktree under .runtime/review/worktrees/ (removed afterwards) and runs in a fresh process with that ref's package and the same frozen inputs. Retriever: the serving activation, its index and vectors, and the dev questions (on their own documents) plus the needle set (whole corpus). Chunking: the elements of every document the serving index holds. Workers get a read-only database session and no API keys; a query vector comes from the cache or a local model, never a paid call. One scorer grades every candidate. .runtime/review/runs/<run-id>/ holds run.json (full commits, input and worker hashes, packages and the host each candidate ran on), inputs/, raw candidates/ output and the view.json the app reads.
• Paid stages. run generation and run ocr freeze the inputs, let every candidate price its own work without a key and stop with estimate.json and state needs_approval; the app does not list the run yet. approve records the person's name in the run and the audit log (it refuses an expired estimate, one past 24 hours, or one that no longer fits the remaining budget envelope). run.json is written before pricing, so a pricing step that failed (a dropped ledger connection after an hour of local OCR reads) or an estimate that expired or met a rate change is redone with reprice: each candidate recorded the token bound of every call it may make, so the new estimate applies the ledger's rates now without running them again. A run stopped midway reprices too, for what it has left (an expired approval never strands its finished work), and the new estimate needs its own approval. resume refuses without that approval, re-checks that the inputs, worker, rate card, refs and the budget for what is left still match the estimate, and only then hands each candidate OPENAI_API_KEY. The approved maximum holds across every pass: the ledger counts what a candidate already spent or left unknown, the gateway refuses any reservation above what is left (that image or answer is shown unresolved with the refusal), and each paid OCR read is kept on disk as it settles. The ledger also names every image region and answer already billed, so a worker that dies midway or cannot write its progress never pays for one again: a billed read whose text was lost is shown unresolved, a finished answer replays from its stored request, an answer billed before its result could be stored is shown lost, and none is priced again. A reservation the worker never dispatched is not spending: the next pass's recovery releases it, so a re-price prices that item again, and the estimate's fit and resume's admission count the balance it holds as available. A candidate stops at that maximum and on unknown_billing (a provider error after dispatch; the owner settles or reconciles that attempt before resuming). A run stays stopped until every approved candidate completes, including one that failed midway (a full disk, a lost connection); resume keeps each candidate's finished answers and pays only for the rest. An answer replayed from a stored request because its earlier output was lost is detected from the ledger, and its latency is reported as unmeasured. The run records each candidate's spend from the shared ledger; rescore re-reads it and re-scores the stored outputs after a scorer fix.
• Generation. Every candidate answers the frozen development set once with gpt-5-mini through the budget gateway (gold_eval envelope, idempotency key <run-id>:<candidate>:<question>:<n>). The spend lands in the ledger of the database the answer path uses, so to count against the shared ledger run against codeit's database through the tunnel (RFP_DATABASE_DSN=$CODEIT_DATABASE_DSN); bidmate.service holds the gateway lock and is stopped only for resume; plan for the downtime, since a 55-question candidate took about 25–40 minutes on 2026-10-10 (an answer took 30–45 s). A candidate whose schema version differs from that database's is refused. With LANGFUSE_* set for the runner, every answer links to its trace; without it there is no link. The view puts each candidate's answer side by side per question, opens a citation's evidence, separates supported and rejected claims, shows validation failures (claim_without_evidence and the rest), required-claim verdicts, and cost and latency with their denominators.
• OCR. The run reads one fixed image set: every image the current OCR cache flags or could not decode, plus a seeded sample of 30 that passed. Each candidate decodes the original bytes and reads them with PaddleOCR-VL (refused on a host without CUDA), then re-reads the images its own checks flag with gpt-5-mini on the personal key (ocr envelope). --ledger-dsn-env names the database holding the shared ledger when the runner's own database is local. An image a candidate cannot decode or read counts as not OCR'd. On an RTX 3080 the estimate's local reads took about 30 minutes per candidate for 200 images, and a paid second read about 16 s. The view shows each image beside every candidate's text with a character diff against the baseline, the flag reasons, and flagged, re-read and unreadable counts with their denominators.
• Configuration. A ref changes code, and may also commit a review-variant.json at the repository root that overrides the frozen activation for the comparison only, e.g. {"retriever": {"mode": "hybrid_rerank", "reranker": "<RERANKERS key>", "limits": {"fusion": "rrf"}}, "chunking": {"profile": "fixed-512-64"}} (retriever also takes embedding and dense_version; generation takes reasoning_effort and max_output_tokens). An accepted variant is reproduced with compare and activate-run; the file itself is not merged.
• GPU. A candidate that needs a local embedding model or reranker on a host without CUDA is refused with its reason and stays in the run, like a failed one; neither is hidden or ranked. Run GPU comparisons on a GPU host, export the run folder (내보내기) and import the zip on another machine (실행 폴더 가져오기); every candidate keeps the host it ran on.
• Decision. The person picks one candidate or none, gives a reason for each rejected one and may note questions or documents. The app writes .runtime/review/decisions/<run-id>.md and .json (run IDs start with the stage) and copies a short prompt that points the agent at the file.
• Answer notes. In a generation run, select any part of an answer's summary, claim, conflict value, next action or missing field and write what is wrong with it in your own words; the span stays highlighted and a click shows, edits or deletes its note. Each answer can be marked 검토함 · 문제 없음; an answer with a note counts as reviewed too, and the toolbar and each candidate's summary show how many of the run's answers a person has read, the denominator for any failure count. 한 답변씩 읽기 shows one answer at a time (question and required claims on top, the answer large, opened evidence beside it; ← → move). Notes live in .runtime/review/annotations/<run-id>.jsonl (review-annotation-1), outside the run folder and git: one append-only record per note, edit, deletion or reviewed mark, with run, question, candidate and commit, field, code-point offsets, the exact selected text, the note, the author (git config user.name) and the time. The server refuses a note whose offsets do not select that text in the stored answer, and any two records sharing an id. Export carries the run's notes as annotations.jsonl in the zip (leaving out a note whose suggestion approval failed to commit, so an approved AI note stays live where it is imported) and import puts them back into the store before it installs the run, so an import that fails leaves no run behind and can simply be retried. Every write under the data folder's review/ takes an OS file lock, so two app windows started over one data folder never overwrite each other's notes or decisions; the decision prompt points the agent at the file. Browser walk on a real run: answer span annotation.
• Failure types. The notes become named failure types that the person accepts. The app and CLI never group them themselves and call no model; the coding agent does the grouping. python -m rfp_assistant.cli error-analysis export (no database) or 실패 유형 › 묶기 프롬프트 복사 writes every live note that is in no type yet to .runtime/review/taxonomy/exports/<export-id>/notes.jsonl, with a prompt.md, and gives a one-line prompt that points the agent at the bundle. The agent writes proposal.json (failure-taxonomy-proposal-1: types of name, definition and note ids only) and runs error-analysis propose --file <proposal.json>. propose refuses the whole file if a type lacks a Korean one-line name, a one-sentence definition, or at least one existing note; if it names a deleted or unknown note, or a note another type already holds; or if it carries any other field. The AI may only group the person's notes, never add types of its own. On 실패 유형 each proposed type shows its definition and the quoted spans and notes behind it, and the person accepts it, accepts it under a new name, merges types, splits one by assigning each note to a part, or rejects it with a reason. A rejected type's notes can be grouped again. Every decision is appended to taxonomy/history.jsonl, and each change to the accepted set writes taxonomy/versions/v<N>.json. 진행 상황 shows breadth (accepted types), depth (answers with a note of each type over all reviewed answers; types under three cases are marked) and how many types first appeared in the last 10, 20 or 50 reviewed answers. Each generation candidate's summary lists its type counts over its own reviewed answers, and release-report adds a "Human error analysis (not a sealed metric)" section with types, counts, denominators and source runs. These counts sit beside the release gates and never change a pass line. Browser walk: failure taxonomy.
• Similar cases. On an accepted type, 비슷한 사례 찾기 프롬프트 복사 writes taxonomy/suggest-exports/<id>/ with the type's definition, its approved spans, the spans the person rejected (with reasons) and every stored generation answer, reviewed or not. It then copies a prompt for the coding agent. The agent writes suggestions.json (failure-type-suggestions-1: run, question, candidate, field, the exact span text, an optional start, and a one-line reason) and runs python -m rfp_assistant.cli error-analysis suggest --file <suggestions.json>. suggest refuses the file for an unknown or unaccepted type, a span not verbatim in its stored answer field (or occurring twice without start), any other field, a span already filed under the type, a repeat of a waiting suggestion, or a span the person rejected. The type's queue shows each suggestion highlighted in its answer with the agent's reason. 승인 (approve) files it as a note with origin: ai_suggested and approved_by; 거절 (reject) keeps it with an optional reason that goes into the next bundle. Neither changes the taxonomy version. Approved suggestions never make an answer reviewed and never enter the rate: the screens and the release report show them as a separate count, "AI 제안 승인 +n", and say that suggestions do not find every case. An answer marked 검토함 · 문제 없음 before a type was accepted shows "기준 추가 후 다시 보기 (type)" until it is marked again or unmarked. 진행 상황 counts how many remain, and a generation run filters them as 다시 볼 답변. A rename, merge or split keeps the earliest acceptance time. Browser walk: similar-case proposals.
• Traffic. Gold questions encode the failures already anticipated, while real requests show the ones nobody expected. stage-review run traffic opens one read-only session (default_transaction_read_only=on) to the database named by --source-dsn-env (codeit's through the tunnel). From it, the run reads every 질문하기 request (idempotency key = generation id, a UUID, with no verifier run, so no evaluation, stage-review or sealed-set request) created since --since that finished with a stored outcome: completed, or failed with its validation failure. The run freezes each request's request_json, result_json and trace_json into .runtime/review/runs/traffic-<stamp>-<hex>/inputs/requests.jsonl.gz. Every member ID becomes a stable pseudonym, an HMAC under .runtime/review/traffic/pseudonym.key, which never leaves that folder. One exit gate writes every file of the run and refuses any file in which a known member ID survived. The DSN is never recorded, only its variable's name. Requests are grouped by rule (mode, outcome, validation failure, document). The run draws one representative per group, validation failures first, then non-answers, then larger groups. Extra picks go to large groups in proportion to their size, with a recorded seed, and every pick records why it was taken. The app shows a traffic run with the warning that the sample is a guess and does not cover everything. 전체 지도 shows every request of the window, one square per request, marked noted, reviewed or sampled, with a filter for the groups the sample missed. Any request can be read from there and added to the sample by hand (sample records in the annotation store). A traffic answer opens in the generation run's one-answer reading view and takes the same span notes and reviewed marks. A validation failure shows the refused answer read back from the trace's raw output, which is capped at 4000 characters. A 전체 문서 metadata answer's project list is stored apart from result_json (request_listings). The read joins it back in the same read-only session, so the frozen record and the reading view hold every listed project and the 확인 필요 rows with their reasons. Each row keeps what the consultant's table showed: institution, type (marked when a reviewer corrected it), amount, the dates the query filtered or sorted on, and the cited description. Each description takes span notes like a claim (listing.<n>). Its notes join the same failure types, counts and similar-case search as gold-run notes. The taxonomy, progress and release-report label each note and count gold or traffic (골드 / 실제 요청). Nothing is written to the serving database and no answer is generated again. Browser walk: traffic sample review.
• Launchers. review.cmd and review.command run review/server.py with the Python 3 on PATH (RFP_REVIEW_PYTHON overrides). The first start runs npm ci and npm run build in review/; later starts skip both while review/ is unchanged. The server binds 127.0.0.1 on a free port and opens the browser.

Phase 4: evaluation and release

Plan: 4-evaluation-and-release.md. Procedure: runbook §10–12. Outcome on this branch: release report and phase-4 handoff.

python -m rfp_assistant.cli validate-gold --dataset dev                       # gold-2 rows; `test` prints IDs/counts only
python -m rfp_assistant.cli freeze-dataset --dataset dev --actor <owner> --reason "<why>"
python -m rfp_assistant.cli evaluate-retrieval --dataset dev --runs K0,K1,D,H  # free; the sealed split is refused
python -m rfp_assistant.cli plan-run --dataset dev --action answer-finalists   # every remaining attempt priced, nothing sent
python -m rfp_assistant.cli run-answers --estimate-id <id> --actor <owner>     # paid, resumable, UI stopped
python -m rfp_assistant.cli plan-run --action embedding-comparison --runs <K1>,<H>,<H>,...   # then run-answers --workers 6
python -m rfp_assistant.cli export-review --run-id <A-run>                     # blind sheet; import-review applies it
python -m rfp_assistant.cli freeze-release --run-id <run> --answer-run <A-run> --decided-by <owner> --rationale "..."
python -m rfp_assistant.cli plan-run --action sealed --freeze-id <F-id>        # then run-answers: once per test set
python -m rfp_assistant.cli plan-run --action latency --waves 5 --users 6      # then latency-run
python -m rfp_assistant.cli backup --destination D:\abs\new-dir --actor <owner>
python -m rfp_assistant.cli restore-check --backup D:\abs\new-dir\manifest.json   # fresh staging, paid off
python -m rfp_assistant.cli check --phase all --provider fake --save
python -m rfp_assistant.cli release-report --latest                            # read-only; ready / limited / blocked

• Gold-2 datasets. dev (.runtime/datasets/dev.jsonl) and the sealed test (.runtime/sealed/test.jsonl) share the review queue with the pilot. Labels are evidence groups of alternative source spans (hash, extraction, element, offsets/cells, exact quote), never chunk IDs; required claims are typed (number + unit, date + time, text patterns) with qualifiers and criticality. The validator rejects unreviewed or self-approved rows, missing quotes, cross-split families and paraphrases, unverified negatives and source failures labeled as absence, and labels any set below the 60 + 60 targets pilot. Sealed rows are reviewed only through the owner's CLI (gold show/decide/second-review).
• Answer evaluation. At most two development finalists (by default the activated run and its recorded finalist). Each answer goes through the service's own answer path with the finalist's retrieval configuration pinned, charged to the gold_eval envelope. Resuming never re-sends a finished row or a row whose billing is unknown. Scores keep denominators and Wilson intervals; text claims and unlabelled citations wait for blind human review, and there is no paid judge.
• Embedding comparison. --action embedding-comparison answers the development set once per named retrieval run, with no two-run cap (the cap stays on release finalists, and a comparison run, E-…, is never a release's answer run). The first run is the baseline. Each other run gets its pass-rate difference on the same questions, an exact McNemar p-value and its Holm adjustment. finalize writes .runtime/compare/tables/answer-embedding.{json,md}, which 실험 비교 shows as 임베딩별 답변 next to the retrieval columns the embedding table measured. run-answers --workers N answers N retrieval runs at once (procedure: runbook §10.3a).
• Sealed run. freeze-release records the release candidate (activated run, code/metric/prompt/model/rate/settings hashes, frozen dev/test manifests, the development selection). The sealed set then runs once; any later run is a labeled post-test regression.
• Release report. release-report decides ready only when every hard check passes and every quality target is met on sealed gold; otherwise limited (unverified, unmeasured, missed or pilot-only) or blocked (a failed hard check).
• Active retrieval mode. Serving uses whatever activate-run last recorded in the owner's .runtime; until then it is the keyword default kiwi_bm25 with no dense or reranker stage. This repository records no activation of its own; release-report and report --phase 2 print the active run.

Access and paid use

Members sign in with their JupyterHub account, the one they use on codeit's :8000 (owner decision 2026-10-06, which replaced the earlier no-login decision). BidMate is registered on the hub as the OAuth service bidmate. 로그인 sends the browser to the hub, the hub sends it back to /api/auth/callback, and BidMate exchanges the code with the hub from the server itself and asks the hub who signed in. BidMate never sees a password and keeps no user table. Only the hub usernames in BIDMATE_ALLOWED_USERS get a session (an HttpOnly cookie, 12 hours, kept in server memory); any other hub user gets 403. Every /api route except sign-in answers 401 without a session, and every allowed member gets all pages: 질문하기, 검증 and 데이터셋 만들기 (layout in DESIGN.md). The hub username is recorded on paid requests, review decisions, corrections and audit rows. Each member's OpenAI key from 설정 belongs to their account in any browser, and only in the server's memory.

The sign-in settings come only from the server's environment (/etc/bidmate/server.env on codeit, runbook §3.1). Without them the API refuses every call. A local run without a hub sets BIDMATE_LOCAL_MEMBER to the developer's name instead. Budget administration (settlement, reconciliation, external adjustments, paid on/off, the audit log) is owner CLI only: unresolved, settle, reconcile, adjust, paid, audit, each with --actor and a reason.

Local verification: verification.json lists the major flows for the local verification service. Each flow runs as python -B tools/verify.py <flow-id> with the fake provider and ends with one local-evidence block. Browser flows need pip install -e .[verify]. If the service uses another Python, set RFP_VERIFY_PYTHON in its private env_file to the absolute executable in the prepared application environment; the runner selects it before importing dependencies. See runbook §8.

Paid answers run in the background on a bounded executor (6 workers, 12 admitted requests); the page polls read-only status every second and shows the reserved maximum, the settled cost or the unknown pending cost of its own request. A request is persisted under its idempotency key before it runs, so reruns and double clicks never start a second call. An answer renders only while the screen still asks exactly what it asked (same documents, question, mode and date); otherwise it stays in "내 최근 요청" as history and its billing still settles. Two selected documents allow a balanced comparison (each side retrieved with the single-document limits, exactly as the retrieval gate measures it; an answer that drops a side is rejected, and only an inference comparing the sides may cite both); basic information (typed CSV values with unknown/zero/conflict states) and the structured requirement list are free.

Paid generation stays disabled until the owner records the project dates, the prior use, the allowance and the cap, after rechecking current model prices. The current configuration is a dedicated $5 allowance with a $5 hard cap. The live ledger also carries the owner adjustment external:pr8-pilot-ledger ($0.448818, what the PR #8 pilot spent on the same account), so its spent total matches the account:

python -m rfp_assistant.cli configure-budget --start 2026-09-30 --end 2026-10-28 --prior-use-usd 0 --prior-use-evidence "<where the number came from>" --allowance-usd 5 --cap-usd 5 --confirm-rates --enable-paid
python -m rfp_assistant.cli budget-status

Every paid call reserves its maximum cost atomically against the cap before dispatch, pricing every input token at the cache-write rate. It settles once from reported usage (uncached, cached and cache-write tokens) and keeps timeouts as unknown (pending) cost until reconciled. SDK retries are disabled; nothing retries automatically. Only one process may own the paid gateway per data directory.

Parser licenses

Recorded before any distribution decision:

Layout

src/rfp_assistant/ holds one package. Its subpackages follow the layers of the .omm architecture diagram; api (the HTTP routes the screens call, one service function each) and cli stay at the root as the two entry points, next to settings and contracts. service/: service (also the bounded request executor), answers (phase-4 answer runs, scoring, blind review, latency sample), drafting, ops (fake-provider load check and the phase-3 report), auth. retrieval/: retrieval, chunking, dense (embedding cache, matrix, reranker), vector_store, models (the pinned embedding and reranker registry, local GPU runners, the Gemini ledger). corpus/: ingestion, ocr, fidelity. evaluation/: evaluation (pilot and gold-2 validation, frozen retrieval runs, source-span metrics), gold (the review queue), sealed (release freeze and the single sealed run), judges, judge_set, compare (the axis-matrix runner and its tables), stage_review and stage_review_worker (baseline and candidate refs on frozen inputs for review/), release (backup, staged restore, release report). gateway/: budget, generation (the only SDK call site), tracing. storage/: postgres, store, postgres_backup, maintenance. web/ holds the three pages, 질문하기, 검증 and 데이터셋 만들기; budget administration is owner CLI only. Tests are standard unittest under tests/.