bge-sidecar

ONNX inference sidecar of the dew_flow_* family — the embedding/rerank engine dew_flow_rag_qln talks to over HTTP (design records live in this repo's research/; the retrieval product's side is dew_flow_rag_qln · research/architecture.md):

• BGE-M3 dense + learned-sparse embeddings — FP32, the official BAAI/bge-m3 ONNX export, two engines over the same model (TextEmbedding + SparseTextEmbedding from fastembed), max_length pinned to 8192 (fastembed's default 512 silently truncates otherwise).
• BGE-Reranker-v2-M3 cross-encoder rerank scores (sigmoid-normalized 0..1, aligned with the input document order — same contract as the retired tools/reranker Python sidecar).

Build (once per machine, like tools/ts-parser needs npm install)

Requires Rust (rustup, MSVC toolchain) + VS Build Tools with C++.

# AMD GPU (DirectML — the default feature)
cargo build --release

# NVIDIA GPU (CUDA)
cargo build --release --no-default-features --features cuda

# Both EPs in one binary
cargo build --release --features cuda

# CPU only
cargo build --release --no-default-features

# AMD on Linux/WSL (ROCm via the MIGraphX EP) — run INSIDE WSL, see the dedicated section below
cargo build --release --no-default-features --features migraphx --target-dir target-wsl

The Windows binary lands at target/release/bge-sidecar.exe (the WSL flavor at target-wsl/release/bge-sidecar — separate target dir so the two OS flavors never clobber each other). The Aspire AppHost registers it via AddExecutable when Aspire:BgeSidecar:Enabled is set and the binary exists (graceful-off otherwise), and injects its URL into the API/worker as Agent__Embedding__SidecarUrl / Agent__Reranker__SidecarUrl.

Model files (~2.2 GB for bge-m3 FP32 + the reranker) download from Hugging Face on first use into .model-cache/ (override with MODEL_CACHE_DIR). First /embed and /rerank calls are slow (lazy load); subsequent calls are warm.

Endpoints

/embed's dimension is the width of the dense rows in that same response, read from one of them — so a caller sizing a vector collection cannot size it wrong. GET /models states what this build can do before a pass starts: dimension there is measured, so it is null until something has been embedded (unknown is a value, never 0), and available reports the engine while tokenizer_available reports the tokenizer file — "engine cold, tokenizer ready" is a real and useful state.

kind is accepted for contract compatibility; BGE-M3 embeds queries and documents identically.

Per-request timings (timings, 2026-08-15)

Both inference responses attribute the request's wall-clock, because every number here used to die in this process's own log file — and the most important one was never measured at all:

request_id is an optional, opaque caller correlation id: echoed verbatim in the response and prefixed to the request's pass log lines, so two concurrent requests can be told apart in both places. Absent ⇒ empty echo, unprefixed logs.

Configuration (env, injected by the AppHost)

The provider chosen at the first model load is pinned until restart (/health reports it) — execution provider is machine-level hardware config; changing the setting requires a sidecar restart.

VRAM budget (why the defaults are small)

bge-m3 and bge-reranker-v2-m3 are XLM-RoBERTa-large: 24 layers, 16 attention heads. One layer's attention-score tensor is [batch, heads, seq, seq] in FP32 and softmax holds a second buffer of the same shape, so the transient peak is

batch × 16 heads × seq² × 4 B × 2

seq is squared, so the token cap is the knob that decides whether a pass fits; 1024 tokens (~4k chars) covers essentially every method body indexed. DirectML does not fail an over-sized allocation — it over-commits into shared (system) memory, so the old defaults surfaced as a full VRAM bar, tens of GB of process RAM and an index pass crawling over PCIe, with no out-of-memory error anywhere.

Both knobs ride on every /embed request (the host's Settings → RAG → Sidecar memory budget page owns them and shows the arithmetic live), so the env values above matter only for the window before the first request and for a bare cargo run.

Read loaded_*, not the configured values (2026-08-13). /health's embed_max_length and max_batch are the CONFIGURED fallbacks — what a request that carries no envelope of its own would get. They are not what the last embed ran at, and reading them as such is how three numbers ended up describing one pass: a host chunk of 126 texts per HTTP call, an operator batch of 64 sequences per forward pass, and a /health that said 4. Only the last was wrong-by-omission — the requests carried 64 and Limits::resolve honoured them; nothing reported it. loaded_embed_max_length and loaded_max_batch now report what actually ran (null until one has), the same requested-vs-active split the provider fields already carry. The configured batch default was also raised 4 → 64 to match SidecarMemory.DefaultMaxBatch: at the shipped 256 cap a batch of 64 costs 512 MiB of attention, and the old 4 was sized for the 1024-token era. The cost model lived in one place in the host — at the time of this note that host was the now-frozen ClaudeRag predecessor (v2.Web.Contracts/Models/SidecarMemoryModels.cs, SidecarMemory, with its tests); today's host is dew_flow_rag_qln, and the citation is kept as the record of where the decision was made.

The wedge detector: the one wait this process cannot cancel (2026-08-16)

An ONNX Runtime forward pass cannot be cancelled. It is a C++ call on a thread this process does not own, and a thread merely stuck inside it never panics — so the poison-healing lock, which recovers a mutex a panic poisoned, can never reach it. Until this landed, nothing else could either: every later /embed queued on .lock() forever, /health reported the freeze exactly as it reports a healthy multi-minute build, and the daemon's deliberately infinite sidecar HTTP timeout composed the two into a system-wide freeze with no symptom anyone could observe.

The remedy is visibility plus a ceiling, not cancellation:

• The holder stamps itself — what it is doing and since when — in a mutex of its own, never inside the engine slot. A holder wedged under the engine mutex could never be observed through it.
• /health says so. status flips to "wedged", wedged: true, and in_flight[] carries { engine, phase, activity, elapsed_seconds, ceiling_seconds, wedged } per held engine.
• New requests fail fast with 503 and a message naming the activity, the elapsed time and how to recover — instead of queueing behind a mutex that will never be released.
• The log says it unasked, once per wedge: the party that would otherwise have polled /health is precisely the one already blocked.
• /unload still answers. It waits on the blocking pool with a ceiling, never on a reactor thread, and an engine it could not take stays loaded and is reported as such.

Why the ceilings are minutes, not seconds

A false "wedged" is expensive in both directions, so the numbers are deliberately generous:

A first-ever MIGraphX shape compile is correct slowness and must never be flagged. That is also why the process-exit last resort is opt-in and off: killing a process mid-compile is exactly how a corrupt .mxr reaches the compiled-model cache, which is the 2026-07-31 incident the build canary exists for. When it is enabled, its ceiling is measured from the wedge verdict, never from the phase start.

Token accounting: why /embed reports what it tokenized (2026-08-13)

An input longer than max_length is truncated to a prefix by the tokenizer and embedded as though that prefix were the whole text — no error, no warning, no counter. The caller receives a perfectly well-formed vector for a document it never sent, and the tail of that document simply stops being searchable. The host cannot detect this on its own: it owns no tokenizer.

So every /embed reply now carries, parallel to texts:

The counter is BGE-M3's own tokenizer.json, resolved through the tokenizer registry at startup (see below) purely to count; it never feeds inference. A missing file logs a warning at startup and turns accounting off rather than failing the sidecar. A text the tokenizer refuses does the same for that response: the count used to be folded to 0, which reads as "measured, and definitely not truncated" — the exact inversion of the only guarantee this accounting exists to give, on the one signal the host cannot compute for itself.

What it is for. The host (SidecarEmbedder) turns any reported truncation into EmbedInputTruncatedException, which fails the index run with an operator-readable diagnosis instead of storing the mutilated vector. It is deliberately the one embed failure that is not degraded-and-retried: the same text at the same cap truncates identically forever.

Measured, on an AMD Radeon AI PRO R9700, by two independent methods that agree — embedding [prefix, prefix + marker] and finding where the dense vectors become bit-identical, and the tokenizer's own count:

Real text runs at 2.99–3.50 chars/token. The host had been budgeting 4 — a ~34 % overshoot: 1024 characters of C# is 300 tokens against a cap of 256, so 44 tokens of every filled chunk were thrown away. EmbedBudget (host side) now spends the measured floor less a safety margin.

The tokenizer registry: counting for models this process never loads (2026-08-16)

This sidecar can count for a wider set of models than it can embed, and that is deliberate. A chunker has to size a chunk with the serving model's own tokenizer — a chunk fitted to a window the model does not have is a chunk that gets truncated — and nothing on the .NET side can read a HuggingFace tokenizer.json at all (Microsoft.ML.Tokenizers has no regex pre-tokenizer and no NFC normalizer). Counting therefore happens where the reference implementation already lives.

The names /tokenize answers for are a table, not a match arm:

Three properties are the contract:

• Registered ≠ available. An unknown name is a 400 that names every registered row, so a caller corrects itself from the answer instead of reading this file. A registered name whose file is missing is a 200 with available: false — the caller's name was right and the deployment is what is missing, and collapsing the two would send a chunker hunting for a typo while a file was simply absent.
• Loaded at startup, never on the first request. The counters used to be OnceLocks filled inside the first /tokenize call, so that request paid a directory walk and a multi-MB parse on the async runtime, and a model cache swapped under a running sidecar silently changed the answer. The startup log names every row and the file behind it — a count whose tokenizer nobody can name afterwards is a count nobody can reproduce.
• The encode runs on the blocking pool, under a cap. "Pure CPU" is a statement about the card, not about the reactor. TOKENIZE_MAX_TEXTS bounds the rows because the body limit does not: enough short texts fit under 2 MB to be tens of thousands of encodes in front of /health and /unload, which are the two endpoints an operator reaches for when something is stalled.

Adding a third model is a row and a path. What each row is — kind, embedding dimension, max sequence length, and whether it can answer right now — is GET /models, so a consumer validates a recipe before a pass rather than discovering a mismatch inside one.

AMD on Linux/WSL (ROCm via the MIGraphX EP)

Background (2026-07): the ONNX Runtime ROCm EP was removed upstream (ORT 1.23; ROCm 7.0 was the last release carrying it) — MIGraphX is the only AMD GPU EP in current ORT. DirectML stays the AMD path on native Windows; this section is the direct-ROCm path through WSL2.

Nobody ships ONNX Runtime with the MIGraphX EP as a prebuilt C library (AMD publishes Python wheels only; pyke/Ubuntu builds have no MIGraphX), so this flavor is ort in load-dynamic mode: the sidecar dlopens a machine-local libonnxruntime.so named by ORT_DYLIB_PATH at startup.

The dylib version is coupled to the ort crate: ort =2.0.0-rc.12 builds with api-24 enabled and refuses anything older than ONNX Runtime 1.24 (ort::MINOR_VERSION). Worse, the crate's own version check deadlocks instead of erroring on an older dylib (rc.12 bug: the error path re-enters the API OnceLock it is initializing), which froze the whole sidecar — and the C# indexing stage — on the first /embed when a 1.23.2 build was installed (diagnosed 2026-07-27). The sidecar now preflights the dylib at startup and exits with the required version instead of hanging. When bumping fastembed/ort, re-check ort::MINOR_VERSION and rebuild the dylib to match.

One-time setup inside WSL (Ubuntu; assumes ROCm + amdrocm-migraphx/-dev installed and rocminfo shows the GPU):

# 1. build ONNX Runtime with the MIGraphX EP — tag must satisfy ort::MINOR_VERSION (>= v1.24 for rc.12)
git clone --recursive --depth 1 -b v1.24.4 https://github.com/microsoft/onnxruntime.git ~/onnxruntime
cd ~/onnxruntime
./build.sh --config Release --build_shared_lib --parallel --skip_tests \
  --use_migraphx --migraphx_home /opt/rocm --rocm_home /opt/rocm \
  --cmake_extra_defines CMAKE_HIP_ARCHITECTURES=gfx1201 \
  --compile_no_warning_as_error         # GCC 15+: also patch missing <cstdint> includes as reported

# 2. install the libs to the stable path the AppHost defaults to. On an UPGRADE, move the old
#    install aside first so stale .so.<oldver> files and symlinks cannot linger next to the new ones.
sudo mv /opt/onnxruntime-migraphx /opt/onnxruntime-migraphx.bak 2>/dev/null || true
sudo mkdir -p /opt/onnxruntime-migraphx/lib
sudo cp -a build/Linux/Release/libonnxruntime*.so* \
           build/Linux/Release/libonnxruntime_providers_*.so /opt/onnxruntime-migraphx/lib/

# 3. build the sidecar's Linux flavor (rustup toolchain inside WSL).
#    pkg-config + libssl-dev: on Linux fastembed's HF downloader links system OpenSSL
#    (Windows uses Schannel, so the Windows build never needed them).
sudo apt install -y pkg-config libssl-dev
cd /mnt/<drive>/<repo>/tools/bge-sidecar
cargo build --release --no-default-features --features migraphx --target-dir target-wsl

AppHost config (Aspire:BgeSidecar:*): set WslDistro (e.g. Ubuntu) to flip the sidecar into WSL mode — it then launches wsl.exe -d <distro> --cd <repo>/tools/bge-sidecar -- bash -lc "… exec ./target-wsl/release/bge-sidecar" instead of the Windows exe. WSLENV carries PORT/ORT_* across the Windows→Linux boundary (Aspire sets them on the wsl.exe process; WSL drops them otherwise). Related keys: WslOrtDylib (default /opt/onnxruntime-migraphx/lib/libonnxruntime.so), WslHipVisibleDevices (default 0 — pins the discrete card in HIP order so an iGPU is never picked), Provider (defaults to migraphx in WSL mode).

The compiled-model cache is MANDATORY (ORT_MIGRAPHX_MODEL_CACHE_PATH)

ROCm 7.x's MIGraphX EP always saves the compiled model. With no cache path it writes to "", the write fails, and the failure propagates into the kernel call — so every /embed answers 500 after a ~2-minute compile, with the GPU idle in between:

migraphx_save: Error: file_buffer.cpp:77: write_buffer: Failure opening file: ""/21000-…-0.mxr
Non-zero status code returned while running MGXKernel_… Status Message: Failed to call function

That is the whole symptom of "the index stage runs for ages and nothing happens" (diagnosed 2026-07-27; the EP ignores ORT_MIGRAPHX_SAVE_COMPILED_MODEL — the path is the only knob). With the path set, measured on the R9700: first call 214 s (compile + save), every later call at the same input shape ~0 s. main.rs therefore preflights the directory at startup and exits with the fix instead of failing per request; the AppHost sets and creates it (Aspire:BgeSidecar:WslMigraphxCacheDir, default $HOME/.cache/bge-sidecar-migraphx/device-N).

Keep it on the Linux filesystem, never under /mnt — each entry is about the model's own size (~2.3 GB) and DrvFs would crawl. Budget disk accordingly and prune the directory when you change the memory envelope (EMBED_MAX_LENGTH / MAX_BATCH produce new shapes, hence new entries).

Shape pinning (PIN_INPUT_SHAPE, on by default under MIGraphX)

MIGraphX compiles per input SHAPE, and fastembed pads with PaddingStrategy::BatchLongest (fastembed/src/common.rs), so a batch's padded length follows its longest text. Real indexing therefore hits a new shape almost every batch — measured on a live Fast pass: ~2-4 minutes of compile per batch with the GPU otherwise idle, and a ~2.5 GB cache entry each time (two batches had already written 5 GB). Over thousands of methods that is days of compiling and hundreds of GB.

So embed normalizes the layout before fastembed sees it (pin_shape): a "ruler" sequence that truncates to max_length leads every chunk, and the tail is filled so the last chunk is full — giving exactly ONE shape (max_batch, max_length), hence one compile and one cache entry per run. The ruler rows are dropped from the response (unpin_rows, which fails loudly rather than return a misaligned row).

The trade is one wasted row per max_batch - 1 real rows, and every row computed at the full token cap instead of its natural length — worth it only where a recompile costs minutes. PIN_INPUT_SHAPE is therefore auto by default: on for migraphx, off for cuda/dml/cpu, which take dynamic shapes in stride. Force it with PIN_INPUT_SHAPE=1 / 0. It needs MAX_BATCH >= 2 (no room for a ruler otherwise) and is skipped below that.

Changing EMBED_MAX_LENGTH / MAX_BATCH changes the pinned shape — expect one fresh compile and a new cache entry, and prune the old ones.

• The model cache (.model-cache/) is shared with the Windows flavor via /mnt/<drive> — no re-download; reads over DrvFs are just slower on first load.
• /health's adapter is null on Linux (DXGI is Windows-only); the raw ORT_DEVICE_ID semantics apply (HIP order, after the HIP_VISIBLE_DEVICES mask).

Regenerating the canary reference

Every freshly built engine is checked against src/canary-reference.f32le before it is allowed to serve — a crash mid-compile can leave a cached program that loads fine and stably produces garbage, and that is the one failure no length or shape check catches. The oracle is regenerated by the binary itself:

# writes over the committed file; add a path to write somewhere else and compare first
bge-sidecar --write-canary-reference [path]

It builds the embed session through the ordinary loader on the configured provider, embeds CANARY_TEXT at the production shape, and prints the cosine against the file it is about to replace. Read that number:

• ~1.0 — nothing needed regenerating. The model has not moved.
• far from 1.0 — either the model deliberately changed (a new export, a new checkpoint) or this build is producing garbage. Only you know which, and regenerating to silence a failing canary is the one misuse this tool must not be put to.

A byte-level diff is normal. Measured on an R9700 through DirectML: cosine 1.000000000 and yet 1012 of 1024 elements differ, max delta 2.868e-07. GPU arithmetic is not bit-reproducible across runs, which is why the check is a cosine and not a comparison of bytes — and why git diff calling this file changed proves nothing on its own.

Licence

This repository is public and proprietary, which is a pair that surprises people, so it is stated rather than implied: the source is visible so a customer can audit what runs on their own machine, and visibility grants no right to use it. The terms are in LICENSE.

Three third-party facts matter more here than the boilerplate, and each was resolved from the artefacts rather than from memory — see THIRD-PARTY-NOTICES.md:

• vendor-fastembed/ is a full vendored copy of the upstream crate (Apache-2.0) carrying one marked patch, kept byte-identical to upstream otherwise so the patch stays reviewable.
• One shipped dependency is MPL-2.0 — option-ext, four levels down through hf-hub. File-level copyleft, unmodified by us; its source stays available and the notice names it.
• DirectML, CUDA and cuDNN are used and never redistributed. They are vendor-licensed, and one cached copy in a registry, an image layer or an offline bundle would make us a redistributor. That is the reason this binary is built on the machine that runs it rather than shipped to it.

NOTICE carries the attribution that must travel with any build.