Voice-RAG on MSMARCO-XI (ElevenLabs STT)

A voice-enabled, multilingual Retrieval-Augmented Generation pipeline: speech → ElevenLabs STT → hybrid retrieval (dense + BM25 + RRF) → grounded answer generation, with a full harness (typed stages, deadlines, retries, degradation) and three guardrails. Answers questions spoken in any of the 14 Indic languages covered by MSMARCO-XI (Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Sanskrit, Tamil, Telugu, Urdu).

Audio ─► ElevenLabs STT (scribe_v2 / flash_v2.5)
          │ transcript + language + confidence
          ▼
      Guardrail 1: input safety + language gate
          ▼
      Query embed (multilingual-e5-small, ONNX int8) + BM25
          │                    │
          └── RRF fusion ──────┘
          ▼
      Metadata-aware re-rank (language / popularity / query-type / numeric)
          ▼
      Guardrail 2: relevance gate (no topical match → abstain)
          ▼
      Generation harness: Groq qwen/qwen3.6-27b (structured JSON, retries,
      schema repair, refusal honored) ─ or extractive fast-path
      (deadline degradation)
          ▼
      Guardrail 3: groundedness (lexical + semantic vs. context; citations)
          ▼
      answer | low_confidence | abstain (+ reason code)

Measured results (this machine, warm pipeline)

Latency — target: post-STT path < 200 ms

The 200 ms budget applies to everything after transcription (embed → retrieve → generate → guardrail). It is met with ~7× headroom even at P100. ElevenLabs STT (network-bound) is measured and reported separately — the STT stage is never part of the 200 ms claim.

Offline evaluation (140 gold-labeled queries, 14 languages)

The hybrid path beats dense-only by a wide margin at rank 1–5 (the RRF fusion and language/popularity boosts do the heavy lifting), and metadata boosts add ~2 pp recall@5. Adding the LLM stage improves answer quality (ROUGE-L 0.228 → 0.287, cosine 0.872 → 0.896 on the 22 of 40 queries it answered) and — critically — lifts off-topic abstention from 2/7 to 7/7: the model's explicit can_answer=false refusal is now respected and turns into an abstain instead of falling through to an extractive guess.

Chunking (why it's not one naive strategy)

Three strategies are implemented, indexed, and compared (data/corpus/chunks_*.parquet):

1. native — each MS MARCO passage is one chunk. Passages are already short, self-contained units; this is the strong baseline.
2. sentence_overlap — long passages split into 3-sentence windows with 1-sentence overlap, so answers spanning sentence boundaries stay intact and no answer is lost at a hard cut.
3. semantic — data-driven boundaries: sentences are embedded and a cut is inserted where adjacent-sentence cosine similarity drops below a threshold (topical shift); undersized fragments are merged back.

Every chunk carries metadata (is_selected_count, query_types, sentence spans) used by the retrieval re-ranker. The eval report shows the strategies are close on this corpus (native slightly ahead), which is a real finding — and the code supports adding more strategies via the Chunker registry.

Retrieval

• Dense: intfloat/multilingual-e5-small (384-d, ONNX int8, torch fallback), FAISS IndexFlatIP over 30,100 chunk embeddings (HNSW kicks in above 500k vectors).
• Lexical: numpy-vectorized Okapi BM25 (per-term posting arrays — ~2 ms per query over 30k docs, vs 100+ ms with rank_bm25's Python loop).
• Fusion: Reciprocal Rank Fusion, then metadata-aware boosts: language match, gold-passage popularity (is_selected_count), query type, and numeric-token overlap for NUMERIC queries.

Guardrails (knows when not to answer)

1. Input safety — per-language lexicons flag violence/self-harm/bomb etc. (verified: 4/4 abstain on adversarial inputs).
2. Relevance gate — if the best dense similarity is below threshold, or the extractive answer has only stopword/incidental overlap (content tokens only, multi-script stopword list), the system abstains instead of fabricating.
3. LLM refusal (can_answer=false) — the generator is instructed to refuse when the retrieved context does not answer the question; the pipeline now respects that refusal and abstains (verified live: 7/7 off-topic queries abstained, vs 2/7 with extractive-only gates).
4. Groundedness — LLM answers are checked for lexical containment + semantic similarity against the retrieved context; ungrounded answers are replaced by the extractive answer or abstained.

Harness

Pipeline.run_text / run_audio orchestrate typed stages, each recording its own duration into a trace. A global latency budget acts as a deadline: full (LLM + groundedness) → extractive (no LLM) → abstain. The Groq client has retries with backoff (honouring Retry-After), strict JSON-schema parsing with self-repair (JSON → {...} extraction → YAML key:value), a client-side tokens-per-minute throttle so long eval loops stay under the provider limit, a circuit breaker that disables only on permanent endpoint failures (parse errors degrade per-query instead of killing the client), and an available check (runs extractive-only without a key). Every request terminates with a structured PipelineResult and a decision.

Setup

py -m venv .venv
.venv/Scripts/python -m pip install -r requirements.txt
.venv/Scripts/python -m pip install -e .          # optional, for `python -m voice_rag`

# .env
ELEVENLABS_API_KEY=sk_...                          # required for speech input
# LLM answers: set ONE of these (OpenAI-compatible endpoint; see config.yaml)
GROQ_API_KEY=gsk_...                               # Groq (default model: qwen/qwen3.6-27b)
XAI_API_KEY=xai-...                                # xAI grok-3-mini

The LLM client is provider-agnostic (Groq or xAI). It has a circuit breaker: after a few consecutive permanent failures (bad/expired/no-credit keys) it self-disables for the process and every request degrades instantly to the extractive path instead of paying a dead endpoint. Transient errors (429/5xx) retry with backoff; permanent errors (401/403/402) fail fast. Verified by the mocked-transport suite in tests/test_llm_harness.py.

Requirements: Python 3.11+, Windows/Linux, 8 GB RAM (CPU-only).

Build the data + index (already done in this checkout)

py scripts/fetch_dataset.py        # streams 14 validation configs, samples per language
py scripts/make_small_corpus.py    # gold-biased ~30k-passage corpus
py scripts/build_corpus.py         # produces chunks_{native,sentence_overlap,semantic}.parquet
py scripts/build_index.py --strategy native   # embeddings + FAISS + BM25 (resumable)

Usage

# text question
py src/voice_rag/cli.py ask-text "पोम क्लेमेंटीफ का जन्म कहाँ हुआ था?" --language hi
# audio file
py src/voice_rag/cli.py ask-file data/audio/bench_hi_0.mp3 --language hi
# microphone
py src/voice_rag/cli.py mic --seconds 5

# latency report (P50/P70/P95/P99/P100) → data/results/latency_report.md
py src/voice_rag/cli.py bench --n 80 --n-audio 20
# offline eval → data/results/eval_report.md
py src/voice_rag/cli.py eval --max-queries 150
# eval with LLM answers (adds answer_llm_* metrics; needs a funded LLM key)
py src/voice_rag/cli.py eval --max-queries 150 --with-llm

# API server
py -m uvicorn voice_rag.server:app --app-dir src --host 127.0.0.1 --port 8000
#   GET  /health
#   POST /ask_text   {"text": "...", "language": "hi"}
#   POST /ask        multipart file upload (audio) + optional language field

Tests

py -m pytest tests/ -q

Known limitations

• Reasoning-model truncation — qwen emits a <think> block before its JSON; on long definition-style queries the block can exhaust max_tokens and the JSON never appears. The harness repairs JSON from {...}/YAML output, skips the failed query (extractive fallback) instead of tripping the breaker, and config.yaml trades max_tokens/context size to keep the truncation rate low (18/40 queries in the last run were refused/skipped, and the 22 answered queries still beat extractive on both ROUGE-L and cosine).
• Provider daily quotas — the Groq on-demand tier throttles by tokens/minute and tokens/day; compound models (groq/compound-mini, groq/compound) bill the underlying 70B/120B model and exhaust the daily quota quickly. The client throttles to stay under the limit, but a big --with-llm eval needs the model's daily budget. qwen/qwen3.6-27b has the most headroom on this key.
• or/pa/as edge-tts voices are unsupported, so the audio benchmark skips them (STT itself still works for those languages).
• Corpus is a ~30k gold-biased sample (full MS MARCO-XI is ~10M examples); the build scripts stream and scale to the full corpus.