FaithfulVoice

Code and data for the paper "FaithfulVoice: Detecting Hallucinations in Voice-Driven Financial RAG with Low-Latency NLI Verification" (anonymous submission).

Real-time faithfulness verification for voice-driven financial RAG. A voice assistant answers questions about SEC 10-K/10-Q filings. It can speak hallucinated claims with the same confident intonation as true ones. FaithfulVoice inserts a lightweight NLI verifier (HHEM on GPU or DeBERTa-v3 ONNX Quantized on CPU) between the LLM output and the TTS engine, catching unsupported claims in ~33 ms (GPU) or ~788 ms (CPU) so they never reach the speaker.

audio → Moonshine STT
      → Qdrant retrieval  (BGE dense + BM25 sparse, top_k=8)
      → Llama-3.1-8B      (Modal L4)
      → Verifier NLI      (HHEM on GPU: ~33 ms | DeBERTa ONNX Quant on CPU: ~788 ms)
      → Kokoro TTS → audio

Table of Contents

1. Pre-Computed Results
2. Datasets Released
3. Installation
4. Reproduce Figures from Pre-Computed Results
5. Running Experiments from Scratch
6. Data Preparation (Reference)
7. Project Structure
8. Configuration
9. Tests
10. License

1. Pre-Computed Results

All experiment outputs are already in results/exp{1..8}/ — you do not need to re-run anything to inspect the data or regenerate figures.

• To regenerate figures only → see §4 Reproduce Figures from Pre-Computed Results.
• To fully re-execute the pipeline → see §5 Running Experiments from Scratch.

2. Datasets Released

As part of this project, we are releasing four datasets used for generating, evaluating, and human-annotating financial RAG hallucinations. All datasets are provided in .jsonl format.

2.1 Full Queries & Reference Answers

Location: data/queries_full_dataset/merged_queries.jsonl

• Size: 1,000 queries spanning 10 companies (AAPL, AMD, AMZN, GOOGL, INTC, JPM, META, MSFT, NVDA, TSLA) and 5 SEC filing domains (Revenue, Supply Chain, Risk Factors, Guidance, Litigation).
• Description: Raw questions, target filing metadata, and high-quality reference/gold answers generated by Claude 3.5 Sonnet.

{
  "ticker": "AAPL",
  "domain": "Revenue",
  "question_type": "Direct",
  "filing": "2024 10-K",
  "question": "What was Apple's total net sales for fiscal year 2024 (ending September 28, 2024)...",
  "answer": "Apple's total net sales for FY2024 were $391.035 billion, an increase of approximately 2%..."
}

2.2 The Audit Dataset (RAG Outputs)

Location: results/exp1/audit.jsonl

• Size: 1,000 generated RAG responses.
• Description: Retrieved SEC context, generated answer from Llama-3.1-8B, and extracted factual claims. Serves as the foundation for testing NLI verifiers.

2.3 Human Annotation Gold-Standard

Location: results/exp5/majority_labels.jsonl

• Size: 100-query subset (evaluating hundreds of individual claims).
• Description: Human-annotated labels (FAITHFUL / UNFAITHFUL) from 3 independent annotators. Includes Fleiss' kappa agreement metrics and final majority vote labels used to establish baseline metrics.

2.4 Teacher Model Distillation Dataset

Location: results/exp7/dataset/ai_train_val.jsonl

• Size: 900 annotated queries.
• Description: Claims audited and labeled by the MiniMax M3 model (teacher). Used to fine-tune and distill lightweight NLI verifiers.

3. Installation

3.1 Requirements

3.2 Install the Package

# 1. Create and activate a virtual environment
python -m venv venv
source venv/bin/activate          # Linux/macOS
# or: venv\Scripts\activate       # Windows

# 2. Install the faithfulvoice library + all dependencies
pip install -e ".[dev]"

pip install -e . makes import faithfulvoice work everywhere, including inside Modal containers that ship the package via add_local_python_source.

3.3 Configure Secrets

cp .env.example .env

Edit .env and fill in at minimum:

3.4 Authenticate with Modal

pip install modal
modal token new        # opens browser — one-time setup, free account

4. Reproduce Figures from Pre-Computed Results

Everything here runs locally, instantly — no GPU, no API calls required.

# Figure 2–3 — Generator comparison (overall bar + domain heatmap)
python experiments/e2_make_charts.py

# Figure 4 — Approach comparison (NLI vs GPT-judge vs RAGAS vs SelfCheckGPT)
python experiments/e4_make_charts.py

# Figures 5–9 — Synthesis: domain scores, claim types, archetypes, correlation, error analysis
python experiments/e8_analysis.py

Output PNGs are written to results/exp{2,4,8}/.

5. Running Experiments from Scratch

Run experiments in phase order — each phase depends on the outputs of the previous one.

Time estimates are wall-clock time for the full 1,000-query dataset.

Phase 0 — One-time Modal Infrastructure Setup

Uploads processed SEC chunk files and pre-downloads all model weights to Modal volumes (fv-data, fv-models, fv-vllm-cache) so GPU containers start instantly on subsequent runs. You need to do this once per Modal account.

# Step A — Upload chunks + queries to fv-data; pre-cache all models to fv-models.
#           Also pre-caches Llama-3.1-8B for E1.
modal run experiments/modal_setup.py

# Step B — Build the Qdrant vector index inside fv-data (CPU only, no GPU cost).
#           Must be done AFTER upload_data() in Step A completes.
modal run experiments/build_qdrant_index.py

modal_setup.py::main() runs five sub-steps in sequence:

1. upload_data — copies data/processed/*_chunks.jsonl + data/queries_full_dataset/merged_queries.jsonl into /data on fv-data
2. cache_embedding_models — downloads BGE-base + BM25 to fv-models
3. cache_verifier_models — downloads all 7 NLI verifiers to fv-models
4. cache_vllm_model — downloads Llama-3.1-8B (the E1 generator) to fv-models
5. cache_voice_models — downloads Moonshine STT + Kokoro TTS to fv-models

To reproduce E2 fully (all 6 vLLM generators), also cache them. Uncomment the models_e2 block near the bottom of modal_setup.py::main() before running Step A, or cache each model individually:

modal run experiments/modal_setup.py::cache_vllm_model --model-name google/gemma-4-12B-it
modal run experiments/modal_setup.py::cache_vllm_model --model-name Qwen/Qwen3-14B
modal run experiments/modal_setup.py::cache_vllm_model --model-name microsoft/phi-4
modal run experiments/modal_setup.py::cache_vllm_model --model-name mistralai/Mistral-Small-24B-Instruct-2501
modal run experiments/modal_setup.py::cache_vllm_model --model-name mistralai/Mistral-Nemo-Instruct-2407

These add ~30–60 min of download time; the 24B Mistral model requires an A100-80GB slot.

Note on SEC filings: The pre-chunked documents (data/processed/*_chunks.jsonl) are included in this repository, so you do not need to scrape SEC EDGAR or run the chunking scripts. Run modal_setup.py directly to upload them.

Phase 1 — E1: Generate the Audit Dataset

Depends on: Phase 0 (Modal volumes set up + Qdrant index built)
Output: results/exp1/audit.jsonl (1,000 entries)
Time: ~25–40 min on Modal L4

modal run experiments/e1_generate_dataset.py

# Optional: run on a small subset first to verify setup
modal run experiments/e1_generate_dataset.py --limit 10

Results are streamed back locally to results/exp1/audit.jsonl.

Phase 2A — E5: Human Annotation Preparation

Depends on: E2 (results/exp2/audit_llama8b.jsonl)
Output: results/exp5/majority_labels.jsonl
Time: < 1 min (local, no GPU) + annotation time

# Step 1 — Sample 100 queries for annotation
python experiments/e5_annotation_prep.py

Distribute results/exp5/sample_100.jsonl to annotators using the interface in experiments/human_eval/annotate.py. Once all three annotators have produced their CSV files:

# Step 2 — Compute Fleiss' κ and produce majority_labels.jsonl
python experiments/human_eval/agreement.py

Output: results/exp5/majority_labels.jsonl — the gold labels used by E3 and E8.

Phase 2B — E2: Generator Comparison

Depends on: Phase 0 (Modal volumes), E1 outputs for context reuse
Output: results/exp2/audit_<model>.jsonl for each generator
Time: ~25–40 min per model on Modal L4 (Mistral-Small 24B requires A100-80GB)

# vLLM models on Modal GPU (L4 / A100-80GB):
modal run experiments/e2_generator_compare.py --model llama8b
modal run experiments/e2_generator_compare.py --model gemma4_12b
modal run experiments/e2_generator_compare.py --model qwen3_14b
modal run experiments/e2_generator_compare.py --model phi4
modal run experiments/e2_generator_compare.py --model mistral_nemo_12b
modal run experiments/e2_generator_compare.py --model mistral_small_24b   # needs A100-80GB

# API models via Groq (no Modal needed; uses GROQ_API_KEY from .env):
python experiments/e2_groq_api.py --model llama70b
python experiments/e2_groq_api.py --model gptoss120b

The script checkpoints locally every 100 queries — safe to interrupt and resume.

# Generate charts after all models complete:
python experiments/e2_make_charts.py

Phase 3 — E3: Verifier Ablation

Depends on: E5 (results/exp5/majority_labels.jsonl)
Output: results/exp3/cpu/, results/exp3/gpu/ — per-verifier scores + latency
Time: ~15–30 min on Modal (CPU + T4 GPU)

# Run all 7 NLI verifiers on both CPU and GPU:
modal run experiments/e3_verifier_ablation.py

# Run only specific verifiers:
modal run experiments/e3_verifier_ablation.py --models deberta_small,deberta_base

# CPU-only or GPU-only:
modal run experiments/e3_verifier_ablation.py --mode cpu
modal run experiments/e3_verifier_ablation.py --mode gpu

Phase 3 — E4: Approach Comparison

Depends on: E5 (majority_labels.jsonl), E2 (audit_llama8b.jsonl)
Output: results/exp4/approach_comparison.*
Time: ~30–60 min (API calls, no GPU)

# GPT-judge (uses GROQ_API_KEY):
python experiments/e4_gpt_judge.py

# RAGAS faithfulness (uses GROQ_API_KEY):
python experiments/e4_ragas.py

# SelfCheckGPT (uses GROQ_API_KEY):
python experiments/e4_selfcheckgpt.py

# Aggregate all approaches and produce comparison table + chart:
python experiments/e4_approach_compare.py

# Generate the comparison chart:
python experiments/e4_make_charts.py

Phase 4 — E7: FinFaithVerifier Fine-Tune

Depends on: E5 (majority_labels.jsonl), E1 (audit.jsonl)
Output: fv-finetuned-models Modal volume; eval results in results/exp7/
Time: ~45–90 min on Modal T4 per base model

# Step 1 — Generate teacher labels via Nvidia NIM (MiniMax model)
python experiments/e7_label.py                       # label all 900 train/val queries
python experiments/e7_label.py --include-test        # include 100 human test queries
python experiments/e7_label.py --limit <N>           # dry-run/smoke test N items
python experiments/e7_label.py --repair-errors       # drop & re-label unparseable outputs
python experiments/e7_label.py --check               # read-only dataset sanity scan

# Note: You must upload the generated local file to Modal before splitting:
modal volume put fv-data results/exp7/ai_train_val.jsonl /exp7/ai_train_val.jsonl

# Step 2 — Create train/val splits from the distilled labels
modal run experiments/e7_finetune.py split

# Step 3 — Fine-tune on Modal T4 GPU (repeat for each base model)
modal run experiments/e7_finetune.py train --base-model cross-encoder/nli-deberta-v3-small
modal run experiments/e7_finetune.py train --base-model cross-encoder/nli-deberta-v3-base
modal run experiments/e7_finetune.py train --base-model vectara/hallucination_evaluation_model

# Step 4 — Export to ONNX (for latency experiments)
modal run experiments/e7_finetune.py export-onnx --base-model cross-encoder/nli-deberta-v3-small
modal run experiments/e7_finetune.py export-onnx --base-model cross-encoder/nli-deberta-v3-base

# Evaluate fine-tuned models vs baselines (local):
python experiments/e7_eval.py eval
python experiments/e7_eval.py eval --base-model cross-encoder/nli-deberta-v3-small

Phase 5 — E6: Latency Measurement

Depends on: E7 ONNX exports in fv-finetuned-models volume
Output: results/exp6/voice/
Time: ~30–45 min on Modal L4

# Full voice pipeline latency (Modal L4 GPU — STT + RAG + LLM + NLI + TTS):
modal run experiments/e6_voice_modal.py

Phase 6 — E8: Synthesis Analysis

Depends on: E1, E2, E5
Output: results/exp8/ — all heatmaps, correlation plots, error analysis
Time: < 1 min (local)

python experiments/e8_analysis.py

# Run a single analysis:
python experiments/e8_analysis.py --analyses domain_scores
python experiments/e8_analysis.py --analyses error_analysis

6. Data Preparation (Reference)

This section is for reference only. All required files are already included in this repository.

SEC Filings

The chunked 10-K and 10-Q filings for the 10 companies evaluated in the paper are provided in data/processed/. You do not need to download the raw HTML filings from SEC EDGAR or run the chunking scripts manually.

Audio Data

Audio files for evaluating the latency of the voice pipeline (E6) are included in data/audio/.

7. Project Structure

.
├── config.yaml                   # single source of truth (models, thresholds, paths)
├── requirements.txt
├── pyproject.toml                # pip install -e . → import faithfulvoice
├── docker-compose.yml            # local Qdrant (optional; Modal uses embedded Qdrant)
├── .env.example                  # copy to .env and fill in API keys
│
├── src/faithfulvoice/
│   ├── config.py                 # load_config() — reads config.yaml + .env
│   ├── pipeline.py               # retrieval → generation → verification (async)
│   ├── evalio.py                 # JSONL load/save helpers
│   ├── ingest/                   # HTML cleaner, chunker, Qdrant upserter
│   ├── retrieval/                # BGE dense + BM25 sparse hybrid retrieval
│   ├── verification/             # verify_single(), compute_metrics()
│   ├── voice/                    # Moonshine STT, Kokoro TTS, VoicePipeline
│   └── modal/                    # Modal app: images, volumes, VLLMService
│
├── experiments/
│   ├── modal_setup.py            # ONE-TIME: upload data + cache models to Modal
│   ├── build_qdrant_index.py     # ONE-TIME: build vector index on Modal volume
│   ├── e1_generate_dataset.py    # Phase 1  → results/exp1/audit.jsonl
│   ├── e2_generator_compare.py   # Phase 2B → results/exp2/audit_<model>.jsonl
│   ├── e2_groq_api.py            # Phase 2B → results/exp2/ (Groq models)
│   ├── e2_make_charts.py         # Figures 2–3
│   ├── e3_verifier_ablation.py   # Phase 3  → results/exp3/
│   ├── e4_approach_compare.py    # Phase 3  → results/exp4/
│   ├── e4_gpt_judge.py
│   ├── e4_ragas.py
│   ├── e4_selfcheckgpt.py
│   ├── e4_make_charts.py         # Figure 4
│   ├── e5_annotation_prep.py     # Phase 2A → results/exp5/sample_100.jsonl
│   ├── e6_voice_modal.py         # Phase 5  → results/exp6/voice/
│   ├── e7_finetune.py            # Phase 4  → fv-finetuned-models volume
│   ├── e7_eval.py                # Phase 4  → results/exp7/eval/
│   ├── e7_label.py               # Phase 4  → distillation labels
│   ├── e8_analysis.py            # Phase 6  → results/exp8/
│   └── human_eval/
│       ├── annotate.py           # annotation UI
│       ├── auto_annotate.py      # model-assisted pre-labelling
│       └── agreement.py          # Fleiss' κ + majority vote → majority_labels.jsonl
│
├── results/
│   ├── exp1/   audit.jsonl              (1000 RAG responses — main dataset)
│   ├── exp2/   audit_<model>.jsonl × 8  (generator comparison outputs)
│   ├── exp3/   cpu/ + gpu/              (verifier ablation)
│   ├── exp4/   approach comparison
│   ├── exp5/   majority_labels.jsonl    (100-item gold set, κ = 0.74)
│   ├── exp6/   voice/                   (latency results)
│   ├── exp7/   eval/ + threshold/       (fine-tune eval)
│   └── exp8/   all synthesis figures
│
└── data/
    ├── audio/                    audio files for E6 latency evaluation
    ├── queries/                  per-domain query files
    └── queries_full_dataset/     merged_queries.jsonl  (1000 queries)

8. Configuration

All model names, thresholds, and retrieval parameters live in config.yaml. Import constants instead of hardcoding values:

from faithfulvoice.config import TOP_K, THRESHOLD, VERIFIER_MODEL, GENERATOR_MODEL

9. Tests

ruff check .

10. License

Code: MIT. Pre-computed results and the query set: CC BY 4.0.
Raw SEC filings remain subject to SEC EDGAR terms of use.