NLA Pipeline

Runs sentences through Qwen2.5-7B-Instruct-AWQ (GPU 0), extracts last-token hidden states at layer 20, then feeds them to the NLA Activation Verbalizer (GPU 1) to get natural-language "thoughts". Results are written to a TSV.

---

Files

---

Model precision

run_rq2.py loads the full Qwen/Qwen2.5-7B-Instruct in bfloat16 (~14 GB VRAM) for faithful activations. The legacy legacy/run_nla.py uses the int4 AWQ build (Qwen/Qwen2.5-7B-Instruct-AWQ, ~4.5 GB) for fitting an 11 GB RTX 2080 Ti. The AV model runs separately on GPU 1 via SGLang.

---

Running with Docker (recommended)

Prerequisites

• Docker + docker compose v2
• NVIDIA Container Toolkit installed on the host

Basic run (defaults: test.txt → result.txt)

docker compose up --build

result.txt appears in the project directory when the pipeline finishes.

Custom input / output

INPUT_FILE=data/my_sentences.txt OUTPUT_FILE=data/my_results.txt docker compose up --build

Paths are relative to the project directory (mounted at /app/data inside the container).

---

Running directly (no Docker)

1. Install dependencies

# legacy/run_nla.py (AWQ):
pip install torch transformers accelerate autoawq httpx numpy "sglang[all]>=0.5.6"

# run_rq2.py additionally needs (for the vendored NLA client + embedder):
pip install orjson pyyaml safetensors sentence-transformers

2. Start the AV SGLang server on GPU 1

CUDA_VISIBLE_DEVICES=1 python -m sglang.launch_server \
    --model-path kitft/nla-qwen2.5-7b-L20-av \
    --port 30000 \
    --disable-radix-cache \
    --dtype bfloat16

Wait for Server is ready in the logs.

3. Run the pipeline

For RQ2, see the RQ2 section below.

The legacy basic pipeline:

# defaults
python legacy/run_nla.py

# custom files
python legacy/run_nla.py --input my_sentences.txt --output my_results.txt

The AV_SGLANG_URL env var controls the server address (default: http://localhost:30000).

---

Input format

Plain TSV, one sentence per line, header optional:

index	sentence
0	The quick brown fox jumps over the lazy dog.
1	Artificial intelligence is reshaping how humans interact with machines.

---

Output format

index	activation_l2	thought
0	142.3801	The token is processing a sentence about a fast animal performing an action...
1	138.9204	The representation encodes concepts related to technology and human interaction...

---

GPU assignment

---

RQ2: NLA Prediction Stability (run_rq2.py)

Hypothesis: NLA's stability in describing activations reflects the target model's uncertainty — and therefore its propensity to hallucinate.

For each example in the HalluLens dataset, run_rq2.py extracts Qwen's layer-20 activations and computes three stability measures, then joins each row with the dataset's hallucination labels so the signals can be correlated against ground truth offline. Verbalization goes through the vendored official NLA client (nla_inference.py, from kitft/nla-inference, Apache-2.0). That client reads the checkpoint's nla_meta.yaml and handles the trained prompt template, the neighbor-verified injection position, the injection scale, and the architecture embed scale automatically — so none of that is hardcoded here. run_rq2.py subclasses it (NLAClientLP) only to add logprobs for M2.

Setup

The NLA client needs the AV checkpoint on local disk (for its nla_meta.yaml sidecar), and the SGLang server must serve that same local dir. Download it once:

huggingface-cli download kitft/nla-qwen2.5-7b-L20-av \
    --local-dir ./nla-qwen2.5-7b-L20-av

Start the AV SGLang server from that dir (--disable-radix-cache is required for input_embeds; logprobs are requested per-call by NLAClientLP):

CUDA_VISIBLE_DEVICES=1 python -m sglang.launch_server \
    --model-path ./nla-qwen2.5-7b-L20-av \
    --port 30000 --disable-radix-cache --dtype bfloat16

Install the extra deps the client + embedder need:

pip install sentence-transformers orjson pyyaml safetensors

run_rq2.py finds the checkpoint via AV_CHECKPOINT_DIR (default ./nla-qwen2.5-7b-L20-av) and the server via AV_SGLANG_URL (default http://localhost:30000).

Run

The --dataset default already points at the bundled Balanced Hallulens Dataset/balanced_dataset.jsonl (400 examples, balanced 200 hallucination / 200 correct, inline labels), so the smoke test needs no paths.

Smoke test — first 10 examples, fewer samples / shorter sequences, finishes in a couple of minutes; sanity-checks the GPU + AV server wiring end to end:

python run_rq2.py --limit 10 --k-samples 4 --max-seq-tokens 64

Sanity-check the injection. On startup the client prints a banner like:

[NLAClient] nla-qwen2.5-7b-L20-av: d_model=3584 inj_scale=150.0 embed_scale=1.00 inj_char='㈎'(id=149705)

If inj_scale/embed_scale/inj_char look right and the first printed thought reads like 2-3 descriptive snippets (not gibberish or stray CJK), the activation injection is working. All-CJK or English-describing-a-CJK-char output means injection failed (template/tokenizer drift) — the client also raises loud asserts for most such cases.

Check that rq2_metrics.jsonl has 10 rows with non-trivial M1/M2/M3 numbers, then do the full run.

Full run — all 400 examples with the paper-faithful settings:

python run_rq2.py \
    --dataset "Balanced Hallulens Dataset/balanced_dataset.jsonl" \
    --output  rq2_metrics.jsonl \
    --k-samples 8 --max-seq-tokens 512 --temperature 1.0

The full run verbalizes up to 512 token positions per example for M1, so it makes thousands of AV calls. Expect it to take a while; start with the smoke test. To trade coverage for speed, lower --max-seq-tokens (e.g. 128).

--temperature (default 1.0) applies to all three measures. T=1 matches the AV's trained sampling distribution (the NLA paper samples at T=1), which keeps M2 perplexity comparable to the paper. It must stay > 0 because M3 resamples the same activation and measures disagreement — at T=0 the AV is deterministic, so all k samples are identical (cosine 1.0, entropy 0) and M3 carries no signal.

Add --use-chat-template to wrap each prompt with Qwen's chat template before extracting activations (matches how the answers in the dataset were generated).

Output (rq2_metrics.jsonl)

One JSON object per example, e.g.:

{
  "idx": 1, "prompt": "What local name did the Philippine ... assign?",
  "answer": "...", "label": "hallucination", "source": "original",
  "is_hallucinated": true, "is_abstaining": false, "hallucinated_strict": true,
  "cross_token_mean_cosine": 0.41, "cross_token_semantic_entropy": 1.79,
  "mean_logp": -1.83, "perplexity": 6.23,
  "sampling_mean_cosine": 0.52, "sampling_semantic_entropy": 1.10,
  "last_token_thought": "...", "thoughts_per_token": ["..."], "samples": ["..."]
}

hallucinated_strict = is_hallucinated AND NOT is_abstaining. The balanced dataset has no abstentions, so this equals is_hallucinated; the formula is kept for robustness. Use it as the prediction target. The expected finding: hallucinated examples show lower cross-token/sampling cosine and higher entropy/perplexity than correct ones.

The AV text outputs (thoughts_per_token, last_token_thought, samples) are saved inline. The raw last-token activation vectors are written to a sidecar rq2_metrics.last_act.npy — a [N, d_model] float32 array where row last_act_row matches each record (~5.7 MB for 400×3584). Load with np.load(...) if you want to re-verbalize or probe activations without re-running the GPU extraction.