Tier 0 V0 — IF-SFT LLaMA2-7B — AWQ INT4 — Fingerprint Verification

Implementation of TIER0_IF_SFT_LLAMA2_AWQ4_CHARACTERIZATION.md. Characterization only (not an adaptive attack): does standard AWQ4 PTQ weaken/destroy the IF-SFT ownership fingerprint in cnut1648/LLaMA2-7B-fingerprinted-SFT?

AWQ backend: this project's own implementation, not AutoAWQ

src/quantize_awq.py quantizes using src/quantization/awq.py + src/quantization/state.py (AWQConfig/AWQQuantizerXL), vendored verbatim from the user's own "smart-flip" research repo, run with post_correction=None (that repo's own "awq_raw" variant — plain AWQ, no bias-correction/smart-flip adaptive post-processing, matching spec section 26's "no adaptive/watermark-aware quantization"). This is a deliberate choice, not AutoAWQ and not mit-han-lab/llm-awq.

The Drive search/rounding code remains unchanged. An additive finalizer in src/quantization/packed_awq.py captures its final integer state, packs two INT4 nibbles per byte, swaps transformer nn.Linear layers for PackedAWQLinear, and provides a standalone reload path. Two details differ from AutoAWQ:

• No CUDA-only kernel. The vendored implementation does groupwise affine int4 quantization in plain PyTorch (search a per-channel scale via search_best_scale, round through an int4 grid, dequantize) — no compiled CUDA extension. It runs on CPU too, just slowly for a 7B model. Unlike AutoAWQ, quantization is not architecturally blocked without a GPU.
• Real packed artifact, emulated compute path. Saved transformer qweights are uint8 buffers containing two 4-bit values per byte, with group-128 scales and zero-points. Validation requires a packed manifest, packed modules after reload, and substantial size reduction; the legacy round-trip-FP artifact now fails hard. At forward time each layer dequantizes and caches its quantized weights because this backend has no fused INT4 GEMM kernel. Thus storage/reload satisfies the spec's real-artifact gate, while runtime memory/speed must be reported as emulated W4A16 rather than claimed as fused INT4 deployment performance.

Environment status — read before running

This project was built and its data-only steps executed on a MacBook Pro M1 (16GB RAM, no NVIDIA/CUDA GPU). Loading the actual 7B checkpoint (src/verify_fingerprint.py, src/eval_wikitext.py, src/quantize_awq.py, src/run_experiment.py) needs a GPU or a lot of RAM for practical CPU inference — that has not been attempted on this machine, since even loading (let alone quantizing) a 7B model is impractical at 16GB RAM. It is not blocked by a hard CUDA-only dependency the way AutoAWQ would have been, though — see above.

What has actually been run and verified on this machine (no GPU needed):

• artifacts/fingerprints/if_sft_llama2_keys.json — the real fingerprint key/target pairs, extracted from the author-published publish.jsonl (cnut1648/LLM-fingerprinted-SFT, file NousResearch/Llama-2-7b-hf/chat_epoch_3_lr_2e-5_bsz_64/publish.jsonl). See Important deviation from the spec below: this yields 8 keys, not 10.
• artifacts/calibration/pileval_seed42_128x512.jsonl — 128 real Pile-val documents, seed 42, tokenized/truncated to ≤512 tokens with the fingerprinted model's tokenizer, verified to contain zero fingerprint prompt/target leakage.
• The full tests/ suite (37 tests, including the target-NLL masking logic, the exact-FSR scoring logic, prompt-extraction/dedup logic, calibration-leakage detection, nibble packing, save/reload parity, and real-artifact validation) — all pass with plain pytest, no GPU, no model weights.
• src.quantization (the vendored AWQ backend) imports and constructs AWQConfig cleanly in this environment — the numerical logic itself needs no GPU to at least load; it has not been run against the real 7B checkpoint here.

What has not been run here (needs a machine that can hold the 7B model): verify_fingerprint.py, eval_wikitext.py, eval_lm_eval.py, quantize_awq.py against the real checkpoint, validate_quantized_artifact.py --load-and-inspect-layers, and the full run_experiment.py orchestration.

Important deviation from the spec

Spec section 2.3/6.1 says to prefer the exact keys from the author-provided publish.jsonl over reconstructing/guessing them, and section 9 says a less-than-expected count must be investigated and reported, not silently forced to match. The official publish.jsonl for this exact checkpoint contains only 8 unique prompts whose ground-truth label is "Based on my fingerprint, the message is: ハリネズミ" (not 10, the paper's typical count). config/if_sft_llama2_awq4.yaml and the keys artifact reflect 8 as the real, verified count. artifacts/fingerprints/if_sft_llama2_keys.provenance.json records this exactly (8 extracted vs 10 expected) for the report to surface, per spec section 9.

Layout

if_awq_tier0/
├── configs/if_sft_llama2_awq4.yaml
├── src/
│   ├── quantization/        # vendored AWQ backend (awq.py, state.py) - see above
│   └── ...                  # rest of the pipeline modules (spec section 20-21)
├── tests/                   # 37 tests, runnable without GPU/model weights
├── artifacts/
│   ├── fingerprints/if_sft_llama2_keys.json         (real, 8 keys)
│   └── calibration/pileval_seed42_128x512.jsonl     (real, 128 samples)
├── outputs/<run_name>/      # created by run_experiment.py
└── quantized_models/        # created by quantize_awq.py

Setup

Easiest path is ./run_all.sh (see below) — it handles all of this automatically. Manually:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

This repo's own .venv (used to build/test everything above) has the same requirements.txt installed minus datasets/lm-eval (only needed for calibration dataset streaming and the optional 0-shot task suite against a real model, neither of which has been run here).

Running the parts that work without a GPU

# already run, artifacts committed under artifacts/ - re-run to reproduce:
python -m src.download_assets --model llama2-7b --official-outputs-dir external/official_outputs
python -m src.prepare_fingerprint_keys \
  --publish-jsonl external/llama2_7b_publish.jsonl \
  --expected-num-keys 8 \
  --output artifacts/fingerprints/if_sft_llama2_keys.json

python -m src.prepare_calibration \
  --num-samples 128 --max-seq-len 512 --seed 42 \
  --output artifacts/calibration/pileval_seed42_128x512.jsonl

pytest -v

Running the full pipeline

./run_all.sh                          # auto-detects cuda vs cpu
./run_all.sh --device cpu             # force CPU (vendored AWQ backend supports it, slowly)
python -m src.make_report --run-dir outputs/if_sft_llama2_awq4_tier0_packed

run_experiment.py will: load the source checkpoint, verify fingerprint FSR + soft metrics (hard gate — stops before quantizing if FSR_exact_source < 100% unless --allow-partial-baseline is passed, per spec section 9), evaluate WikiText-2 PPL, quantize with the vendored AWQ backend (W4A16, group=128, plain/raw variant), reload the saved checkpoint from disk, require its packed INT4 manifest, inspect real PackedAWQLinear modules and assert strong disk reduction, re-run the identical fingerprint + PPL evaluation, and write outputs/<run>/summary.json + (via make_report.py) report.md.

Scope (spec section 26/29)

One official fingerprinted checkpoint, one PTQ method (AWQ, plain/raw variant), one bit-width (W4A16 INT4/group128). No adaptive/watermark-aware quantization (i.e. no bias-correction/smart-flip post-correction from the vendored repo), no fine-tuning, no GPTQ/AWQ comparison. Results only support claims of the form "under this IF-SFT LLaMA2-7B checkpoint and this standard AWQ4 setting, ...".