Sparse-Aware Computing-in-Memory Acceleration via Lookup Table for Edge Multimodal LLM — hardware/software co-design of the Progressive Token Elimination Unit (PTEU), the token-pruning block on the SALUTE logic die.
hardware/ Synthesizable RTL for the PTEU + self-checking testbenches
syn/ Nangate45 synthesis flow (yosys+ABC) + the evidence JSONs
tools/ Icarus Verilog installer (see hardware/README.md)
vectors/ Fixed-point stimulus for the testbenches (gitignored)
software/ Everything Python
pteu/ THE FROZEN MODEL — reference implementation, tests, benchmarks
baselines/ Reimplementations of prior work (PSA, HeatViT, MHTP) — see Status
stimulus/ Real per-block PTEU inputs (.npz) + .hex vectors (gitignored)
archive/ Superseded scripts and dead results, kept for provenance only
cache/ HuggingFace dataset/model cache (gitignored, regenerable, ~525 MB)
paper/ IEEE TCAD manuscript (Overleaf export)
refs/ The two papers the PTEU builds on: HeatViT and PSA
The one rule: software/pteu/pteu_reference.py is the single source of
truth. The paper's Algorithm 1, the RTL, and every reported number must
agree with it. If they disagree, the model is right and the other one is a bug.
Start with HANDOFF.md — its "State in 60 seconds" is the orientation, and its
"Verify state" block is the command list for reproducing every claim below.
cd software/pteu
python3 -m pytest test_pteu_reference.py -q # 109 tests: RTL equivalence + properties
python3 pteu_reference.py # self-check against the RTL golden vector
python3 bench_cuboid.py # the Section IV numbers, cuboid-scored
Requires numpy and pytest. Nothing in software/pteu/ needs torch or a GPU,
but the measurement scripts (mm_accuracy.py, lb_accuracy.py,
gen_stimulus.py) do — use /home/rnafde/venv/bin/python3.
Last verified end to end on 2026-08-31.
| Area | State |
|---|---|
| PTEU algorithm | Frozen and tested, 109 tests green. One-sided error against exact coverage is proven and verified on real data. |
| RTL | Bit-exact against the reference on 80 real prefills — 15,388 checks, 0 mismatches, across four testbenches. The three defects this table used to list are fixed or retracted. |
| Silicon | Synthesized on Nangate45 (yosys+ABC, no P&R). The streaming selector is flat in context: 9.3× less area than the buffered build at N=512, 8.6× lower leakage. |
| Evaluation | Multimodal and text. LLaVA-1.5-7B and Qwen2.5-VL-3B on ScienceQA-IMG (300 questions each); Llama-3.1-8B on LongBench (750 samples), the benchmark that can separate selection rules. |
| Manuscript | All PTEU sections written — §II-A, §III, §IV-F-1, the §V rows, two figures, the contribution bullet and the abstract's PTEU sentences. Merged with the co-authors' 2026-09-01 draft into paper_updated/, which builds at 20 pages. The repo copy still does not compile, for four co-author reasons: two uncommitted figure PDFs and two unescaped & in Ref.bib. See COAUTHOR_HANDBACK.md. |
baselines/ | F12 fixed 2026-08-26. psa_pruning.py and mhtp_pruning.py scored each block with a block-local softmax whose sum is identically 1.0, so both stopping rules degenerated to keeping ⌈ε·n⌉ blocks regardless of the data — the root cause of the unquotable results deleted under F7. Both now return the unnormalised score sum, PSA prunes 8× at ε=0.70 on concentrated attention, and test_baselines.py (10 tests) pins it. No paper number moved: nothing imports this directory. evaluation.py end-to-end and heatvit.py accuracy remain unvalidated. |
software/stimulus/ and hardware/vectors/ are matched by the *.hex /
*.npz rules in .gitignore, so they are not tracked. The real-data
tests skip themselves when absent, and an RTL testbench with no vectors
reads nothing and trivially "passes" — regenerate first, with
python3 software/pteu/gen_stimulus.py --hex (needs a GPU and gated access
to Llama-3.1-8B) then python3 software/pteu/export_rtl_vectors.py.software/archive/ is provenance, not code. Nothing there is maintained and
several files are known to be wrong — read software/archive/README.md first.
In particular, do not regenerate stimulus from legacy_scripts/; the frozen
generator is software/pteu/gen_stimulus.py.software/pteu/bench_pteu.py is legacy. It reads one number per block, which
forces cs = mass and hands the selector a free perfect ranking. The Section
IV numbers come from bench_cuboid.py.Sparse-Aware Computing-in-Memory Acceleration via Lookup Table for Edge Multimodal LLM — hardware/software co-design of the Progressive Token Elimination Unit (PTEU), the token-pruning block on the SALUTE logic die.
hardware/ Synthesizable RTL for the PTEU + self-checking testbenches
syn/ Nangate45 synthesis flow (yosys+ABC) + the evidence JSONs
tools/ Icarus Verilog installer (see hardware/README.md)
vectors/ Fixed-point stimulus for the testbenches (gitignored)
software/ Everything Python
pteu/ THE FROZEN MODEL — reference implementation, tests, benchmarks
baselines/ Reimplementations of prior work (PSA, HeatViT, MHTP) — see Status
stimulus/ Real per-block PTEU inputs (.npz) + .hex vectors (gitignored)
archive/ Superseded scripts and dead results, kept for provenance only
cache/ HuggingFace dataset/model cache (gitignored, regenerable, ~525 MB)
paper/ IEEE TCAD manuscript (Overleaf export)
refs/ The two papers the PTEU builds on: HeatViT and PSA
The one rule: software/pteu/pteu_reference.py is the single source of
truth. The paper's Algorithm 1, the RTL, and every reported number must
agree with it. If they disagree, the model is right and the other one is a bug.
Start with HANDOFF.md — its "State in 60 seconds" is the orientation, and its
"Verify state" block is the command list for reproducing every claim below.
cd software/pteu
python3 -m pytest test_pteu_reference.py -q # 109 tests: RTL equivalence + properties
python3 pteu_reference.py # self-check against the RTL golden vector
python3 bench_cuboid.py # the Section IV numbers, cuboid-scored
Requires numpy and pytest. Nothing in software/pteu/ needs torch or a GPU,
but the measurement scripts (mm_accuracy.py, lb_accuracy.py,
gen_stimulus.py) do — use /home/rnafde/venv/bin/python3.
Last verified end to end on 2026-08-31.
| Area | State |
|---|---|
| PTEU algorithm | Frozen and tested, 109 tests green. One-sided error against exact coverage is proven and verified on real data. |
| RTL | Bit-exact against the reference on 80 real prefills — 15,388 checks, 0 mismatches, across four testbenches. The three defects this table used to list are fixed or retracted. |
| Silicon | Synthesized on Nangate45 (yosys+ABC, no P&R). The streaming selector is flat in context: 9.3× less area than the buffered build at N=512, 8.6× lower leakage. |
| Evaluation | Multimodal and text. LLaVA-1.5-7B and Qwen2.5-VL-3B on ScienceQA-IMG (300 questions each); Llama-3.1-8B on LongBench (750 samples), the benchmark that can separate selection rules. |
| Manuscript | All PTEU sections written — §II-A, §III, §IV-F-1, the §V rows, two figures, the contribution bullet and the abstract's PTEU sentences. Merged with the co-authors' 2026-09-01 draft into paper_updated/, which builds at 20 pages. The repo copy still does not compile, for four co-author reasons: two uncommitted figure PDFs and two unescaped & in Ref.bib. See COAUTHOR_HANDBACK.md. |
baselines/ | F12 fixed 2026-08-26. psa_pruning.py and mhtp_pruning.py scored each block with a block-local softmax whose sum is identically 1.0, so both stopping rules degenerated to keeping ⌈ε·n⌉ blocks regardless of the data — the root cause of the unquotable results deleted under F7. Both now return the unnormalised score sum, PSA prunes 8× at ε=0.70 on concentrated attention, and test_baselines.py (10 tests) pins it. No paper number moved: nothing imports this directory. evaluation.py end-to-end and heatvit.py accuracy remain unvalidated. |
software/stimulus/ and hardware/vectors/ are matched by the *.hex /
*.npz rules in .gitignore, so they are not tracked. The real-data
tests skip themselves when absent, and an RTL testbench with no vectors
reads nothing and trivially "passes" — regenerate first, with
python3 software/pteu/gen_stimulus.py --hex (needs a GPU and gated access
to Llama-3.1-8B) then python3 software/pteu/export_rtl_vectors.py.software/archive/ is provenance, not code. Nothing there is maintained and
several files are known to be wrong — read software/archive/README.md first.
In particular, do not regenerate stimulus from legacy_scripts/; the frozen
generator is software/pteu/gen_stimulus.py.software/pteu/bench_pteu.py is legacy. It reads one number per block, which
forces cs = mass and hands the selector a free perfect ranking. The Section
IV numbers come from bench_cuboid.py.