Open research toward model-specific mask-ROM LLM inference silicon: analytical studies, an ABI 3.0 compiler/runtime and cycle model, synthesizable RTL, and open-PDK physical evidence — gated against itself by a 19-rung board that is deliberately mostly red.
Verilog
4
24,291 commits
updated Oct 19, 2026
Open research toward instant, affordable AI inference.
What works today · What you can reproduce · How to contribute · Why speed matters · Plain-English overview · Documentation · Specification · Results
More than 10,000 tokens per second per user is now a publicly reported silicon result. Taalas reports 16,960 tokens/s per user for its fabricated HC1 accelerator running Llama 3.1 8B at batch one. That result changes the conversation: model-specific inference is no longer only a paper architecture. It is a credible path toward AI systems that respond at interactive timescales.
OpenTallas asks what it would take to make that architectural direction open, inspectable, and extensible to larger models and longer contexts. The repository connects model accounting, architecture simulation, a shared compiler/runtime ABI, public-reference RTL, and open-PDK circuit experiments in one evidence chain. It is independent of Taalas and does not claim to reproduce HC1.
For implementation work, use the revised chip architecture, checked resource budgets, and review handoff. These correct physical placement and service assumptions; the historical analytical studies below remain separate from a correctness-qualified chip throughput result.
| Signal | Result | Evidence class |
|---|---|---|
| Public model-specific silicon reference | 16,960 tokens/s per user on Taalas HC1 | Fabricated product; first-party Taalas run, publicly documented, not an independent benchmark |
| OpenTallas redesigned ROM array, DeepSeek-V4.1-Flash, 200K context | 4,084 tokens/s per user at batch 1; 3,290 with 64 concurrent users | Deterministic analytical result on the analytical report framework, with the serial path priced from the per-token operator graph; not measured silicon |
| Same ROM array versus B200 on NVLink/NVL72 at the same silicon | 3.87× per user at batch 1, 5.37× at 64 users | Modeled; the GPU picks its own best layout, including NVL72 and expert parallelism, and runs decode as a fused persistent kernel whose dependent boundaries pay only their measured Blackwell dependency cost (no launch) |
| Each side at its best | 3.97× throughput, 11.84× tokens per joule | Best design and batch on each side at the same silicon |
The Taalas and OpenTallas rates are not an apples-to-apples benchmark. They use different models, contexts, systems, and evidence classes. The Taalas number is useful because it is a public proof point from fabricated hardware; the OpenTallas number is useful because its assumptions and derivation are open to inspection. Taalas publishes a product page, a public chatbot, and API documentation, but the benchmark itself was run by Taalas Labs rather than MLPerf or an independent laboratory. The exact source boundary is recorded in the source register.
Research status
OpenTallas is an active hardware research program, not a fabricated product. Its headline performance and cost figures are deterministic model outputs. Bounded functional executions, RTL campaigns, public-PDK layouts, and extracted circuit experiments exist only at their documented scopes. A complete target implementation, foundry signoff, packaged system, and OpenTallas silicon do not. Current source-locked and retained historical execution horizons are separated explicitly in the execution checklist. A cycle result that depends on assumed machine values is correctness evidence, not a performance measurement or projection.
Every figure below is a model output. Whether the design works is tracked separately, by a gate board of 19 rungs that is deliberately mostly red. Run it yourself on a bare checkout — no install, no EDA tools, under half a second:
PYTHONPATH=. python3 tools/check_redesign_gates.py
At the time of writing it prints 9 of 19 rungs passing and 1 of 4 terminal gates, and exits non-zero:
TERMINAL GATES FAILING: G1, G3, G4 -- the release criteria of
docs/OPENTALLAS_REDESIGN_PLAN.md are not met
What those failures mean, in plain terms:
Two further facts a reader should have before quoting anything here:
results/derived/source_currency_drift.json). A drifted artifact is
inadmissible as evidence until re-taken.python3 tools/run_roofline_studies.py --force
regenerates every figure in the analytical report.STATUS.md explains every failing gate in plain language and names the contributor task attached to it. What you can and cannot reproduce states the full boundary in three tiers.
High token rates are not valuable simply because a benchmark number is large. They change both the economics of inference and the kinds of products that can be built.
When a system produces more useful tokens during each paid second of hardware, its capital and operating costs are amortized across more output. That does not make lower cost automatic: utilization, yield, power, lifetime, model quality, and non-recurring engineering all matter. It does make throughput a powerful economic lever.
The analytical report prices energy per delivered token rather than cost: at each side's best, the redesigned ROM machine uses 8–12× less energy per token than B200 on DeepSeek models and 9× less on Qwen3-8B. The earlier partial-TCO scenarios came from the retired iso-node analytical model and are no longer published. A cost model would add mask and NRE amortisation, which falls on the ROM side, and model-refresh risk.
Here, stock does not mean a measured off-the-shelf server run. It means an allowed conventional GPU or GPU cluster assembled from the published device characteristics and compatibility rules in the study configuration. The performance comparator is the GPU layout that is best at each metric, at approximately equal silicon area.
The executable stack keeps this architecture choice explicit:
checkpoint + workload
│
▼
canonical model graph ──► backend-neutral IR ──► ABI 3.0 program
│
┌─────────────────────┴─────────────────────┐
▼ ▼
HBM/SRAM deployment ROM deployment
│ │
└──────── verifier + device + counters ─────┘
│
functional / cycle / RTL evidence
Qwen3-8B and DeepSeek-V4-Flash both use this shared compiler/runtime path. The analytical studies also use DeepSeek-V4-Pro as a scaling workload; it is not a third executable ABI target. See the architecture specification, ABI decision, and wire format for the contract.
The analytical report is the single source for performance. It compares a redesigned model-specific ROM machine with a general-purpose HBM machine at best shipping practice, at equal silicon area.
The ROM machine keeps weights in ROM, packages four dies over UCIe-class links, connects packages over direct SerDes, stripes experts across banks, and pipelines layers across packages. The HBM machine is B200 or A100 on NVLink (HGX and NVL72) and may use tensor, pipeline, hybrid or expert parallelism.
Summary at N5 against B200, per user:
Specialisation, not weights-in-ROM alone, produces most of the large-model single-user gain. With the GPU's own links, a ROM array only ties B200 on NVL72 for one user.
| Evidence layer | Present in the repository | Claim boundary |
|---|---|---|
| Model and architecture analysis | Checked traffic, iso-node, area-constrained, NoC, routing, cost, and sensitivity studies | Deterministic model output; not measured hardware |
| Compiler and runtime | Canonical model ingestion, backend-neutral IR, ABI 3.0 deployments, independent verification, functional and cycle devices | Bounded workloads and declared token-agreement horizons; not unrestricted model correctness |
| Digital implementation | Synthesizable public-reference RTL, formal/static/simulation/fault campaigns, and open-library implementation proxies | Qualified blocks and correlated deployments only; not a complete target chip or wafer |
| Circuit and physical methods | SKY130A and IHP controlled-via ROM slices, extracted simulations, and predictive/open-library routed blocks | Local methodology evidence; not leading-node ROM density, yield, or product signoff |
| Product silicon | Not present | No tapeout, fabricated OpenTallas device, package, full-chip P&R, foundry DRC/LVS, or silicon benchmark |
The evidence ladder defines how conceptual diagrams, simulated results, routed geometry, and extracted circuits may be interpreted. Functional, cycle, RTL, synthesis, place-and-route, SPICE, published, measured, and assumed evidence are intentionally not interchangeable. Start with the current program report, generated status, and evidence methodology before extending a claim.
| If you want to… | Start here |
|---|---|
| Understand why instant inference matters | Vision and product implications |
| Learn the architecture without a hardware background | OpenTallas in plain English |
| Navigate the full technical library | Documentation hub |
| Audit the performance comparison | Methodology, assumptions, and sources |
| Inspect the executable program | ABI 3.0 program report and compiler guide |
| Implement against the contract | Specification index, wire format, and operator conventions |
| Review digital hardware evidence | RTL inventory and RTL result reports |
| Review ROM circuit evidence | SPICE guide and physical methodology |
| Track active work | Unified execution checklist |
| See what passes today | PYTHONPATH=. python3 tools/check_redesign_gates.py — 19 rungs in 0.47 s |
| Know what you can reproduce | Reproducibility boundary |
The core Python analyses and tests run on a CPU workstation with Python 3.10 or newer:
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -e ".[test,compiler]"
make model-traffic
make spec-check
make test
Rebuild the headline studies and verify every annotated prose figure with:
make iso-node
make roofline
make check-figures
The complete make verify flow additionally needs RTL synthesis and simulation
tools. Its spice step checks circuit topology and source invariants; the
optional device, physical, and extracted-layout campaigns are separate and
require pinned SKY130A or IHP SG13G2 PDK/tool installations. See
spice/README.md. Model profiling reads checkpoint metadata
with range requests and does not download full checkpoints by default.
| Path | Purpose |
|---|---|
src/opentallas/ | Analytical architecture and performance models |
compiler/ | Model ingestion, canonical IR, ABI lowering, and deployment construction |
runtime/ | Independent checks, functional device, cycle model, and reference numerics |
spec/ | Governed system, microarchitecture, interface, numeric, RAS, firmware, and verification contracts |
rtl/ | Public-reference SystemVerilog, benches, formal harnesses, and campaigns |
spice/ and physical/ | Circuit and physical-methodology vehicles |
configs/ | Explicit model, hardware, benchmark, and PDK inputs |
results/ | Generated, reviewable evidence artifacts and reports |
docs/ | Vision, orientation, methodology, decisions, evidence notes, plans, and status |
tests/ | Unit, differential, integration, ABI, runtime, and simulation tests |
tools/ | Reproduction, checking, reporting, and campaign entry points |
OpenTallas welcomes work that makes an assumption more explicit, a result more reproducible, an implementation more complete, or a claim easier to audit. Read CONTRIBUTING.md before changing a specification, generated report, evidence grade, or headline figure. Add new documents to the documentation hub, and coordinate with current owners before renaming or moving an active plan or checklist.
The most valuable contribution is often not a larger number. It is a clearer boundary between what the repository demonstrates, what it models, and what still has to be measured in silicon.
OpenTallas is released under the MIT License.
See also CONTRIBUTING.md, the
code of conduct, and the security policy —
which distinguishes a security report (crafted input to the toolchain, a broken
tool pin) from a correctness report (a wrong number, a drifted artifact, a
failing gate). Correctness reports are welcome as ordinary issues; this project
is deliberately public about them.
NOTICE lists third-party material that is redistributed inside this
repository under its own terms and is not covered by MIT — notably ASAP7
standard-cell geometry embedded in three routed GDSII files, and one weight
tensor from a third-party model checkpoint committed as RTL test data. Read it
before redistributing.
Vendor comparator figures are quoted from published sources and remain the property of their publishers. OpenTallas is independent of Taalas and of every other vendor named here.
Open research toward model-specific mask-ROM LLM inference silicon: analytical studies, an ABI 3.0 compiler/runtime and cycle model, synthesizable RTL, and open-PDK physical evidence — gated against itself by a 19-rung board that is deliberately mostly red.
Verilog
4
24,291 commits
updated Oct 19, 2026
Open research toward instant, affordable AI inference.
What works today · What you can reproduce · How to contribute · Why speed matters · Plain-English overview · Documentation · Specification · Results
More than 10,000 tokens per second per user is now a publicly reported silicon result. Taalas reports 16,960 tokens/s per user for its fabricated HC1 accelerator running Llama 3.1 8B at batch one. That result changes the conversation: model-specific inference is no longer only a paper architecture. It is a credible path toward AI systems that respond at interactive timescales.
OpenTallas asks what it would take to make that architectural direction open, inspectable, and extensible to larger models and longer contexts. The repository connects model accounting, architecture simulation, a shared compiler/runtime ABI, public-reference RTL, and open-PDK circuit experiments in one evidence chain. It is independent of Taalas and does not claim to reproduce HC1.
For implementation work, use the revised chip architecture, checked resource budgets, and review handoff. These correct physical placement and service assumptions; the historical analytical studies below remain separate from a correctness-qualified chip throughput result.
| Signal | Result | Evidence class |
|---|---|---|
| Public model-specific silicon reference | 16,960 tokens/s per user on Taalas HC1 | Fabricated product; first-party Taalas run, publicly documented, not an independent benchmark |
| OpenTallas redesigned ROM array, DeepSeek-V4.1-Flash, 200K context | 4,084 tokens/s per user at batch 1; 3,290 with 64 concurrent users | Deterministic analytical result on the analytical report framework, with the serial path priced from the per-token operator graph; not measured silicon |
| Same ROM array versus B200 on NVLink/NVL72 at the same silicon | 3.87× per user at batch 1, 5.37× at 64 users | Modeled; the GPU picks its own best layout, including NVL72 and expert parallelism, and runs decode as a fused persistent kernel whose dependent boundaries pay only their measured Blackwell dependency cost (no launch) |
| Each side at its best | 3.97× throughput, 11.84× tokens per joule | Best design and batch on each side at the same silicon |
The Taalas and OpenTallas rates are not an apples-to-apples benchmark. They use different models, contexts, systems, and evidence classes. The Taalas number is useful because it is a public proof point from fabricated hardware; the OpenTallas number is useful because its assumptions and derivation are open to inspection. Taalas publishes a product page, a public chatbot, and API documentation, but the benchmark itself was run by Taalas Labs rather than MLPerf or an independent laboratory. The exact source boundary is recorded in the source register.
Research status
OpenTallas is an active hardware research program, not a fabricated product. Its headline performance and cost figures are deterministic model outputs. Bounded functional executions, RTL campaigns, public-PDK layouts, and extracted circuit experiments exist only at their documented scopes. A complete target implementation, foundry signoff, packaged system, and OpenTallas silicon do not. Current source-locked and retained historical execution horizons are separated explicitly in the execution checklist. A cycle result that depends on assumed machine values is correctness evidence, not a performance measurement or projection.
Every figure below is a model output. Whether the design works is tracked separately, by a gate board of 19 rungs that is deliberately mostly red. Run it yourself on a bare checkout — no install, no EDA tools, under half a second:
PYTHONPATH=. python3 tools/check_redesign_gates.py
At the time of writing it prints 9 of 19 rungs passing and 1 of 4 terminal gates, and exits non-zero:
TERMINAL GATES FAILING: G1, G3, G4 -- the release criteria of
docs/OPENTALLAS_REDESIGN_PLAN.md are not met
What those failures mean, in plain terms:
Two further facts a reader should have before quoting anything here:
results/derived/source_currency_drift.json). A drifted artifact is
inadmissible as evidence until re-taken.python3 tools/run_roofline_studies.py --force
regenerates every figure in the analytical report.STATUS.md explains every failing gate in plain language and names the contributor task attached to it. What you can and cannot reproduce states the full boundary in three tiers.
High token rates are not valuable simply because a benchmark number is large. They change both the economics of inference and the kinds of products that can be built.
When a system produces more useful tokens during each paid second of hardware, its capital and operating costs are amortized across more output. That does not make lower cost automatic: utilization, yield, power, lifetime, model quality, and non-recurring engineering all matter. It does make throughput a powerful economic lever.
The analytical report prices energy per delivered token rather than cost: at each side's best, the redesigned ROM machine uses 8–12× less energy per token than B200 on DeepSeek models and 9× less on Qwen3-8B. The earlier partial-TCO scenarios came from the retired iso-node analytical model and are no longer published. A cost model would add mask and NRE amortisation, which falls on the ROM side, and model-refresh risk.
Here, stock does not mean a measured off-the-shelf server run. It means an allowed conventional GPU or GPU cluster assembled from the published device characteristics and compatibility rules in the study configuration. The performance comparator is the GPU layout that is best at each metric, at approximately equal silicon area.
The executable stack keeps this architecture choice explicit:
checkpoint + workload
│
▼
canonical model graph ──► backend-neutral IR ──► ABI 3.0 program
│
┌─────────────────────┴─────────────────────┐
▼ ▼
HBM/SRAM deployment ROM deployment
│ │
└──────── verifier + device + counters ─────┘
│
functional / cycle / RTL evidence
Qwen3-8B and DeepSeek-V4-Flash both use this shared compiler/runtime path. The analytical studies also use DeepSeek-V4-Pro as a scaling workload; it is not a third executable ABI target. See the architecture specification, ABI decision, and wire format for the contract.
The analytical report is the single source for performance. It compares a redesigned model-specific ROM machine with a general-purpose HBM machine at best shipping practice, at equal silicon area.
The ROM machine keeps weights in ROM, packages four dies over UCIe-class links, connects packages over direct SerDes, stripes experts across banks, and pipelines layers across packages. The HBM machine is B200 or A100 on NVLink (HGX and NVL72) and may use tensor, pipeline, hybrid or expert parallelism.
Summary at N5 against B200, per user:
Specialisation, not weights-in-ROM alone, produces most of the large-model single-user gain. With the GPU's own links, a ROM array only ties B200 on NVL72 for one user.
| Evidence layer | Present in the repository | Claim boundary |
|---|---|---|
| Model and architecture analysis | Checked traffic, iso-node, area-constrained, NoC, routing, cost, and sensitivity studies | Deterministic model output; not measured hardware |
| Compiler and runtime | Canonical model ingestion, backend-neutral IR, ABI 3.0 deployments, independent verification, functional and cycle devices | Bounded workloads and declared token-agreement horizons; not unrestricted model correctness |
| Digital implementation | Synthesizable public-reference RTL, formal/static/simulation/fault campaigns, and open-library implementation proxies | Qualified blocks and correlated deployments only; not a complete target chip or wafer |
| Circuit and physical methods | SKY130A and IHP controlled-via ROM slices, extracted simulations, and predictive/open-library routed blocks | Local methodology evidence; not leading-node ROM density, yield, or product signoff |
| Product silicon | Not present | No tapeout, fabricated OpenTallas device, package, full-chip P&R, foundry DRC/LVS, or silicon benchmark |
The evidence ladder defines how conceptual diagrams, simulated results, routed geometry, and extracted circuits may be interpreted. Functional, cycle, RTL, synthesis, place-and-route, SPICE, published, measured, and assumed evidence are intentionally not interchangeable. Start with the current program report, generated status, and evidence methodology before extending a claim.
| If you want to… | Start here |
|---|---|
| Understand why instant inference matters | Vision and product implications |
| Learn the architecture without a hardware background | OpenTallas in plain English |
| Navigate the full technical library | Documentation hub |
| Audit the performance comparison | Methodology, assumptions, and sources |
| Inspect the executable program | ABI 3.0 program report and compiler guide |
| Implement against the contract | Specification index, wire format, and operator conventions |
| Review digital hardware evidence | RTL inventory and RTL result reports |
| Review ROM circuit evidence | SPICE guide and physical methodology |
| Track active work | Unified execution checklist |
| See what passes today | PYTHONPATH=. python3 tools/check_redesign_gates.py — 19 rungs in 0.47 s |
| Know what you can reproduce | Reproducibility boundary |
The core Python analyses and tests run on a CPU workstation with Python 3.10 or newer:
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -e ".[test,compiler]"
make model-traffic
make spec-check
make test
Rebuild the headline studies and verify every annotated prose figure with:
make iso-node
make roofline
make check-figures
The complete make verify flow additionally needs RTL synthesis and simulation
tools. Its spice step checks circuit topology and source invariants; the
optional device, physical, and extracted-layout campaigns are separate and
require pinned SKY130A or IHP SG13G2 PDK/tool installations. See
spice/README.md. Model profiling reads checkpoint metadata
with range requests and does not download full checkpoints by default.
| Path | Purpose |
|---|---|
src/opentallas/ | Analytical architecture and performance models |
compiler/ | Model ingestion, canonical IR, ABI lowering, and deployment construction |
runtime/ | Independent checks, functional device, cycle model, and reference numerics |
spec/ | Governed system, microarchitecture, interface, numeric, RAS, firmware, and verification contracts |
rtl/ | Public-reference SystemVerilog, benches, formal harnesses, and campaigns |
spice/ and physical/ | Circuit and physical-methodology vehicles |
configs/ | Explicit model, hardware, benchmark, and PDK inputs |
results/ | Generated, reviewable evidence artifacts and reports |
docs/ | Vision, orientation, methodology, decisions, evidence notes, plans, and status |
tests/ | Unit, differential, integration, ABI, runtime, and simulation tests |
tools/ | Reproduction, checking, reporting, and campaign entry points |
OpenTallas welcomes work that makes an assumption more explicit, a result more reproducible, an implementation more complete, or a claim easier to audit. Read CONTRIBUTING.md before changing a specification, generated report, evidence grade, or headline figure. Add new documents to the documentation hub, and coordinate with current owners before renaming or moving an active plan or checklist.
The most valuable contribution is often not a larger number. It is a clearer boundary between what the repository demonstrates, what it models, and what still has to be measured in silicon.
OpenTallas is released under the MIT License.
See also CONTRIBUTING.md, the
code of conduct, and the security policy —
which distinguishes a security report (crafted input to the toolchain, a broken
tool pin) from a correctness report (a wrong number, a drifted artifact, a
failing gate). Correctness reports are welcome as ordinary issues; this project
is deliberately public about them.
NOTICE lists third-party material that is redistributed inside this
repository under its own terms and is not covered by MIT — notably ASAP7
standard-cell geometry embedded in three routed GDSII files, and one weight
tensor from a third-party model checkpoint committed as RTL test data. Read it
before redistributing.
Vendor comparator figures are quoted from published sources and remain the property of their publishers. OpenTallas is independent of Taalas and of every other vendor named here.