granite-4.1-tt-metal

Community port of IBM Granite 4.1-8B to Tenstorrent Wormhole (via tt-metal and the Tenstorrent vLLM plugin).

非公式・非提携のコミュニティポートです。 本プロジェクトは IBM、Tenstorrent、株式会社ネットワールド (Networld Corporation) の いずれとも提携・関連関係になく、各社は本プロジェクトの開発、検証、公開に関与していません。 また、本プロジェクトは各社によって承認または推奨されたものではありません。

Community, non-affiliated port. IBM, Tenstorrent, and Networld Corporation were not involved in the development, validation, or publication of this project and do not endorse it.

This repository contains porting code, verification harness, and measured performance results. It does not ship model weights — obtain them from IBM's canonical HF repository.

現在の検証状況 / Status at a glance

「確認済み」は、記載されたデバイス、固定ソフトウェアスタック、および測定条件の 範囲内だけを意味します。その他の構成での動作や性能を保証するものではありません。

"Confirmed" applies only to the stated devices, pinned software stack, and measurement conditions. It is not a claim for other configurations.

代表性能 / Representative performance — msq=4, mml=8192, N300、single-stream、 n=2 の平均値、percentile なし: end-to-end greedy 24.8 tok/s。 完全な測定方法、並行実行、prefill scaling は docs/performance.md、 検証済み / 未検証範囲の詳細は docs/validation.md を参照。

クイックスタート: 検証済みの N300 固定スタック / Quickstart (N300 with the pinned stack)

以下は、このポートで検証済みの標準手順です。検証結果は、固定された device、firmware、 container image、tt-metal、vLLM(tt)、および IBM weights の revision の組み合わせに 限定されます。最初に scripts/verify/preflight.sh を実行してください。

• PASS: 検証済み構成と一致しています
• WARN: 検証範囲外の差異があります。続行できますが、結果が異なる可能性があります
• FAIL: 必須項目が不足しています。修正してから続行してください

その他の mesh 構成やソフトウェアスタックは未検証です。正確な固定バージョンは docs/stack.md、「検証済み」の範囲は docs/validation.md を参照してください。

This is the validated one-line path. Other mesh / stack combinations are untested — see docs/stack.md for the exact pin, and docs/validation.md for what "validated" scope means.

# 1) Clone the port at a specific release tag
git clone --branch v0.1.0 --depth 1 https://github.com/unnowataru/granite-4.1-tt-metal.git
cd granite-4.1-tt-metal

# 2) Preflight: does your box match the validated stack?
bash scripts/verify/preflight.sh
# → PASS   = your device / FW / image / weights match the pin
# → WARN   = something is out of the validated range; you can proceed, results may drift
# → FAIL   = required item missing (driver, docker, container image); fix before continuing

# 3) Download IBM Granite 4.1-8B weights at the pinned revision (~16GB)
bash scripts/verify/model-files.sh --download
# HF_MODEL, HF_REVISION are env-overridable — defaults match the Validated pin.

# 4) Start the pinned dev container, apply the 6-file overlay, patch the vLLM(tt) plugin,
#    launch the server
bash scripts/quickstart.sh
# The quickstart walks through each remaining step with copyable commands, including
# scripts/sync/tt_files.sh as the canonical overlay-application path (ad-hoc cp is not supported).

# 5) Judge the smoke test (single command, pass/fail, waits up to 15 min for first-launch compile)
bash scripts/integration/smoke.sh
# → PASS = server reachable, chat completion returns, SSE streaming works
# → FAIL = the script prints where it stopped; collect diagnostics with:
bash scripts/verify/collect-diagnostics.sh
# The diagnostics bundle goes to /tmp/tt-diagnostics-<epoch>/; grep it for internal
# identifiers before attaching to any Issue.

If any step is unclear, docs/bringup-playbook.md is the long form.

Supported hardware

Validated on the exact stack listed below — see docs/stack.md for full pin.

The preflight script warns rather than fails on FW/image mismatch — this port has not measured those combinations but does not know they are broken.

Repository layout

overlays/tt_transformers/tt/       6-file Granite diff (SPDX + Apache-2.0 §4(b) modified notice)
scripts/
  quickstart.sh                    Guided end-to-end from preflight to smoke
  verify/                          preflight, HF weight check, teacher-forced PCC, generation compare,
                                     diagnostics collector
  serving/                         vLLM(tt) server launch, Granite registration patch, curl recipes
  perf/                            Long-context / trace / perf server
  integration/                     smoke.sh (single pass/fail), sdk_smoke.py, tool_call_test.py
  sync/                            6-file SHA256 manifest + verify + canonical sync entrypoint
  publish-hf/                      Maintainer tool: mirror hf/README.md to the HF card repo
docs/
  stack.md                         Supported hardware + software stack (full pin)
  performance.md                   Measured performance + methodology
  validation.md                    Confirmed vs Not Yet Validated
  traps.md                         Symptom → Cause → Verify → Fix table
  implementation/                  Detailed implementation reference (multipliers, hooks)
  bringup-playbook.md              Long-form method for porting other HF LLMs
  operations/                      **Maintainer / contributor documentation** (release policy,
                                     issue triage, revalidation matrix, HF publish, community
                                     features, upstream PR candidates) — not required for first use
  evidence/                        Raw curated evidence logs
requirements/                      client (OpenAI SDK) / hf-ref (torch, transformers) / dev (CI)
hf/README.md                       HuggingFace Model Card body (compatibility card only, no weights)

Where the weights come from — and where they don't

• Use IBM's canonical: ibm-granite/granite-4.1-8b, Apache-2.0. Pin to revision 1504002f650e656a0a3789d99574df12e3e94ed0 for exact reproduction of the numbers in this repo.
• Do NOT use this repo's HF companion for weights. The HuggingFace mirror at unnowataru/granite-4.1-8b-tt-metal is a compatibility card only (no weights). Passing that ID to from_pretrained(), pipeline(), or vLLM will fail — this is intentional.
• No fine-tuned or quantized artifacts are hosted anywhere by this port.

Honest caveats

• Numerical validation is N150 single-device, single-prompt. N300/T3K tensor-parallel numerical equivalence has not been measured.
• Single stack. All numbers anchor to tt-metal 805f43d / container digest sha256:6cbb9532.... Drift on any of those is uncharacterized.
• msq=8 on N300 is uncharacterized. One startup attempt hit chip-lock contamination from a prior process, not KV OOM — do not read that as a capacity ceiling.
• No percentile latency, no multi-prompt statistical PCC, no sustained-load characterization.
• Community port, single maintainer. Not endorsed by IBM or Tenstorrent. See SECURITY.md and CONTRIBUTING.md.

License

• Repository content: Apache-2.0 (see LICENSE).
• Model weights: IBM's Apache-2.0 (not redistributed here — obtain directly from IBM's HF repo).
• Upstream tt-metal and vLLM plugin: Tenstorrent's Apache-2.0 (see NOTICE).
• The 6 files under overlays/tt_transformers/tt/ are Derivative Works of the upstream tt_transformers at commit 805f43d, carrying the required Apache-2.0 §4(b) modified-notice header.

Contributing

See CONTRIBUTING.md. Reproduction reports on other Wormhole hardware are particularly welcome. Security issues go through private vulnerability reporting.

Maintainer / contributor documentation

The following are for maintaining or extending this port, not for first-time users:

• docs/operations/release-policy.md — versioning, retract flow
• docs/operations/triage.md — issue label / SLA / security incident
• docs/operations/revalidation-matrix.md — stack drift + repo change → required smoke
• docs/operations/community-features.md — CoC / Discussions
• docs/operations/upstream-pr-candidates.md — return-upstream candidates

Related

• Model card companion (no weights): unnowataru/granite-4.1-8b-tt-metal
• Upstream: ibm-granite/granite-4.1-8b · tenstorrent/tt-metal · tenstorrent/vllm