README
TR-HASH MoE 200M — 70B Unique / 130B Replay
TR-HASH MoE 200M is a 201.2M-parameter decoder-only base language model with grouped-query attention, a shared SwiGLU path, and deterministic token-ID routed experts. It is trained on a 70B-token unique selection from a larger pretokenized mixture, replayed to 130B total trained-token exposures.
This is a base completion model, not an instruction-tuned assistant. It has no chat template or instruction-following guarantee.
Model summary
| Component | Value |
|---|---|
| Parameters | 201.2M |
| Transformer layers | 16 |
| Hidden size | 896 |
| Attention | GQA, 14 query heads / 2 KV heads |
| Vocabulary | 32,000 |
| Configured maximum context | 2,048 tokens |
| Routed MLP | 4 deterministic token-ID experts, top-2 active |
| Embeddings | Input/output weights tied |
Each block combines a wide shared SwiGLU path with a small set of experts selected deterministically from the token ID — routing has no learned gate, so there is no contextual router decision at inference time.
Training data
Trained on a deterministic replay plan over the 200B-token pretokenized mixture (DCLM, FineWeb-Edu, Stack-Edu, FineMath, InfiWebMath, Cosmopedia v2), selecting ~70B unique tokens and replaying to ~130B total trained-token exposures.
Status
Training is complete — the run reached its full ~130B planned token exposure.
The raw step-by-step training log is included in this repo as
training_log.csv for anyone who wants the full curve.
A validated held-out evaluation has not been run yet; treat this as a
released base checkpoint, not a benchmarked one.
Intended use
For language-model research, architecture comparisons, continued pretraining, and supervised refinement. As a base model, it should be prompted for text completion rather than conversation.
Limitations
This is an experimental small base model. It may repeat text, lose coherence, invent facts, emit unsafe or biased content, reproduce patterns from training data, and fail on reasoning, arithmetic, code, multilingual text, and safety-sensitive prompts. It has not been independently evaluated for benchmark contamination, memorization, privacy leakage, toxicity, or bias.
Do not use it for consequential decisions. Evaluate the exact checkpoint and serving stack for your intended application before deployment.
Project links
License
CC BY-NC 4.0. Source datasets and individual documents retain their own licenses and terms. Users are responsible for reviewing dataset provenance and compliance for their intended use.
Contributors
12 commits
Similar repos
README
TR-HASH MoE 200M — 70B Unique / 130B Replay
TR-HASH MoE 200M is a 201.2M-parameter decoder-only base language model with grouped-query attention, a shared SwiGLU path, and deterministic token-ID routed experts. It is trained on a 70B-token unique selection from a larger pretokenized mixture, replayed to 130B total trained-token exposures.
This is a base completion model, not an instruction-tuned assistant. It has no chat template or instruction-following guarantee.
Model summary
| Component | Value |
|---|---|
| Parameters | 201.2M |
| Transformer layers | 16 |
| Hidden size | 896 |
| Attention | GQA, 14 query heads / 2 KV heads |
| Vocabulary | 32,000 |
| Configured maximum context | 2,048 tokens |
| Routed MLP | 4 deterministic token-ID experts, top-2 active |
| Embeddings | Input/output weights tied |
Each block combines a wide shared SwiGLU path with a small set of experts selected deterministically from the token ID — routing has no learned gate, so there is no contextual router decision at inference time.
Training data
Trained on a deterministic replay plan over the 200B-token pretokenized mixture (DCLM, FineWeb-Edu, Stack-Edu, FineMath, InfiWebMath, Cosmopedia v2), selecting ~70B unique tokens and replaying to ~130B total trained-token exposures.
Status
Training is complete — the run reached its full ~130B planned token exposure.
The raw step-by-step training log is included in this repo as
training_log.csv for anyone who wants the full curve.
A validated held-out evaluation has not been run yet; treat this as a
released base checkpoint, not a benchmarked one.
Intended use
For language-model research, architecture comparisons, continued pretraining, and supervised refinement. As a base model, it should be prompted for text completion rather than conversation.
Limitations
This is an experimental small base model. It may repeat text, lose coherence, invent facts, emit unsafe or biased content, reproduce patterns from training data, and fail on reasoning, arithmetic, code, multilingual text, and safety-sensitive prompts. It has not been independently evaluated for benchmark contamination, memorization, privacy leakage, toxicity, or bias.
Do not use it for consequential decisions. Evaluate the exact checkpoint and serving stack for your intended application before deployment.
Project links
License
CC BY-NC 4.0. Source datasets and individual documents retain their own licenses and terms. Users are responsible for reviewing dataset provenance and compliance for their intended use.
Similar repos
Linked in READMEs
Contributors
12 commits