README
TR-HASH MoE 200M — Interrupted 160B-Token Refinement Checkpoint
Phase 2 stopped at step 8,156 / 17,802. The root
model.safetensorsis this intermediate checkpoint (45.8%, approximately 32.07B refinement tokens), not a completed 70B-token refinement. Together with the 130B-token base run, it represents roughly 162B trained token exposures and is reported as the 160B-token source checkpoint.
Plan
Phase 2 takes the finished, fully-decayed TR-HASH 200M base checkpoint and runs a full-parameter refinement pass over the same 70B unique tokens used for base pretraining -- no replay, no augmentation, a fresh optimizer and learning-rate schedule, initialized from the base checkpoint's weights only.
| Setting | Value |
|---|---|
| Base checkpoint | TR-HASH-200M-130B (post-decay) |
| Training tokens | 69,997,690,880 (70B unique, exact match required) |
| Training progress represented by current weights | Step 8,156 / 17,802 (45.8%, ~32.07B tokens) |
| Replay / augmentation | None |
| Optimizer | Fresh at phase-2 start (not resumed from base pretraining state) |
| Parameters refined | Full model (not LoRA) |
Interim PIQA result
The step-8,156 checkpoint already exceeds the released base checkpoint on PIQA. This is an interim measurement, not the final phase-2 result.
| Checkpoint | PIQA acc | PIQA acc_norm | Correct (acc) | Examples |
|---|---|---|---|---|
| TR-HASH-200M-130B base final | 65.45% | 65.61% | — | 1,838 |
| Phase 2, step 8,156 | 68.66% | 68.39% | 1,262 / 1,838 | 1,838 |
Evaluation protocol: full PIQA validation split, zero-shot causal
continuation log-likelihood, no chat template, maximum sequence length
2,048. The checkpoint was evaluated in FP16 through MLX using the same model
weights. acc_norm selects the continuation by length-normalized
log-likelihood. Final results will be measured again after step 17,802.
Model summary
Same architecture as the base checkpoint: 201.2M parameters, 16 transformer layers, GQA (14 query heads / 2 KV heads), deterministic token-ID routed TR-Hash MoE (4 stored experts, top-2 active), tied embeddings. See TR-HASH-MoE-200M-130B for the full architecture and base pretraining details.
Status
- Base pretraining run complete
- Phase-2 refinement launched
- Intermediate step-8,156 weights, optimizer state, curves, and PIQA evaluation uploaded
- Phase-2 run stopped at step 8,156
- Full 70B-token refinement completed (not reached)
No active process is expected to finish this repository. The root weights remain the evaluated step-8,156 intermediate checkpoint.
Instruction post-training
The earlier experimental LoRA repository has been retired. The next planned
stage is a full-parameter instruction SFT, documented in
AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT.
Project links
License
The model release is CC BY-NC 4.0. Source datasets and individual documents retain their own licenses and terms.
Contributors
18 commits
Similar repos
Linked in READMEs
README
TR-HASH MoE 200M — Interrupted 160B-Token Refinement Checkpoint
Phase 2 stopped at step 8,156 / 17,802. The root
model.safetensorsis this intermediate checkpoint (45.8%, approximately 32.07B refinement tokens), not a completed 70B-token refinement. Together with the 130B-token base run, it represents roughly 162B trained token exposures and is reported as the 160B-token source checkpoint.
Plan
Phase 2 takes the finished, fully-decayed TR-HASH 200M base checkpoint and runs a full-parameter refinement pass over the same 70B unique tokens used for base pretraining -- no replay, no augmentation, a fresh optimizer and learning-rate schedule, initialized from the base checkpoint's weights only.
| Setting | Value |
|---|---|
| Base checkpoint | TR-HASH-200M-130B (post-decay) |
| Training tokens | 69,997,690,880 (70B unique, exact match required) |
| Training progress represented by current weights | Step 8,156 / 17,802 (45.8%, ~32.07B tokens) |
| Replay / augmentation | None |
| Optimizer | Fresh at phase-2 start (not resumed from base pretraining state) |
| Parameters refined | Full model (not LoRA) |
Interim PIQA result
The step-8,156 checkpoint already exceeds the released base checkpoint on PIQA. This is an interim measurement, not the final phase-2 result.
| Checkpoint | PIQA acc | PIQA acc_norm | Correct (acc) | Examples |
|---|---|---|---|---|
| TR-HASH-200M-130B base final | 65.45% | 65.61% | — | 1,838 |
| Phase 2, step 8,156 | 68.66% | 68.39% | 1,262 / 1,838 | 1,838 |
Evaluation protocol: full PIQA validation split, zero-shot causal
continuation log-likelihood, no chat template, maximum sequence length
2,048. The checkpoint was evaluated in FP16 through MLX using the same model
weights. acc_norm selects the continuation by length-normalized
log-likelihood. Final results will be measured again after step 17,802.
Model summary
Same architecture as the base checkpoint: 201.2M parameters, 16 transformer layers, GQA (14 query heads / 2 KV heads), deterministic token-ID routed TR-Hash MoE (4 stored experts, top-2 active), tied embeddings. See TR-HASH-MoE-200M-130B for the full architecture and base pretraining details.
Status
- Base pretraining run complete
- Phase-2 refinement launched
- Intermediate step-8,156 weights, optimizer state, curves, and PIQA evaluation uploaded
- Phase-2 run stopped at step 8,156
- Full 70B-token refinement completed (not reached)
No active process is expected to finish this repository. The root weights remain the evaluated step-8,156 intermediate checkpoint.
Instruction post-training
The earlier experimental LoRA repository has been retired. The next planned
stage is a full-parameter instruction SFT, documented in
AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT.
Project links
License
The model release is CC BY-NC 4.0. Source datasets and individual documents retain their own licenses and terms.
Similar repos
Linked in READMEs
Contributors
18 commits