Gemmeh

Gemmeh is a 1.1B-parameter decoder-only transformer trained for educational purposes from scratch on 20B tokens of pre-2024 FineWeb-Edu.

Architecture is Gemma 3-inspired without sliding-window attention. Custom 32k SentencePiece BPE tokenizer trained on the same corpus.

This repository contains:

• config.json
• modeling_gemmeh.py
• model.safetensors
• tokenizer.model

For GGUF quantizations, see ni-co-la-s/gemmeh-GGUF.

For the instruction-tuned model, see ni-co-la-s/gemmeh-it-GGUF.

Usage

This model uses custom modeling code, so trust_remote_code=True is required.

from transformers import AutoModelForCausalLM
from sentencepiece import SentencePieceProcessor
from huggingface_hub import hf_hub_download
import torch

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "ni-co-la-s/gemmeh",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    token=True,
)
model.eval()
print("Model loaded")

# Load tokenizer
sp_path = hf_hub_download(
    repo_id="ni-co-la-s/gemmeh",
    filename="tokenizer.model",
    token=True,
)
sp = SentencePieceProcessor()
sp.Load(sp_path)
print("Tokenizer loaded")

# Test generation
def generate(prompt, max_new_tokens=40, temperature=0.0):
    ids = sp.Encode(prompt, out_type=int)
    input_ids = torch.tensor([ids], dtype=torch.long)
    with torch.no_grad():
        for _ in range(max_new_tokens):
            out = model(input_ids)
            next_logits = out.logits[0, -1, :]
            if temperature == 0:
                next_id = next_logits.argmax().item()
            else:
                probs = torch.softmax(next_logits / temperature, dim=-1)
                next_id = torch.multinomial(probs, 1).item()
            if next_id == sp.eos_id():
                break
            input_ids = torch.cat([input_ids, torch.tensor([[next_id]])], dim=1)
    return sp.Decode(input_ids[0][len(ids):].tolist())

print(generate("The capital of France is", max_new_tokens=40, temperature=0.0))

Training details

Benchmarks (base 1B model, 20B tokens)

Evaluated through the BF16 GGUF served via llama.cpp with lm-eval. Other results are sourced from SmolLM2 and Gemma3 technical reports.

Trained on roughly 10–100× less data than the references, with a much smaller (32k vs 262k) vocabulary, and no distillation.

Limitations

• Smaller and less benchmark-competitive than similarly-sized models trained on more data.
• English only.
• 4,096 token context.
• Pre-2024 knowledge only.