APEX-2

A from-scratch Mixture-of-Experts LLM: 3.87B total / 1.45B active parameters, pretrained on 86.5B tokens and instruction-tuned with SFT.

• Full code, architecture writeup and training diary: github.com/DW-dev-UE/LLM-from-scratch.
• The full benchmark writeup this card summarizes is BENCHMARK v3.
• Base model (for continued pretraining or your own fine-tuning): YOON1v/Apex-2-Base.

This checkpoint is the SFT release (pretrain → 2-stage SFT). A DPO stage was tried and dropped because it lowered code, math and instruction-following scores (see below).

Model details

Usage

The model uses the ChatML template (<|im_start|>role\n...<|im_end|>) shipped in tokenizer_config.json; generation ends at <|im_end|>.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("YOON1v/Apex-2", dtype=torch.bfloat16, device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained("YOON1v/Apex-2")

messages = [{"role": "user", "content": "Write a Python function that checks if a number is prime."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

vLLM loads it directly as well: LLM(model="YOON1v/Apex-2", dtype="bfloat16", max_model_len=4096).

Benchmarks

Measured by us with vLLM greedy decoding and the chat template (prompt + answer ≤ 4096 tokens). Model-written code was executed in a sandbox.

Compared with similar-size models (official numbers; protocols differ, so this is a rough comparison):

_{DeepSeek-Coder numbers are the Qwen2.5-Coder report's re-evaluation. * Different IFEval metric: Apex-2 and Qwen report prompt-level strict; the others report an average of metrics or do not specify.}

With 1/23 to 1/400 of the pretraining data of these models, Apex-2's base model matches Qwen2.5-1.5B (18T tokens) on HumanEval+ (32.9 vs 32.9). The big gaps are knowledge (MMLU) and math, which mainly track pretraining scale.

[!NOTE] SFT data was 13-gram decontaminated against HumanEval(+), MBPP(+), GSM8K test, MATH, MMLU, ARC, HellaSwag, PIQA, WinoGrande, IFEval and LAMBADA. For LiveCodeBench only the newer v5–v6 problems (after 2024-08) were used, to reduce contamination risk.

Why no DPO: DPO on allenai/Dolci-Instruct-DPO (220K pairs) made answers 2.3× longer (320 → 733 tokens on average). Chosen-answer likelihood also fell during training. In evaluation HumanEval+ went 41.5 → 32.3, MBPP+ 48.9 → 37.3, GSM8K 32.4 → 14.3 and IFEval 44.7 → 35.7, so this SFT checkpoint is the release.

Known limitations

• English-centric. Other languages, including Korean, barely work (multilingual data was excluded from SFT).
• Limited world knowledge (MMLU ~29%), so it often states wrong facts with confidence.
• Weak at competitive programming (LiveCodeBench medium ~1%) and at predicting code execution (CRUXEval).
• 4096-token context. No tool use, no thinking mode.

Training data

• Pretrain: 40 deduplicated sources, including:
  • web: FineWeb-Edu, DCLM, FinePDFs
  • code: StarCoder-family and The Stack-derived code in 11 languages, OpenCoder corpora, synthetic code
  • math: FineMath, InfiWebMath, Nemotron-CC-Math
  • curated: Wikipedia, StackExchange, peS2o, Cosmopedia, Gutenberg
• SFT:
  • general: allenai/Dolci-Instruct-SFT (tool-use, multilingual and identity subsets removed)
  • code: OpenCoder opc-sft-stage1/2, nvidia/OpenCodeInstruct (unit-test pass rate ≥ 0.9), bigcode self-oss-instruct, m-a-p/Code-Feedback
  • math: nvidia/OpenMathInstruct-2, AI-MO/NuminaMath-CoT

Some SFT sources contain synthetic data generated by third-party models (e.g. GPT-4-class, Llama-3.1-405B, Qwen2.5-Coder); check each dataset's terms for your use case.

License

Apache 2.0.