LeWM Qwen3-32B Token-Similarity World Model

This repository contains a LeWorldModel-style JEPA checkpoint trained on activation-derived observations from Qwen/Qwen3-32B.

The model was trained on the dataset Artvv/qwen3-32b-token-similarity-activation-triplets, which contains 50,000 transition triplets:

obs_t      : [3, 64, 64] float32 token-token similarity activation image
action_t   : [256] float32 PCA projection of generated token embedding
obs_t+1    : [3, 64, 64] float32 next activation image
teacher    : Qwen/Qwen3-32B
layers     : [10, 20, 30]
encoding   : token_similarity_v1

Files

• checkpoints/latest.pt: epoch-20 training checkpoint with model and optimizer state dicts.
• checkpoints/lewm_qwen3_epoch_20_object.ckpt: serialized epoch-20 model object.
• configs/qwen3_reasoning.yaml: training config.
• reports/train_summary.json: full epoch-by-epoch training metrics.
• reports/voe.json: first VoE report.
• reports/voe_calibrated.json: calibrated VoE report with temporal, intra-domain, and inter-domain controls.
• tensorboard/: TensorBoard event file from the run.

Training Summary

Run setup:

examples       : 50,000
train / val    : 47,500 / 2,500
device         : A100 80GB
epochs         : 20
elapsed        : 368.4 seconds

Final epoch metrics:

train_pred_loss          = 0.03288
train_sigreg_loss        = 3.64351
latent_model_mse         = 0.02749
latent_identity_mse      = 0.16838
latent_improvement_ratio = 6.12520
probe_auc_ovr            = 0.99939

Decision at epoch 5:

continue_to_epoch_20
latent_improvement_ratio = 2.379 > 1.5

Calibrated VoE

The calibrated violation-of-expectation evaluation compares the true temporal transition against mismatched next observations from the same domain and from different domains.

coherent_latent_mse_mean     = 0.02708
intra_domain_latent_mse_mean = 1.62214
inter_domain_latent_mse_mean = 1.93429
intra_domain_voe_ratio       = 59.90
inter_domain_voe_ratio       = 71.43

coherent_image_mse_mean      = 0.00518
intra_domain_image_ratio     = 6.27
inter_domain_image_ratio     = 6.94

The latent VoE ratios are much larger than the raw image-distance ratios, suggesting the model learned a transition-compatibility function beyond pixel distance alone. Same-domain mismatches remain highly surprising, so the current checkpoint should be interpreted as learning local trajectory compatibility, not yet architecture- or domain-level semantic invariance.

Usage

Clone the research code:

git clone https://github.com/ArthurVigier/CODE-JEPA.git
cd CODE-JEPA
python -m pip install -e .

Load the checkpoint with the LeWM-Qwen adapter code in third_party/le-wm:

from pathlib import Path
import sys
import torch
import yaml

root = Path("third_party/le-wm").resolve()
sys.path.insert(0, str(root))

from train_qwen3_reasoning import build_lewm_model

cfg = yaml.safe_load(Path("third_party/le-wm/config/train/qwen3_reasoning.yaml").read_text())
model = build_lewm_model(cfg, obs_channels=3, action_dim=256)
ckpt = torch.load("checkpoints/latest.pt", map_location="cpu")
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

Limitations

• This is a research checkpoint, not a general-purpose language model.
• It consumes activation-derived images, not raw text.
• The teacher was Qwen/Qwen3-32B; users must comply with the upstream model license and terms.
• Calibrated VoE is strong, but intra-domain mismatches are also highly surprising, so claims should remain trajectory-local until further Phase 3 evaluation.

Citation

If this artifact is useful, cite the CODE-JEPA / Activation-as-View repository and the upstream LeWorldModel work.