H-JEPA — GGUF for jepa.cpp

H-JEPA (arXiv 2610.06805, Zhang, Terver, Rabbat, LeCun, Balestriero) trains a hierarchy of action-conditioned JEPA world models end to end: level 1 predicts one step ahead in the latent space of a ViT image encoder, each level above re-encodes every stride-th state of the level below and predicts one of its steps ahead, conditioned on a latent macro-action. These are the authors' released checkpoints (jepa-world-models/h-jepa, code at kevinghst/H-JEPA) converted to GGUF for jepa.cpp, a ggml C/C++ engine with no Python and no PyTorch at run time.

hjepa_l{N} is the N-level H-JEPA, hwm_l{N} the HWM baseline (identity upper encoders), lewm the flat one-level baseline. One seed of every released configuration is here (seed 42 for the simulated environments, seed 1 for DROID).

What runs

The encoder, every level's state encoder, action encoder, predictor and latent rollout, and the paper's top-down gradient planner (jepa_hjepa_* in include/jepa.h; jepa-plan goes from an observation and a goal image to actions).

hf download jepacpp/h-jepa-GGUF ant/ant_hjepa_l3-seed42-f16.gguf --local-dir models/gguf
jepa-plan -m models/gguf/ant/ant_hjepa_l3-seed42-f16.gguf --obs now.png --goal goal.png --horizons 2,2,5 --samples 100,100,400

Parity with the authors' PyTorch

Every file below was checked by test-predictor --hjepa against activations dumped from the authors' own modules (scripts/hjepa_reference.py): encoder tokens, each level's states, latent actions, action embeddings, teacher-forced predictions and autoregressive rollouts, and the whole chain from frames to top-level states. cos min is the worst row cosine over all of those comparisons, rel max the largest max-abs error relative to the reference's largest value.

Measured at jepa.cpp 871b994, CPU. Bars: tests/test-predictor.cpp thresholds_for(): f32 cos >= 0.9999 and rel <= 1e-3; f16 cos_mean >= 0.9999, cos_min >= 0.99; q8_0 cos_mean >= 0.999, cos_min >= 0.98. q4_k and below do not clear the q8_0 bars on these small models and are not published.

The planner is checked against the authors' GradientSolver / HierarchicalSolver replayed on their own random draws: at f32 every cost, autograd gradient, solve and hierarchical plan matches, and at f16 every cost and gradient. A q8_0 file is verified as a forward model only; whether it plans as well as f32 has not been measured in closed loop.

Inputs

Frames are squash-resized to the encoder's image size and ImageNet-normalised (jepa.pre.*). Actions and proprioception are z-scored with the training statistics stored in the file (norm.action.*, norm.proprio.*, jepa_hjepa_norm_stats); a level-1 action is jepa.hjepa.frameskip consecutive raw actions stacked.

Licence

MIT, as the source checkpoints (jepa-world-models/h-jepa) and code. Cite the paper:

@article{zhang2026hjepa,
  title   = {H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning},
  author  = {Zhang, Wancong and Terver, Basile and Rabbat, Michael and LeCun, Yann and Balestriero, Randall},
  journal = {arXiv preprint arXiv:2610.06805},
  year    = {2026}
}