H3-World — action-conditioned world model

DANNY621/H3-World is a rank-32 LoRA for MiniMaxAI/MiniMax-H3 that turns it into a playable world model: you give it a first frame and a key sequence, and it renders what happens as those keys are held.

forward*20, forward-right*10, pan-right-fast*7

How the actions actually reach the model

The checkpoint's own run manifests are unambiguous about this, and it is the thing that makes the LoRA demo-able at all:

"checkpoint": {"action_dim": 0, "action_mode": "text", "action_tensors": 0,
               "lora_tensors": 208, "lora_pairs": 104}
"config": {"num_frames": 124, "height": 480, "width": 832, "fps": 24, "steps": 50,
           "conditioning": "first_frame+8d_actions",
           "action_columns": ["W", "A", "S", "D", "I", "J", "K", "L"]}

action_dim: 0 and action_tensors: 0 — there is no action encoder and no action embedding. The 8-dimensional key state is carried through the text channel: one short English sentence per latent video frame, appended to the scene prompt, in the register the author's own runs use.

So a 124-frame request (37 latent frames) is conditioned on a prompt that looks like:

A third-person view of a man walking through a city intersection...
the man walks forward, camera follows him
the man walks forward, camera follows him
...
the man walks forward and strafes right, camera follows him
...
the man stands still, camera pans right sharply

Every sentence ends in a camera clause — there is no "no camera" shape in the training data. With no camera key held the reference still states what the camera is doing: camera follows him while the character is moving, camera holds steady while he is not. caption_for() in app.py is a port of the reference's annotate_from_keys9 (code/abot/action_script.py), sentence-for-sentence across all 130 reachable key combinations, so the strings the model receives here are the strings it was trained on. The resolved per-frame captions are shown under Conditioning after every run.

The directed attention mask

The model card is explicit that the LoRA weights alone do not reproduce the reported behavior: the training run used a directed attention mask that binds each per-frame caption to the latents of its frame. Without it every video row attends to all 37 sentences at once and the sequence collapses into an average action.

MiniMax-H3 is a single packed 1-D sequence under full self-attention — [text | keyframe anchors | audio | video], no cross-attention — so the mask is a constraint inside one attention call, not a separate cross-attention mask.

app.py reimplements it as an exact log-sum-exp merge rather than a dense [S, S] mask (which would force SDPA off its flash kernel for the whole 21k-row sequence). The keys are split into three regions:

Each returns its output and its log-sum-exp; the three are recombined with the online-softmax identity, which is numerically identical to one masked softmax over the full row.

What "directed" means. The reference's mask_mod cuts the annotations' outgoing edges, and only those:

• a sentence row, as a key, is readable by itself and by the video rows of its own latent frame — the static prompt, the keyframe anchors, the audio and every other sentence are blocked from reading it;
• a sentence row, as a query, reads the video rows of its own latent frame only. It still reads the static prompt, the anchors and the audio freely — that is information flowing into the sentence, giving it scene grounding, not a bypass around frame i.

Blocking A_j → A_k is the load-bearing half. With it, V_k is a cut vertex between A_k and the rest of the sequence; without it A_j's content reaches A_k first, and V_k reading A_k is reading A_j too.

The mask is toggleable in Advanced so you can see the difference. With it off, the same script produces a video that drifts through a blur of every action at once.

Three guards, all in app.py:

• The token spans are located by re-tokenizing prefixes of the prompt. If a BPE merge straddles a sentence boundary the spans would be wrong, so build_conditioning_text() verifies cuts[-1] == total and refuses to mask rather than mask the wrong rows.
• Once the mask actually bites, most rows see no sentence key at all, so the masked region's log-sum-exp is -inf for them. The merge treats that as weight zero, but softmax over an all--inf row is NaN — so _masked_lse zeroes those rows explicitly.
• MiniMaxH3TokenRefinerBlock runs the same attention module over the text stream alone, which the processor detects by its length (cap_end is the whole text length by construction). That stream is attended block-diagonally — one segment for the scene prompt, one per sentence — matching the reference's split refiner_cu. Running it as a single document lets sentences mix before the DiT backbone, and no mask on the DiT side can close an upstream leak. The LoRA adapts token_refiner.refiner_blocks.{0,1}, so those deltas were trained under this segmentation.

Known divergence from the reference

One difference remains, and it is upstream of everything above. The reference encodes each sentence independently through the text encoder and concatenates the row blocks after the head, caching by string so a repeated sentence is bit-identical. This Space builds one prompt string and sends it to the shared multimodalart/qwen3vl-conditioner, so sentence k is contextualised against the image and every preceding sentence before the DiT ever sees it. Closing it needs a raw text-encode endpoint on the conditioner, which it does not currently expose. Related: the reference also gives annotation rows a mirrored positional offset (origin = text_len - s_k[-1] - 1), which is not reachable from this side of the split either.

Architecture — why two Spaces

MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at 150 GB of storage, so the pipeline is split at its text_encoder step, exactly as in multimodalart/minimax-h3:

• the 62.14 GiB Qwen3-VL conditioner runs in multimodalart/qwen3vl-conditioner, which this Space calls over the gradio API for every request;
• this Space holds the 61.73 GiB transformer (with H3-World merged into it) plus the video and audio autoencoders.

The wire format between them is prompt_embeds (1, N, 5120) bf16 + text_token_tags (N,) int64 in a single safetensors file. The caption spans this Space needs are recovered from it by offset arithmetic, because MiniMaxH3TextEncoderStep appends the prompt verbatim — no chat template, no special tokens — so offset = num_text_tokens - num_prompt_tokens.

h3_split_blocks.py is the blockset with the text_encoder step removed, copied from that Space.

LoRA merge

The LoRA is published against the original MiniMax-H3 layout, not the diffusers port, so load_lora_weights() does not apply. load_and_apply_lora() replays convert_minimax_h3_to_diffusers.py's renames on the way in — blocks. → transformer_blocks., attn.out_proj → attn.to_out.0, mlp.fc1 → ff.net.0.proj with the SwiGLU gate/value halves swapped, and the fused attn.qkv_proj de-interleaved per head before being split into to_q / to_k / to_v. 104 LoRA pairs become 208 merged weight deltas; any target that fails to resolve is fatal, never skipped.

28 steps vs the 8-step turbo LoRA

Sampling in the UI picks between two configurations of the same request — same seed, same action script, same directed mask — so the quality cost of the distillation is directly visible:

h3_turbo_lora.py folds the repo's own recommended checkpoint — the v4 line, step 600, EMA:

• the file is a ComfyUI-side checkpoint in the original MiniMax-H3 naming (blocks.N.attn.qkv_proj.lora_A.weight, blocks.N.adaln_proj.linear, final_layer.adaln_proj.linear), so like H3-World it is replayed through convert_minimax_h3_to_diffusers.py's renames: 259 LoRA pairs become 363 weight deltas, covering to_q/to_k/to_v/to_out.0/ff.net.0.proj/ff.net.2 on all 52 attention blocks plus the 50 block-level AdaLN projections and norm_out.linear;
• one transform differs from the H3-World merge: the adapter was trained against comfy.ldm.minimax.model, whose attention reads qkv_proj(x).split(heads * head_dim, dim=-1), so Comfy-Org/MiniMax-H3 holds the fused QKV as contiguous [q_all; k_all; v_all] and these rows are split into thirds without the raw-shard per-head de-interleave. The mlp.fc1 [gate; value] → [value; gate] swap still applies, and both AdaLN projections are pure renames;
• ranks are mixed by design — 64 on the attention/FFN projections, 16 on the AdaLN ones — and the file's metadata says application: W_eff = W + lora_B @ lora_A with the card's "alpha = rank, so no extra scaling", so the fold scale is exactly 1.0 and H3_TURBO_STRENGTH is left as the card's strength dial;
• the card's useful range is 4–8 steps (6–8 recommended, nothing gained past 8), so the step count is overridden to 8 NFE. No scheduler swap and no CFG change: MiniMax-H3 is already guidance-distilled and this Space keeps its native MiniMaxH3Scheduler with the turbo LoRA folded.

It is folded into the bf16 weights, like H3-World, because this Space patches MiniMaxH3AttnProcessor and drives the transformer's live weights. Fold and unfold are the same operation with a sign, so the low-rank factors stay resident and set_active flips the mode in place inside the @spaces.GPU call, through one bf16 rounding.

The Steps slider still overrides the mode's count (4–50), so 50 steps + turbo LoRA or 8 steps without it are both reachable for the sake of the comparison.

Generation constraints

Fixed by the checkpoint: 24 fps, num_frames snapped to 17n + 5, no CFG and no negative prompt (it is guidance-distilled). H3-World was trained at 832x480; the conditioner's canvas list does not offer that exact size, so the default here is its nearest neighbour, 960x544.

The offered canvases are the cheap tier of each aspect ratio rather than the conditioner's full list. The mask term scales as sequence x caption rows, so a 1344x768 / 8 s request would want ~35 GPU-minutes — past what any visitor could book — and it is off-distribution for a LoRA trained at 832x480 anyway.

Measured

On this Space, driven over gradio_client, at 960x544 with a keyframe:

Startup is 95 s: the 66.3 GB download, the load, the LoRA merge, and the ZeroGPU pack. get_duration is fitted to exactly these three points — the unmasked block cost linear + quadratic in the packed sequence, the mask's own term linear in sequence x captions — and books ~15% over the fit. The default request books 348 s.

Examples

The three bundled first frames are extracted from acvlab/ABot-World-Explorer-500h (Apache-2.0), which is the same kind of third-person game footage H3-World was trained on. Each is paired with that clip's own manifest prompt.

Space variables

License

The LoRA is Apache-2.0, but usage is governed by the base model's license (MiniMaxAI/MiniMax-H3).