TT-VidT is a self-supervised video encoder built for motion. A DINOv3 ViT-B/16 processes every frame independently (the appearance path), while a compact Temporal Transfer pathway turns each frame into a few motion tokens that exchange information across time. This repository holds the pretrained TT3D encoder (195.5M parameters).

Model

• Encoder: 12 DINOv3 ViT-B/16 layers interleaved with 12 Temporal Transfer (TT3D) layers. Each TT3D layer runs block-causal attention over the frame's K = 8 motion tokens together with its 4x-downsampled spatial tokens, and writes the result back to the spatial stream.
• Pretraining objective: Diff Compression. A DiT decoder reconstructs every later frame from the first frame's features plus that frame's motion tokens, so the motion tokens carry what the first frame cannot explain.

Quick start

With transformers only (the model code ships in this repository):

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("KBlueLeaf/TTVidT", trust_remote_code=True).cuda().eval()

video = torch.rand(1, 8, 3, 256, 256, device="cuda") * 2 - 1    # [B, T, C, H, W] in [-1, 1]
with torch.no_grad(), torch.autocast("cuda", dtype=torch.float16):
    motion = model(video).motion_output                           # [B, T, 1, 768]

With the TT-VidT codebase (training, evaluation, feature extraction):

from ttvidt.hub import load_model

model = load_model("KBlueLeaf/TTVidT", device="cuda")
motion = model.encoder(video).motion_output

Loading needs no access to the (gated) DINOv3 base weights: every weight is in this repository.

Results

Frozen attentive probe on the motion embeddings, top-1 accuracy (%), mean of 3 seeds, 8 frames (evaluation protocol of the paper):

For the full study (24 architecture–objective pairs at matched scale, fine-tuning and diagnostics), see the paper.

Possible downstream uses

The encoder gives a compact sequence of per-frame motion tokens alongside the DINOv3 appearance features. Some directions we think are worth trying:

• From image models to video models: pair an existing image model with the motion token sequence to get a video model for understanding in the broad sense: classification, retrieval, captioning, question answering, or any other task.
• Generation and motion transfer: use the motion tokens as a conditioning signal for video generation, or take them from one clip and apply them to another subject or scene.

[!TIP] Further exploration and feedback are very welcome, and so are attempts at larger scale (bigger backbones, more data, longer training). Please open an issue or a discussion on GitHub or in the Community tab.

Related resources

Files

License

Apache-2.0. The spatial path is initialised from DINOv3, which is released under its own license.