Figure 1. Swift Sampling efficiently identifies temporal surprises in videos by measuring how much a frame deviates from the trajectory predicted by its preceding context. Using a Taylor expansion of visual features, we select frames with the largest residuals within their temporal neighborhood as keyframes. Top: Temporal surprise captured using Taylor residual over time. Bottom: input frames and frames selected by $\color{orange}\textsf{Uniform sampling}$, $\color{olive}\textsf{Cosine Uniqueness}$, and $\color{green}\textsf{our method}$. Swift Sampling captures the video's most informative frames with 30× less overhead than Cosine Uniqueness, while delivering a +12.5% improvement on VQA tasks on long videos with tight frame budgets.

🔖 Table of Contents

• ✨ Highlights
• 💡 Motivation
• 📐 Method
• 🔬 Analysis
• 📦 Installation
• 🚀 Quickstart
• 📊 Evaluation
• 👏 Acknowledgement
• 📜 Citation

✨ Highlights

• A training-free frame selection algorithm that operationalizes predictive coding by scoring each frame by its Taylor-series residual in the VLM's latent space — no auxiliary model and no video-specific tuning, i.e. hyperparameter-light and efficient.
• State-of-the-art over uniform sampling and prior training-free methods across different VLM backbones, on video question answering, token compression, and over ten reasoning tasks across diverse video lengths.
• A systematic analysis of design choices that connects latent temporal dynamics to frame selection.
• Especially powerful for long videos with limited frame budgets, with gains of up to +12.5 points in accuracy.

💡 Motivation

Most frames in long-form video are redundant; the critical information resides in temporal surprises — moments where the actual visual features deviate from their predicted evolution. Inspired by the human brain's predictive coding, Swift Sampling models a video as a differentiable trajectory in the visual latent space, computes the velocity and acceleration of its features, and applies a Taylor expansion to project the expected path of subsequent frames.

📐 Method

Given a video with $T$ frames and a budget $K \le T$, Swift Sampling selects the $K$ most informative frames using a Taylor-series expansion in the VLM's latent space. For each frame $t$, we form a Taylor predictor from the $N$ preceding frames that captures the velocity (first order), acceleration (second order), and jerk (third order) of the feature trajectory. The Taylor residual, the distance between the predicted and the observed feature, serves as a per-frame informativeness score; we select the $K$ local maxima of this residual sequence. The residuals are computed on the intermediate representations of the VLM's vision encoder.

Figure 2. Each frame is represented on the $\color{red}\textsf{latent feature trajectory}$, where we apply Taylor expansion over preceding frames to $\color{green}\textsf{predict the next frame}$ feature. The residual between the prediction and the actual feature measures how much the trajectory deviates from a smooth continuation. Frames with large residuals correspond to temporal surprises, e.g., seal suddenly emerging from the ice, which Swift Sampling effectively captures.

🔬 Analysis

Taylor residuals can be computed at any layer of the vision encoder. We analyze this choice and use the layer that yields the most informative residuals at the lowest cost.

Figure 5. Taylor residual across different layers. $\color{green}\textsf{Key features (green)}$ yield lower residuals than $\color{orange}\textsf{hidden output features (orange)}$ at every layer, and the residual is smallest at the earliest layers, indicating that they are the most predictable from their temporal context. We use layer ℓ = 0 key features throughout our experiments.

📦 Installation

git clone https://github.com/kim-dahye/SwiftSampling.git
cd SwiftSampling
pip install -r requirements.txt

🚀 Quickstart

bash scripts/eval_swift.sh llava_onevision longvideobench_val_v

Pool size $T$, kept frames $K$, and Taylor order $N$ are controlled via environment variables (defaults reproduce the paper):

TARGET_FRAMES=32 TAYLOR_ORDER=3 \
  bash scripts/eval_swift.sh llava_video videomme

Swift Sampling also exposes a standalone selector that operates on any per-frame feature sequence:

import torch
from swift_sampling import swift_sampling_select

# frame_repr: [T, D] tensor — one feature vector per candidate frame
frame_repr = torch.randn(128, 768)
selected = swift_sampling_select(frame_repr, target_frames=32)

📊 Evaluation

To reproduce Table 1 ($T{=}128 \to K{=}32$):

bash scripts/eval_swift.sh llava_onevision videomme
bash scripts/eval_swift.sh llava_onevision mlvu
bash scripts/eval_swift.sh llava_onevision longvideobench_val_v
bash scripts/eval_swift.sh llava_video    videomme
bash scripts/eval_swift.sh llava_video    mlvu
bash scripts/eval_swift.sh llava_video    longvideobench_val_v

Backbones: LLaVA-OneVision-7B and LLaVA-Video-7B. Benchmarks: Video-MME, MLVU, LongVideoBench.

👏 Acknowledgement

SwiftSampling builds on LLaVA-NeXT and lmms-eval.

📜 Citation

@article{kim2026swift,
  title   = {Swift Sampling: Selecting Temporal Surprises via Taylor Series},
  author  = {Kim, Dahye and Sachdeva, Bhuvan and Uppal, Karan and
             Gupta, Naman and Balasubramanian, Vineeth N. and Ghadiyaram, Deepti},
  journal = {arXiv preprint arXiv:2605.22678},
  year    = {2026}
}