SRT omni / cross-vendor states

Frozen hidden states for the same 7,000-item manifest encoded through four multimodal hosts from four vendors, plus the fitted retrieval towers and the results computed from them.

A gallery encoded by one vendor is searchable by another vendor's text encoder at a rate statistically indistinguishable from native, and one linear map places image, audio and video in a single searchable space more effectively than one map per modality.

The two results

One tower beats one-per-modality (Qwen3-Omni-30B, 1,376 holdout items):

Derangement floor 695 +/- 21 against an analytic 688.

Cross-vendor retrieval matches within-vendor:

Both intervals contain 1.0. The claim is indistinguishability, not a specific retained fraction. Four of the twelve cross directions score above the within-vendor baseline of the gallery they search.

What retention does and does not tell you (added after publication)

The ratio above is close to insensitive to the vendor boundary it is named after, and later work established why. Both of its terms are limited by the caption head, not by the difference between vendors:

The encoders agree with each other about pictures roughly eight times better than any one of them connects its own captions to its own pictures. A ratio of two caption-head-limited numbers cannot report much about the encoders.

Tested directly by swapping the head, a test proposed by Dipankar Sarkar. Our first run of it was degenerate and is superseded: it put one shared all-MiniLM-L6-v2 in every vendor's caption slot, which removed the swapped side's dependence on the text vendor, so both swapped arms measured the same thing and the ratio test collapsed. Redone with a distinct caption tower per slot, five seeds, 1,000 held-out radiology items:

The gap between the two ratios is 0.0244 against a spread of 0.0476, so both legs move together when the caption head changes. That is what a shared bottleneck predicts. Read head_swap_multi_roco.json, not head_swap_roco.json.

Use the 0.8024 image-agreement number for portability claims. It is measured without a shared space, against a floor of 0.0007, and it is not throttled by the caption head.

A probe trained on one backbone reads another (added 2026-08-29)

Retrieval says two vendors agree about which picture. It does not say they agree about what is in it. On satellite imagery, a linear scene probe fitted on one vendor's frozen states and read on a different vendor's states through a ridge map fitted on train rows only, 17 land-use classes, 12 cross directions:

Transport costs 0.0024, and 4 of the 12 transported pairs beat the native target. The labels are weak and the number should not be quoted without that: scene classes are keyword-matched from the RSICD captions, the same shape of supervision ChestX-ray14 uses for its findings, and they are coarser than the medical case. The probe reads the image tower while the label comes from the caption, so the text side cannot leak the answer.

One frame for four backbones, and reading them together

Everything above is bilateral: one ridge map per ordered pair, twelve maps for four vendors. Fitting a single shared frame from all four at once (MAXVAR generalised CCA) gives eight maps instead of twelve and costs little:

Three vendors read together beat the best single vendor, on all four targets in both domains, eight for eight. Satellite +0.0345 mean (mistral 0.8310 to 0.8810), radiology +0.0165 (qwen3omni 0.9670 to 0.9940). The backbones are not redundant; each carries something the others do not.

Routing every hop through the shared frame also flattens the iterated-loop degradation to 1.0000 at 36 hops. Read that with the caveat recorded in the artifact: the composed map collapses to exactly the joint width, so later hops cannot lose dimensions that are already gone. Rank-matched pairwise maps rule out the trivial bottleneck reading, but the result may only restate that the joint route reuses one subspace per hop.

Read this before using the states

Centering is not optional. Raw cosine between unrelated items on these states is +0.869 and raw retrieval sits exactly at chance. Every number above is per-modality centered, and the fitted towers ship with their centering vectors. Skip the centering and you will measure the anisotropy.

Pool content tokens only. These states pool positions where the input id equals the modality's content token. Averaging every position lets the shared chat prompt dominate: unrelated items then sit at cosine 0.9987 and you are measuring the template.

Scope

• The four-vendor result is image only. Two of the four hosts carry no audio or video tower, so audio evidence is single-vendor and video is two-vendor.
• Audio is the thinnest leg despite the highest score: 164 holdout items, and 12.6% of the source clips downloaded empty because AudioCaps is YouTube-sourced. The surviving set is "clips still available", not a random sample.
• Each configuration is one fit on one split. Refitting moved the two-vendor retention from 0.960 to 0.968, so fit-to-fit variation is real; the bootstrap interval covers sampling only.
• All four hosts are trained on overlapping web-scale corpora. Convergence shows the structure survives a change of vendor, architecture and training run. The stronger reading, that independent minds would converge on it, remains untested and is not asserted here.

Contents

Vendors: qwen3omni (Qwen3-Omni-30B-A3B), gemma4 (gemma-4-31B-it), mistral (Mistral-Small-3.1-24B), aria (rhymes-ai/Aria).

Reproducing

Align on the manifest key, never on array position. The vendors were sharded differently and any row one encoder dropped shifts every later row, so positional alignment is silently wrong the moment failure counts differ.

python scripts/xvendor_fit_n.py \
  --vendor "qwen3omni:states/omni_states_s*.npz:manifests/omni_manifest_s*.json" \
  --vendor "gemma4:states/gemma4_states_s*.npz:manifests/xv_manifest_s*.json" \
  --vendor "mistral:states/mistral_states_s*.npz:manifests/img_s*.json" \
  --vendor "aria:states/aria_states_s*.npz:manifests/img_s*.json" \
  --out xvendor4.json

Sources: COCO val2017 images, AudioCaps clips, MSR-VTT videos. Written up in paper_nla.md section 11.9.