Empirical study: do language and vision models converge on the same representations? A 250-concept, 25-model benchmark says the answer is more structured than yes or no. Paper in prep for TMLR.
Python
1
16 commits
updated Apr 9, 2026
Cross-Modal Convergence Is Family-Local, Vision-Anchored, and Depth-Dependent: Evidence from a 250-Concept Benchmark
Abinash Karki (Independent Research) · March 2026 · Paper in preparation for TMLR
Do language models and vision models converge on the same internal representation of the world? The Platonic Representation Hypothesis says yes — that as models scale, modality should matter less.
This study tests that claim on Scale250, a 250-concept, source-balanced benchmark evaluated across a 25-model panel spanning language models, self-supervised vision models, contrastive vision-language encoders, and autoregressive VLMs. The answer is more structured than yes or no.
| Mode | What happens | Strength |
|---|---|---|
| Within-family | Language–language and VLM–VLM models agree strongly | RSA 0.61 (lang), 0.89 (VLM) |
| Bridge | VLMs sit closer to vision than language, regardless of architecture | Vision–language gap ~0.47 |
| Cross-modal | Language–vision alignment is weak | RSA 0.02; only 15/80 pairs survive FDR |
Expanding the benchmark from 20 to 250 concepts on the same model panel collapses language–image RSA from 0.24 to 0.02 while preserving within-family structure — small benchmarks materially overstate cross-modal convergence.
Additional analyses show convergence is depth-dependent (peaks at mid-to-late layers, not the terminal layer) and scale-independent in this panel (language model size 0.6B–4B shows no monotonic relationship with cross-modal alignment).
Left: RSA heatmap across the 25-model panel — strong within-family blocks, weak cross-modal signal.
Right: Expanding from 20 to 250 concepts collapses cross-modal estimates while preserving within-family structure.
The core panel of 22 models (8 language, 10 vision SSL, 4 contrastive VLMs) plus a 3-model autoregressive VLM extension. Evidence combines:
This repo is intentionally lean. It keeps the canonical manuscript, figure pack, manifests, core scripts, and small paper-facing summary artifacts in git. Heavy compiled outputs, raw embedding dumps, caches, and image payloads are excluded from git and described in artifacts/release_manifest.json.
These modes are intentionally separate. The checked-in data/data_manifest_250.json is the canonical analysis manifest and references image payloads excluded from git. For a fresh public rebuild, start from data/concept_roster_250_scaffold.json and docs/dataset_reconstruction.md.
# 1. Create environment
conda env create -f environment.yml
conda activate cross-modal-representations
# 2. Run release guardrail check
python src/release_checks.py
# 3. (Optional) Materialize heavy artifacts from local archive
python src/materialize_release_artifacts.py --from-local-archive --all
# 4. (Optional) Regenerate paper figures and renders
make paper-figures
make paper-render
Full guides: release_reproducibility.md · dataset_reconstruction.md · modular_runbook.md
data/ Canonical analysis manifest, concept roster, provenance metadata
docs/ Model panels, runbook, release/repro documentation
manuscript/ Canonical paper, figure pack, archival material
results/ Canonical paper artifacts and archived comparison outputs
src/ Core pipeline — extraction, compilation, robustness, scaling, figures
artifacts/ Manifest and checksums for large excluded artifacts
fixtures/ Smoke-test data for dataset rebuild verification
| Script | Purpose |
|---|---|
src/main_replication.py | Main extraction and replication entry point |
src/compile_results.py | Compile raw results into analysis-ready artifacts |
src/robustness_analysis.py | Bootstrap and permutation robustness analysis |
src/scaling_analysis.py | Model-scale relationship analysis |
src/analyze_arvlm_extension.py | Autoregressive VLM architecture extension |
src/generate_scale250_paper_figures.py | Generate all paper figures |
src/visualize_replication_results.py | Interactive result visualization |
src/release_checks.py | Pre-release guardrail verification |
Image payloads, full per-model raw embeddings, large compiled result artifacts, local run logs,
caches, smoke runs, model weights, and HF caches. These are indexed in
artifacts/release_manifest.json and can be restored from
.local_artifacts/ in a local working copy.
Earlier small-benchmark materials are retained for comparison context:
@software{karki2026crossmodal,
author = {Karki, Abinash},
title = {Cross-Modal Convergence Is Family-Local, Vision-Anchored,
and Depth-Dependent: Evidence from a 250-Concept Benchmark},
year = {2026},
url = {https://github.com/abinashkarki/cross-modal-representations},
license = {MIT}
}
Full citation metadata: CITATION.cff
Empirical study: do language and vision models converge on the same representations? A 250-concept, 25-model benchmark says the answer is more structured than yes or no. Paper in prep for TMLR.
Python
1
16 commits
updated Apr 9, 2026
Cross-Modal Convergence Is Family-Local, Vision-Anchored, and Depth-Dependent: Evidence from a 250-Concept Benchmark
Abinash Karki (Independent Research) · March 2026 · Paper in preparation for TMLR
Do language models and vision models converge on the same internal representation of the world? The Platonic Representation Hypothesis says yes — that as models scale, modality should matter less.
This study tests that claim on Scale250, a 250-concept, source-balanced benchmark evaluated across a 25-model panel spanning language models, self-supervised vision models, contrastive vision-language encoders, and autoregressive VLMs. The answer is more structured than yes or no.
| Mode | What happens | Strength |
|---|---|---|
| Within-family | Language–language and VLM–VLM models agree strongly | RSA 0.61 (lang), 0.89 (VLM) |
| Bridge | VLMs sit closer to vision than language, regardless of architecture | Vision–language gap ~0.47 |
| Cross-modal | Language–vision alignment is weak | RSA 0.02; only 15/80 pairs survive FDR |
Expanding the benchmark from 20 to 250 concepts on the same model panel collapses language–image RSA from 0.24 to 0.02 while preserving within-family structure — small benchmarks materially overstate cross-modal convergence.
Additional analyses show convergence is depth-dependent (peaks at mid-to-late layers, not the terminal layer) and scale-independent in this panel (language model size 0.6B–4B shows no monotonic relationship with cross-modal alignment).
Left: RSA heatmap across the 25-model panel — strong within-family blocks, weak cross-modal signal.
Right: Expanding from 20 to 250 concepts collapses cross-modal estimates while preserving within-family structure.
The core panel of 22 models (8 language, 10 vision SSL, 4 contrastive VLMs) plus a 3-model autoregressive VLM extension. Evidence combines:
This repo is intentionally lean. It keeps the canonical manuscript, figure pack, manifests, core scripts, and small paper-facing summary artifacts in git. Heavy compiled outputs, raw embedding dumps, caches, and image payloads are excluded from git and described in artifacts/release_manifest.json.
These modes are intentionally separate. The checked-in data/data_manifest_250.json is the canonical analysis manifest and references image payloads excluded from git. For a fresh public rebuild, start from data/concept_roster_250_scaffold.json and docs/dataset_reconstruction.md.
# 1. Create environment
conda env create -f environment.yml
conda activate cross-modal-representations
# 2. Run release guardrail check
python src/release_checks.py
# 3. (Optional) Materialize heavy artifacts from local archive
python src/materialize_release_artifacts.py --from-local-archive --all
# 4. (Optional) Regenerate paper figures and renders
make paper-figures
make paper-render
Full guides: release_reproducibility.md · dataset_reconstruction.md · modular_runbook.md
data/ Canonical analysis manifest, concept roster, provenance metadata
docs/ Model panels, runbook, release/repro documentation
manuscript/ Canonical paper, figure pack, archival material
results/ Canonical paper artifacts and archived comparison outputs
src/ Core pipeline — extraction, compilation, robustness, scaling, figures
artifacts/ Manifest and checksums for large excluded artifacts
fixtures/ Smoke-test data for dataset rebuild verification
| Script | Purpose |
|---|---|
src/main_replication.py | Main extraction and replication entry point |
src/compile_results.py | Compile raw results into analysis-ready artifacts |
src/robustness_analysis.py | Bootstrap and permutation robustness analysis |
src/scaling_analysis.py | Model-scale relationship analysis |
src/analyze_arvlm_extension.py | Autoregressive VLM architecture extension |
src/generate_scale250_paper_figures.py | Generate all paper figures |
src/visualize_replication_results.py | Interactive result visualization |
src/release_checks.py | Pre-release guardrail verification |
Image payloads, full per-model raw embeddings, large compiled result artifacts, local run logs,
caches, smoke runs, model weights, and HF caches. These are indexed in
artifacts/release_manifest.json and can be restored from
.local_artifacts/ in a local working copy.
Earlier small-benchmark materials are retained for comparison context:
@software{karki2026crossmodal,
author = {Karki, Abinash},
title = {Cross-Modal Convergence Is Family-Local, Vision-Anchored,
and Depth-Dependent: Evidence from a 250-Concept Benchmark},
year = {2026},
url = {https://github.com/abinashkarki/cross-modal-representations},
license = {MIT}
}
Full citation metadata: CITATION.cff