Qwen3-32B Token-Similarity Activation Triplets

This dataset contains 50,000 transition triplets generated from internal residual activations of Qwen/Qwen3-32B and encoded as image-like token-token similarity maps for Activation-as-View / world-model experiments.

The dataset is intended for research on whether language-model activation states can be converted into visual observations suitable for JEPA-style or latent world-model training.

Contents

• data/qwen3_32b_train_token_similarity.h5: canonical HDF5 dataset with 50,000 triplets.
• reports/generation_report.json: generation summary returned by the Modal job.
• reports/validation_report.json: local HDF5 integrity and statistics report.
• reports/dynamics_baselines.json: identity/mean/delta baselines for the transition task.

Schema

The HDF5 file contains:

Important HDF5 attrs:

model = Qwen/Qwen3-32B
encoding = token_similarity_v1
layers = [10, 20, 30]
action_encoding = embedding_pca256
obs_shape = [3, 64, 64]
split = train

Encoding

token_similarity_v1 is computed per selected transformer layer:

1. collect residual activations [seq_len, hidden_dim];
2. center each token vector;
3. L2-normalize token vectors;
4. compute cosine token-token similarity z @ z.T;
5. map similarities from [-1, 1] to [0, 1];
6. resize to 64 x 64;
7. stack layers [10, 20, 30] as the 3 observation channels.

This avoids assigning arbitrary spatial meaning to hidden dimensions and preserves relational token structure.

Validation Summary

Local validation on the full HDF5 file:

obs_shape       = [50000, 3, 64, 64]
actions_shape   = [50000, 256]
next_obs_shape  = [50000, 3, 64, 64]
obs mean/std    = 0.3968 / 0.1920
next mean/std   = 0.3975 / 0.1917
nonfinite       = 0

Dynamics baselines:

identity_mse mean       = 0.0051056
mean_next_mse mean      = 0.0188126
identity / mean ratio   = 0.2714
obs_next_cosine mean    = 0.9866
delta_l1 mean           = 0.0511

The identity baseline is strong, so downstream training should always report model MSE relative to identity MSE. MSE alone is not sufficient.

Intended Use

• Activation-as-View experiments.
• JEPA-style or latent world-model training over language-model activation states.
• Transition prediction and violation-of-expectation experiments.
• Ablations comparing activation-to-image encodings.

Limitations

• The dataset is generated from Qwen/Qwen3-32B; users must comply with the upstream model license and terms.
• The transition horizon is local (t -> t+1), so identity-copy baselines are strong.
• Prompt sources are synthetic/stratified scaffolds used for experimental comparability, not a natural benchmark distribution.
• The dataset is not intended as a text corpus or as a benchmark of Qwen model quality.

Citation / Project

This dataset was produced for the Activation-as-View research prototype in CODE-JEPA.