MatSteer

Controlling scalar properties — band gap and volume per atom — of crystal structures generated by CrystaLLM, via activation steering. We modify the residual stream at a chosen layer during generation and measure whether the generated structures shift toward the target.

Three steering methods have been tried; see Steering methods. The short version of the results: the linear method moves volume per atom slightly and nothing else, and a PCA-subspace method built on the assumption that the property direction is curved does no better.

See tasks.md for what is currently in progress, and session_review.md for a standalone account of the PCA-subspace work.

Overview of the pipeline

CIF prompt ──► CrystaLLM (steered)  ──► generated CIF
                  │  hook @ layer 14, one of:
                  │    linear        h += alpha * v
                  │    pca_centroid  h += t * (centroid − z) @ W     (z = subspace coord)
                  │    pca_local     ditto, centroid rebuilt per prompt
                  ▼
            validate  ──► flags only, valid rows continue
                  ▼
            M3GNet-PES relaxation (relax_venv)        [band gap only so far]
                  ▼
            property prediction  ──► MEGNet band gap (megnet_venv, CPU)
                                     or volume/atom read from the structure

Each stage writes its own parquet keyed on (id, sample); downstream stages join rather than carry data forward.

Components

CrystaLLM

• crystallm_v1_large: 201.74M-param nanoGPT trained on CIFs.
• Prompt = composition (+ optional space-group header). Generation: temperature 1.0, top-k 10, 3 samples/prompt.
• Test split: CrystaLLM/cifs_v1_test.pkl.gz (~10k structures).

Steering methods

All three inject through one forward hook on model.transformer.h[layer], applied to every token position at every step (ChemSteer's inject_timestep='all'). Layer 14 is the working layer. steer_generate_cif.py --method picks between them, and each names its runs distinctly so they coexist in one generated_cifs/.

linear — h ← h + alpha * v, the normalized difference of class means, from compute_steering_vector.py → steering_vectors/<property>/layer{N}.parquet. Clean band-gap version: insulators ≥ 1.0 eV (n = 1,905) vs metals ≤ 0.05 eV (n = 575,593), raw_norm 17.49. Runs are named steered_*.

pca_centroid — assumes the property direction is curved rather than straight, so a single global direction aims off the manifold. Projects into the top-K PCA subspace of the training activations, interpolates a fraction t toward the centroid of a target class, and maps the change back:

z ← (h − mu) @ W.T                   project into the subspace
h ← h + t * (centroid_pca − z) @ W   interpolate, map the change back

Only the K principal directions move — the off-subspace residual passes through untouched, so the residual stream keeps supplying what the centroid does not specify. There is no negative class: the target centroid alone sets the destination. t = 0 is no injection (so the alpha = 0 run is the shared control for every method), t = 1 snaps the subspace coordinates onto the centroid, t > 1 extrapolates past it. Built by compute_pca_basis.py (once per layer) then compute_centroid_target.py (once per target) → steering_vectors/pca_centroid/. Runs are named steered_pca_*.

pca_local — the same equation, but the centroid is rebuilt per prompt from the --neighbours class members nearest that prompt, rather than averaging the whole class. Needs compute_centroid_target.py --save-bank. Runs are named steered_pcalocal_*.

Steering strength — alpha and t

Residual-stream norm at layer 14 ≈ 164/token, so for the linear method:

alpha = 80 is the numerical breaking point for the band-gap vector (RuntimeError: probability tensor contains inf/nan); the band-gap sweep tops out at 60. The density vector is better behaved and does run at 80.

Validity falls off well before generation breaks — 88.8% at alpha 60 against ~95% elsewhere — and the prompts that drop out are not a random sample, so per-alpha means must be compared on shared prompts rather than each run's own set.

The pca methods use t instead, on its own scale, and degrade at the same place regardless of property or target: ~95% valid at t ≤ 0.5, ~82-86% at t = 0.75, 12-20% at t = 1.0, and 0% at t = 2.0. Injecting at layer 9 instead of 14 breaks the model far sooner (0.7% valid at t = 0.75), since the perturbation has five more blocks to compound through.

Relaxation

• M3GNet-PES (matgl) via relax_venv (torch 2.4.1+cu121, V100/sm_70 compatible).
• crystallm_venv's cu130 build cannot run on the V100 — use relax_venv for M3GNet.

Band-gap prediction

• MEGNet-BandGap-mfi-MP-2019.4.1 (matgl), fidelity 0 (PBE/GGA), via megnet_venv, CPU.
• Fully deterministic (verified std = 0 over 10 repeats) — there is no value in averaging repeated predictions; randomness lives in generation, not prediction.
• Predicts from both the raw generated CIF and the relaxed CIF.

Key results so far

Band-gap steering — no effect at any strength

Over the full test set (10,286 prompts, 3 samples each, no space group in the prompt), each prompt contributing the mean of its valid samples:

Finding: steering does not move the band gap, in either direction. Each alpha paired against alpha 0 on the prompts they share (steering_ttest.py) gives Holm-adjusted p >= 0.60 and |Cohen's d| <= 0.017 on the relaxed gaps, and p = 1.00 with |d| <= 0.010 unrelaxed. At n ~ 10,000 the design would detect d ~ 0.04, so these are tight nulls, not inconclusive ones. Median per-prompt differences are order 1e-7 — the typical prompt does not move at all.

Two traps this analysis had to avoid, both of which produce fake effects:

• Comparing runs on different prompt sets. Generation and relaxation fail on different prompts each run, and because gaps are so skewed (median 0.005 eV, max ~8 eV) a handful of insulators entering or leaving dominates any mean. alpha 60's own-prompt mean of 0.154 looks like a real drop; against alpha 0 restricted to the prompts they share it is 0.1516 vs 0.1518. Always pair.
• Counting gap > 0. MEGNet returns small noise either side of zero for a metal, so that statistic counts the sign of the noise: the same test set read 13.6% and 94.8% before and after an unrelated parsing fix that moved the median from -0.0085 to +0.0048 eV. Use the 0.05 eV metal cutoff.

The one real difference between generated and real structures is in the tail, not the centre, and relaxation removes it. On the raw generated CIFs p99 is 3.24 eV for the originals against 2.17-2.30 eV for every generated run at every alpha; after M3GNet relaxation the originals sit at 3.14 eV and the generated runs at 3.01-3.17.

So the deficit is a property of the geometry the model emits, not of what it represents: raw generated structures have unrelaxed bond lengths that suppress the predicted gap, and relaxing recovers the tail. Steering moves neither version.

The same shows up in the means: unrelaxed, the steered runs average 0.12-0.15 eV against the baseline's 0.194, but relaxed they land at 0.154-0.190 against 0.188.

Volume-per-atom steering — small but real

The density vector is the only one that does anything. Paired against alpha 0 on log10 volume/atom: alpha 40 gives d = 0.128 (p = 6.9e-05), alpha 80 gives d = 0.165 (p = 9.1e-07), so the effect scales with strength. In absolute terms it is +0.4% and +0.7% on the median.

It exists only as a per-prompt nudge, not a moved distribution: unpaired Welch on the same data gives p ~ 0.6-0.8, and Levene ~ 0.97 shows the spread does not change.

PCA-subspace centroid steering — no effect, on either property

Nine arms testing whether a curved-path method beats the linear one: two properties, two targets each, two layers, plus a per-prompt local-centroid variant. Full table in analysis/crystallm/v1_all/test/steering_runs.csv (steering_ttest.py --all).

Read cohens_d, not the p-values. With ~980 paired prompts a 0.5% shift reaches p = 1e-20 while |d| stays at 0.14. Every arm sits at |d| ≤ 0.17, the same range as the linear method's.

Three explanations were tested and ruled out:

• Reach. Target 22.5 needs +0.068 log10 from the control median, target 30 needs +0.192 — 2.8x further. Both produced the same ~0.002 absolute move, so the displacement does not scale with where the centroid sits. Head-to-head, p = 0.21.
• Degradation. The cleanest nulls are at the model's best operating points: band gap t = 0.75 is 85.9% valid with p = 0.55; density t = 0.5 is 95% valid with d = 0.12.
• Centroid choice. pca_local aims a median 10.63 from the global centroid — 2.5x the global centroid's whole offset from the corpus mean — with matched displacement and matched model damage. Result: p = 0.65 at t = 0.25, p = 0.17 at t = 0.5.

Relaxation erases the raw band-gap drift, exactly as it did for the linear method: +0.03 eV in raw CIFs becomes −0.001 to −0.013 after M3GNet, every p ≥ 0.56. Steering perturbs the written cell in a way MEGNet reads as slightly gapped and relaxation settles it out. Always read the relaxed column.

Why it fails: the target class is not a place. Its centroid sits 4.22 from the corpus mean while class members sit a median 13.43 from that centroid — signal-to-spread 0.31 — and the class matches a single Gaussian of the same covariance (13.43 vs 14.02 median, near-identical tails), so it is one diffuse cloud, not several lobes with the average falling in the gap. Structures at 30 ų/atom are scattered through the same region as everything else, so "interpolate toward the class" has nothing to aim at. That explains every null without any claim about geometry, and two genuinely different destinations giving the same non-result points upstream of centroid choice: layer-14 activations may carry a decodable signal for these properties without a controllable one.

A trap this analysis had to avoid, on top of the two above: parse success is not validity. Measured over every CIF that parsed, density t = 1.0 looked like a large win (median 19.29 → 25.05, d = 0.677). Only 12.5% of those were chemically valid and the failures were malformed grammar, not oversized cells; on valid rows d = 0.053. Anything computed from parsed text shows a false signal at any strength that breaks the CIF grammar, and the apparent effect grows as the model degrades.

Caveat worth knowing: band gap steers on composition-only (nosg) prompts and density on prompts that dictate the space group, because that is where each property's alpha = 0 control exists. Each is internally valid, but the two are not a like-for-like cross-property comparison.

LayerNorm is not why steering fails

CrystaLLM is pre-norm (x = x + attn(ln_1(x))), so a vector added to the residual stream at layer 14 is normalised by block 15 before anything reads it. That is the one mundane explanation that would invalidate every null above, so it was measured directly (analysis/layernorm_survival.py, forward passes only, no generation):

The injection survives. It reaches the logits at 13–23% relative magnitude, and the part LayerNorm annihilates for free — whatever lies along the uniform direction — is under half a percent. So the sweeps applied what they were meant to apply, the model's output distribution really is being perturbed, and the property still does not move. The nulls stand as measured.

Both rows are full-sequence. _model.py runs lm_head on the last position only unless targets is passed, so a naive model(x) reports the logit change for one token rather than the sequence — the scripts pass targets for exactly this reason.

No layer translates the property in a single step

Steering at layer 14 does nothing and the injection is not being erased, so the next question is whether some other layer carries a usable handle. A generation sweep per layer would cost days; analysis/layer_causal_probe.py answers a narrower version with forward passes. CrystaLLM writes numbers digit by digit (_cell_volume 160.9257 → ['1','6','0','.',…]), so the model's belief about the volume is a distribution over digit tokens at known positions. Teacher-force a real CIF, inject at layer L, compare against clean.

Scope, and it matters — the probe is teacher-forced, so it measures a single-step effect from a well-formed prefix. It cannot see anything that arises from errors compounding over a long autoregressive generation, and that turns out to be where the one visible behavioural effect lives (below). Read it as "does layer L shift the next digit given a correct prefix", not as "does layer L control the property".

Linear vector, injected at a fixed fraction of each layer's own residual norm (--frac 0.21, matching alpha=40 at layer 14):

The injection lands on the right tokens and fails to move them. Selectivity — KL at the volume digits over KL everywhere else — is 2.4–3.1 across layers 1–15, so the volume tokens are perturbed ~2.6× more than the rest of the CIF. This is not undifferentiated noise. But the magnitude does not shift: the integer-digit count is exactly zero at every layer, and that is the term carrying orders of magnitude (a volume goes 90 → 900 by gaining a digit). All movement is in the leading digit, at most 0.057 log10, and negative at 14 of 16 layers even though the vector points low → high. Target 30 needs +0.192. pca_centroid behaves the same way at layer 14: d_log10_vol = −0.0009 at t=0.5 and +0.0011 at t=1.0, with the digit count flat at both.

So in a single step the injection scrambles the digit distribution rather than translating it.

The one real behavioural effect is drift, not steering

pca_centroid at t=1.0 did make the model write much bigger volumes — that was reported here for a while as a large win. Splitting those generations by validity shows what it was:

The entire effect lives in structures that parse but are chemically invalid. The valid ones do not move. The probe agrees and explains why: at layer 14, t=1.0, selectivity falls to 0.598 — the injection now disturbs non-volume tokens more than volume tokens (KL 1.672 vs 0.999), which is the fingerprint of general degradation rather than targeting. At layer 9, t=1.0 the digit count finally does move (−0.165) with KL of 10.3, i.e. the grammar coming apart.

A strong injection therefore does change the number the model writes, by pushing it off-distribution over hundreds of autoregressive steps, and ~87% of what it then writes is not a valid crystal. That is a different phenomenon from steering, and it is invisible to a teacher-forced probe by construction.

Two checks that the probe tracks reality where its scope allows: it predicts a negative shift at layer 9 under the linear vector and the layer-9 generation sweep did come out negative (d = −0.084); it predicts ~0 at layer 14 and those sweeps gave d ≈ +0.11, i.e. nothing.

Read selectivity and d_log10_vol together — selectivity alone says only that the injection arrives, not that it directs anything.

Spectral co-clustering "incoherence" — a construction artifact

• spec_cocluster_analysis.py runs Dhillon spectral co-clustering on per-layer embeddings and reports subspace incoherence epsilon = ||P_k P_i||_op across layers 1–14 and K = 10/50/150/300.
• All reported epsilons are 1e-14 to 1e-17 (machine zero): analysis/crystallm/v1_all/full/all/cocluster_results/v1/layer*/[K]_clusters/pairwise_epsilons.csv.
• This is trivial/circular, not a real property. The code sparsifies X by zeroing every dimension a point isn't assigned to (X_sparse), forcing disjoint coordinate supports per cluster — so the subspaces are orthogonal by construction and epsilon ≈ 0 always. To measure the real incoherence, build the per-cluster SVD bases from X_std (all 1024 dims) instead of X_sparse.

Directory layout

How embeddings/ and analysis/ are organised

Built by utils.py:analysis_dir() / analysis_root() / embeddings_paths().

embeddings/<dataset>/<model>/<variant>/cif_layer{0..N}[.parquet]

analysis/<model>/<dataset>/<variant>/<partition>/
├── <global outputs>            # span every layer: probe tables, separability CSVs
└── plots/
    └── layer{0..N}/            # one figure per layer

analysis/corpus/<dataset>/<partition>/     # never touches a model -- see below

The model level sits in different places in the two trees, on purpose. analysis/ puts it first because an analysis directory is a per-model deliverable: reading one model's results should not mean walking every dataset tree. embeddings/ has the opposite pressure — a layer's shards for one corpus are read together whichever model wrote them — so it keeps <dataset> first.

analysis/corpus/ is not a model. Corpus property histograms, Wyckoff/space-group counts, MP hull coverage and metadata property coverage never load a model and are byte-identical whichever one is registered, so they are written once here instead of duplicated under every model. analysis_dir() and analysis_root() accept model=CORPUS_DIR; add_partition_args does not, since nothing can load corpus as weights.

Note that variant=None does not mean corpus. The steering tables pass variant=None — they come from generation, not from reading a CIF variant — and are entirely model-dependent. Inferring one from the other would file every steering result under corpus/.

1. dataset — which corpus the CIFs came from.

They overlap by only 58,650 ids, and each is sampled independently, so a t-SNE from one is not comparable to a t-SNE from the other — different point clouds give different shapes no matter what the colouring shows.

2. model — which model produced the hidden states. Registered in utils.MODELS.

The same corpus read by two models gives two unrelated sets of vectors: same ids, same CIF text, different dimensionality, no shared basis. So a layer index does not mean the same thing in each — layer 7 is 7/16 of the way through crystallm and 7/32 through llamat2 — and a steering vector, PCA basis or manifold fitted on one cannot be applied to the other. Nothing in the repo mixes two models in one artifact; the path level is what keeps that true.

--model names the model; --ckpt-dir is the filesystem path to its weights. The two were both called --model before a second model existed.

3. variant — which CIF text the embeddings were extracted from.

full cannot answer whether the model represents symmetry: the space group is written verbatim into the text, so a probe just reads the copied token back. That is what nosym is for. full_stale records when something was made, not which directory a script pointed at — figures predating the restructure live there and stay there.

Outputs that read metadata but never embeddings (the property histograms) pass variant=None and drop this level entirely. Those same outputs also pass model=CORPUS_DIR, so they land at analysis/corpus/v1_mp/all/: they would be byte-identical under full/ and nosym/, and under every model.

4. partition — which slice of CrystaLLM's own train/val/test split, one of all / train / val / test. analysis_dir has no default and add_partition_args makes --partition required, on purpose: 89.6% of labelled structures are in the model's training set and only 0.45% in its test set, so a result on all cannot separate learning from memorisation. Filter with utils.py:filter_partition().

5. inside a partition — files directly in the partition dir are global (they span every layer); plots/ holds figures, with plots/layer{N}/ for per-layer ones and cross-layer comparisons sitting directly in plots/.

Worked example:

analysis/crystallm/v1_all/nosym/val/plots/layer5/…
        crystallm on symmetry-stripped v1_all embeddings, validation split, layer 5

analysis/llamat2_cif/v1_mp/crystal_uncond/val/…
        llamat2-cif on crystal strings in the unconditional generation prompt

analysis/corpus/v1_mp/all/metadata_mp_*.png
        MP metadata histograms — no variant level and no model level, since no
        embeddings were read and no model was loaded

Key scripts

Data conventions

• Ground-truth band gap = metadata.parquet:dos_electronic.band_gap (eV, 582,596 non-null), from NOMAD results.properties.electronic.dos_electronic.band_gap[0].value. electronic.band_gap is the same numbers (identical non-null set, correlation 1.0). Not energy_lowest_unoccupied - energy_highest_occupied — those are raw Joules and their difference is the corrupt LUMO-HOMO gap. The three scripts that used it have been quarantined in scripts/legacy/ (compute_bandgap_steering.py, analyze_steering_norms.py, bandgap_percentile_stats.py) — kept because published numbers came from them, but nothing imports or runs them; see scripts/legacy/README.md. Note utils.py:DEFAULT_LABEL_COLS asks for band_gap_ev, which does not exist in metadata.parquet and is silently dropped, so load_labeled_embeddings never returns a gap — join it yourself.
• Symmetry must be restored before parsing. A CIF lists the asymmetric unit plus a space group; the full cell comes from applying that group's operators, and CrystaLLM writes a placeholder operator list holding only the identity. Parse without restoring them and you get the asymmetric unit alone — wrong atom count, wrong formula, volume/atom inflated by the multiplicity factor. Always go through utils.py:postprocess() (or restore_symmetry_operators()), never CrystaLLM's replace_symmetry_operators directly: the upstream regex assumes unindented CIFs, which is true of generated ones and false of the pymatgen-written references, and a miss returns the text unchanged without raising. This silently corrupted the test-set baseline (87.5% wrong atom counts, 52.6% wrong formula) and the density_atomic_raw column (2.67x high at the median) until 2026-08-19.
• CIF strings live in exactly two places: generated_cifs/ (cif_steered, raw) and relaxed/ (cif_relaxed, relaxed). validation/ holds flags only. Downstream scripts (novelty, relax, predict) join flags with the CIF source on (id, sample) and process is_valid == True rows only.
• steering_results/<property>/property_predictions/<stem>.parquet: one file per source run — id, sample, predicted_bandgap_ev (relaxed), predicted_bandgap_ev_raw (unrelaxed).
• "best-of-3": per prompt take the max predicted gap over samples, then mean across prompts.
• Ground-truth volume per atom = density_atomic_v1.parquet, not MP's density_atomic column: that covers only 58,650 of the 2.29M v1 ids. The same quantity is in every CIF — _cell_volume over the atom count from _chemical_formula_sum, both full-cell — so it is read by regex with no pymatgen parse (utils.py: density_atomic_from_text). All 2,285,719 structures, 0 unreadable, median 19.10 ų/atom.
• A steering run is keyed by (layer, target, strength), and family is part of its identity. Any two of those alone collide, silently: two targets swept over the same t, or one target swept at two layers, share a key and one overwrites the other in discovery. sg and nosg are different prompt sets, so a run may only be paired against a control of its own family — only nosg was ever generated at alpha = 0 for band gap, so the four sg runs are listed without paired statistics rather than compared against the wrong control.
• Results tables follow one schema, utils.py:RESULT_COLUMNS, written through write_results_table() which raises on a missing core column. unit names the value scale (a median is eV for band gap, log10 ų/atom for density) and strength holds alpha for the linear method and t for the pca ones, so one column orders every sweep. Never write one of these files by hand.

LLaMat-2 was trained on a different CIF layout than the one we feed it

Established 2026-09-17 by reading the LLaMat repo (llamat/, an untracked clone of M3RG-IITD/llamat) and comparing it against our own corpus files. This affects llamat2 embeddings only; nothing about crystallm changes.

What LLaMat-2 saw. Its CIF instruction tuning (llamat/src/cifs/notebooks/cif_tasks.ipynb) poses seven question tasks whose input is a whole CIF file, unedited, as Below is a CIF file.\n{cif}\n{question}. 14,046 such records survive in that repo's saved notebook outputs, and all 14,023 space-group lines among them read 'P 1' — no exceptions. They are pymatgen-written P1 files: quoted 'P 1', _symmetry_Int_Tables_number 1, the identity operator alone, 8-decimal numbers, one line per atom in the cell with multiplicity 1 and unique site labels (Te0, Te1, …). Matching that, the authors' evaluation scores the space-group question at 99.1%, i.e. the answer was almost always the constant P1. The model therefore has no training that teaches it to expand symmetry operators.

What we feed it. extract_cif_embeddings.py deliberately gives both models identical text: CrystaLLM-corpus CIFs through preprocess_cif. That text differs from LLaMat-2's training layout in four ways:

The second row changes meaning, not just surface form: the model reads one Fe line and has no reason to infer two atoms.

Which of those come from CrystaLLM's preprocessing. Checked by pulling NOMAD_GjZzUnOlEIJpuQwmit4GsRKZjrAr out of both cifs_v1_orig.pkl.gz (3,551,492 entries, unfiltered) and the prep pickle. Only the rounding, the atom-property block, the data_ rename and the operator collapse are added by bin/preprocess.py. The symmetry-unique atom layout and the real space group are already in the originals, so converting is needed either way. One consequence for parsing: the original lists all 8 operators for Pmmm and pymatgen reads it correctly on its own (4 sites, BaFe₂P), while only the preprocessed file needs utils.py:postprocess() first.

Conversion. Convert from cifs_v1_orig.pkl.gz, not the prep pickles: it keeps 8 decimals and full operators, and has no atom-property block to strip.

s = Structure.from_str(orig_cif, fmt="cif")          # prep pickles: postprocess() first
s = Structure(s.lattice, s.species, s.frac_coords)   # fresh labels: Ba0, Fe1, Fe2, P3
llamat_cif = s.to(fmt="cif")

Verified on that structure with pymatgen 2024.11.13 (crystallm_venv): the result matches the training layout line for line. Skipping the second line leaves duplicate Fe1 labels. cifs_v1_orig is larger than the 2.29M v1 corpus because it is not deduplicated — filter it by the ids in the split pickles to keep the same structures.

Token budget. Measured on 1,000 CIFs from the LLaMat repo's own test set with LLaMat-2's tokenizer (llamat/src/tokenizer_l2.model): a raw CIF is 332 tokens plus ~44 per site (median 767, max 1,244 at ≤20 sites); the compact "crystal string" the model generates is median 193. At the 4,096 context this truncates at roughly 85 atoms, and conversion to P1 pushes structures over that sooner than their CrystaLLM files suggest. Numbers tokenize digit by digit, so most tokens in a CIF are coordinate digits and a mean pool is dominated by them.

Two consequences for variants and probes.

• After conversion the H-M line is always 'P 1' and every multiplicity is 1, so --variant nosym removes almost nothing and the Wyckoff leak noted in strip_symmetry disappears. A nosym llamat2 run is not the same experiment as a nosym crystallm run.
• _cell_volume and _chemical_formula_sum survive conversion, so the density parse ceiling documented above still applies. The space group does not survive, so a space-group probe on converted text has no parse shortcut.

Reading vs writing. LLaMat-2 reads CIFs but never writes them: its generation tasks emit a compact crystal string (lengths / angles / element and fractional-coordinate lines), which is turned into a CIF in Python afterwards. A direction fitted on CIF-reading activations may not transfer to generation. For steering work, embed crystal strings placed where the model's answer goes. Every generation prompt, the crystal-string encoder and a parser are in llamat/cif_prompts/ (README.md, cif_prompts.py, examples.txt), verified against the authors' own saved records.

The released checkpoint, inspected 2026-09-17. m3rg-iitd/llamat-2-cif is public and ungated; the weights are in models/llamat2_cif (gitignored, 27 GB of safetensors — the repo also ships identical .bin copies and a separate adapter_ckpt/ LoRA for the Gruver conditional-generation route, neither of which we use). Four facts from its config.json and tokenizer that the code has to respect:

The vocab gap is a packaging artifact, not something to defend against. All five added tokens (<CLS> <SEP> <EOD> <MASK> <PAD>, ids 32,000–32,004) sit past the end of the embedding matrix, so none of them can be embedded — they come from Megatron-LM's tokenizer, which adds them by default (llamat/Megatron-LLM/megatron/tokenizer/ tokenizer.py:362), and the Megatron → HF conversion carried them over without resizing the embedding.

They never appear in our input: checked over 3,000 real prompt+crystal-string sequences, the highest id produced is 29,999. So we simply do not use one. The only place the question arises is the filler for batch padding, and there it is pure bookkeeping — excluded from the pooled mean, and never attended to because padding sits at the end of a causally-attended row — so the filler just has to be a valid embedding row. LlamatBackend uses 0, the same value the CrystaLLM backend uses. Asking tokenizer.pad_token_id was the wrong question; the authors' own inference code sets its own pad token too (src/cifs/crystal-text-llm/condtional_generation.py:112).

The wrapper question is now settled, near enough. It was open because the committed training pipeline builds ChatML (<|im_start|>system …) while every inference script the authors wrote uses {system} input-{input}output-. The released tokenizer contains no <|im_start|> or <|im_end|> and ships no chat_template, although src/ft_pipeline.sh passes --vocab_extra_ids_list "<|im_start|>,<|im_end|>" when it converts a ChatML-trained model. So this checkpoint was not ChatML-trained, and --wrapper notebook is the default. That is strong evidence, not proof; --wrapper chatml remains available.

Token budget against the real tokenizer (measured on 3,000 v1_mp and 2,000 v1_all structures, replacing the earlier CIF-based estimate). The unconditional prompt is exactly 203 tokens, and a crystal string costs 18.0 tokens per site, so about 102 sites fit:

MP structures are three times larger than the v1_all median, so the smaller corpus is the one that loses more to the context limit — and what it loses is systematic: the dropped cells have a median of 142 sites against 26 for those kept. --on-overflow skip therefore biases toward small structures and the skip count belongs in any result. It is still the right default, because a truncated crystal string describes a cell with fewer atoms, so its embedding is not that structure's.

Environments

• CrystaLLM/crystallm_venv — generation and crystallm embedding extraction (cu130; GPU generation only).
• relax_venv — M3GNet-PES relaxation (cu121, V100-compatible).
• megnet_venv — MEGNet band-gap prediction (CPU).
• llamat_venv — llamat2_cif extraction (torch 2.4.1+cu121 + transformers). Built on the same torch as relax_venv because that is the build known to run on these V100s. run.sh selects it with VENV="llamat"; the older extraction slurms pick it whenever MODEL != crystallm, overridable with VENV=<path>.