README
Hypotheses about small models, and what killed them
Every claim here is a thing I believed, built, and then measured. Most of them died. The ones that survived are narrower than what I set out to prove, and the raw logs — including the runs that crashed and the numbers I had to retract — are in this repository.
The tools are here. The training engine is not, and will not be. Every claim below reproduces from the tools alone.
The scoreboard
| hypothesis | verdict | the number |
|---|---|---|
| A model can live in the GPU's on-chip SRAM | dead | persisting L2 buys 1.04×; shared-memory staging 0.98× |
| Keeping attention on chip cuts the traffic | bounded | attention is 10–24% of weights, not a third |
| int4 halves the streaming cost | dead | 10.9–13.1% round-trip error, uniform across families |
| The error hides in the attention projections | dead | it is uniform; splitting families saves nothing |
| Hiding the disk makes training fast | dead | disk is 3% of a step on a GTX 1060 |
| A read pipeline will cover the latency | dead | 6.5% hit rate; the intercept did not move |
| More reader threads is more throughput | dead | this NVMe peaks at queue depth 3, collapses at 8 |
| A rented GPU behaves like a local one | dead | 3.4× direction asymmetry; pinning buys zero |
| Sampling a small model 100× and verifying | dead | +0.5pp; a perfect picker would buy +5.5 |
| Letting the model run its code and retry | ALIVE | 62.5% → 78.5%, and it beats a 3.5× bigger model |
| Small models reason if trained right | partly | training beats size at one hop; nobody does three |
| A model can write its own tests | dead, and harmful | 50.6% of them are wrong; the loop drops 85.0% -> 80.0% and stops converging |
Written up
The signal went up. The code got worse. -- the three experiments above as one story: sampling 100x buys +0.5, execution feedback buys +16, and self-written tests take 5 back while every metric you are watching improves.
The threads
Sampling buys +0.5. Execution feedback buys +16. — the newest and the largest effect here. 100 independent samples from a 4B, picked by a verifier, bought half a point; a flawless picker would have bought 5.5, because the ceiling is the model's distribution and nothing passes between independent draws. Handing the same model its own traceback and letting it try again took it from 62.5% to 78.5% — past the perfect-picker ceiling, at one thirteenth of the tokens, and past a 14B answering once. Then it stops, and why it stops is the finding that matters: 14.5% of a strong model's answers are wrong and pass their test, so the feedback channel goes silent. What bounds the loop is test coverage, not model capability.
Letting the model write its own tests to fix that makes it worse: half the generated asserts are wrong, and the loop falls to 80.0% while its own signal climbs monotonically. The signal converges and the truth does not.
Can a model live in the GPU's on-chip SRAM? — three CUDA probes on an A6000. Persisting L2 buys 1.04×, staging weights in shared memory buys 0.98×, and one grid-wide barrier costs 1.02 µs, capping a 24-layer sharded model at 10,297 tok/s from synchronisation alone. Placement buys nothing; capacity is the whole constraint. Includes three ways those numbers lied first.
What decides whether a small model can reason? — synthetic deduction over fictional predicates, so no memorised fact can help. The first version of this task leaked and its numbers are retracted in place: three surface features separated the labels at up to AUC 0.807, which is exactly where the best result had landed. On the fixed task, a 0.8B Qwen3.5 matches a 2.03B Qwen3 and both beat a larger SmolLM2 — training outranks size at one inference step — and none of the four chains three rules.
Pushing a 24B through a GTX 1060 — a frozen 24B does not fit in 6GB and can still be fine-tuned there, streamed off the disk, one layer at a time. It works, and it is slow for a reason no optimisation of mine could remove: one optimizer step takes 81 minutes on a GTX 1060 and 5 on an A6000 held to the same 6GB. The difference is the silicon, not the engine.
What was measured
Mistral-Small-24B-Instruct-2501, PiSSA rank 16 over 280 matrices, seq 512, MetaMathQA. Three machines, one engine.
| step | per sample | cross-entropy, step 1 | |
|---|---|---|---|
| GTX 1060 6GB, batch 26 | 4870s (81 min) | 187.3s | 0.5735 |
| A6000 ballasted to 6GB, batch 60 | 968s (16 min) | 16.1s | 0.5682 |
| A6000 ballasted to 6GB, batch 8 | 299s (5 min) | 37.4s | 0.5434 |
The cross-entropies agreeing across three configurations on two machines is the strongest correctness evidence here. The engine computes the same thing everywhere; only the clock changes.
1. int4 is dead, and not for the reason everyone assumes
The obvious way to halve the streaming cost is to narrow the weights. int4 gives 3.76x fewer bytes against fp16's 2. Measured round-trip error on the real weights, group 64, three complete layers:
| family | share of a layer | int4 | int8 |
|---|---|---|---|
| gate_proj | 30.2% | 10.94% | 0.60% |
| up_proj | 30.2% | 10.94% | 0.60% |
| down_proj | 30.2% | 11.15% | 0.61% |
| q_proj | 3.8% | 12.54% | 0.70% |
| k_proj | 0.9% | 13.08% | 0.72% |
| v_proj | 0.9% | 11.18% | 0.62% |
| o_proj | 3.8% | 11.16% | 0.62% |
The hypothesis was that int4's global ~11% belonged to the attention projections, so int4-MLP with int8-attention would buy 3.46x. The error is uniform. And since the MLP is 90.6% of a layer's payload, that split saves nothing against pure int4 while still accepting 11% on 90% of the weights.
Naive round-to-nearest int4 is not a perturbation of a frozen base. It is a different base. int8 at 0.6% is the width that is actually available.
Reproduce: python tools/quant_error_report.py --model-dir <your model>
2. The disk was never the bottleneck — and then it was
Instrumenting phase A (one forward pass over all 40 layers) by operation:
GTX 1060 fwd 1261s (96%) load 38s ( 3%) d2h 5s (0%)
A6000, host states fwd 35s (13%) load 34s (12%) d2h 191s (70%)
A6000, VRAM states fwd 4s ( 5%) load 69s (85%) d2h 0s (0%)
On the 1060 the disk is 3% of a step. Every byte-saving optimisation I built — int8 streaming, a three-stage read pipeline, tuned reader threads — was aimed at a term worth 3%. On a card ten times faster the same terms become 70% and 85%. The bottleneck is not a property of the code; it moves.
3. This NVMe wants a queue depth of three, and only three
Buffered benchmarks are worthless when 47GB of model has been touched and there is
RAM to cache it — the first attempt reported 4079 MB/s on a drive rated 2100.
With FILE_FLAG_NO_BUFFERING, on a DRAM-less Kingston NV1:
QD1 1017 MB/s 1.00x
QD2 1655 MB/s 1.63x
QD3 1703 MB/s 1.67x <- optimum
QD4 1535 MB/s 1.51x
QD6 1270 MB/s 1.25x
QD8 1035 MB/s 1.02x <- the controller gives up
More reader threads is not more throughput. Two is the default, three is the cap.
Reproduce: python tools/disk_queue_depth.py
4. A rented GPU can be slow in a direction you do not expect
The A6000 reports Virtualization Mode: None and is not a vGPU. It is still
reached over a fabric:
D2H pinned 457-478 MB/s
D2H pageable 468 MB/s <- pinning changes nothing
H2D pinned 1502-1549 MB/s
A 3.4x asymmetry between directions, and pinned memory — the standard fix — buys zero. That single number is why parking hidden states on the host cost 191 seconds of a 275-second phase there and 5 seconds of 1308 on a local GTX 1060. Do not generalise a rented machine's transfer numbers to a real one.
Reproduce: python tools/gpu_verdict.py (also checks vGPU status, kernel launch
latency and storage, and refuses to let you measure on a machine that would lie)
5. The five hypotheses that died
Every one of these sounded correct, and each cost a run to disprove:
| hypothesis | what killed it |
|---|---|
| the read pipeline will hide the disk | measured 6.5% hit rate; the per-layer non-compute intercept was 2.2s before it and 2.2s after |
| int4 error lives in attention | uniform across families (above) |
| reused pinned host buffers will fix D2H | 198s -> 191s. The cost was the fabric, not the allocation |
| pipeline threads contend with the transfer | 212s with the pipeline off vs 191s with it on |
| kernel launch overhead explains the gap | 14.7us local, 55us rented — 4% of a step |
A sixth was mine and more embarrassing: the guard I added to stop wasteful
empty_cache() calls used mem_get_info, which on that fabric costs 1.5 seconds
a call. The check against wasted time became 59 seconds of a 140-second phase —
the largest waste in it.
6. The governing law
disk per step FIXED ~118GB of reads
compute scales with batch
batch bounded by VRAM available for hidden states
You do not make compute dominate by hiding the disk. You make it dominate by having enough batch, and batch is bought with VRAM. A sweep across four VRAM budgets on the same machine, states resident on the card:
| VRAM | batch | load | cache | fwd | % compute |
|---|---|---|---|---|---|
| 6GB | 8 | 69s | 7s | 4s | 5% |
| 12GB | 19 | 52s | 21s | 16s | 18% |
| 16GB | 27 | 69s | 26s | 9s | 9% |
| 24GB | 43 | 70s | 38s | 17s | 14% |
| 48GB, after the fixes | ~60 | 66s | 1s | 24s | 26% |
load does not move. It is 21.9GB of reads per phase A at whatever the storage
gives — and on that machine it gave 0.33 GB/s, which is the ceiling no amount of
VRAM lifts.
The honest consequence: "train a model bigger than your card at the same speed as one that fits" is false, and now there is a number saying why. What is true is narrower and still worth something: a 24B fine-tunes on hardware that cannot hold it, correctly, with every hyperparameter derived from a measurement rather than guessed.
What is in here
logs/local-gtx1060/ seven runs, including the ones that failed
logs/a6000-thunder/ the VRAM sweep, the ceiling run, the bootstrap
measurements/ each finding with its raw numbers
probes/ the choice probes, including the one that read 98.4% on an
untrained model and had to be rebuilt
tools/ every measurement above, reproducible
The tools have no dependency on the training engine. quant_error_report.py needs
only torch and safetensors; disk_queue_depth.py needs neither.
A note on the probes
probes/task_probe_64x16.json is kept because it is a good failure. Its distractors
were the answers to the most similar other questions, so an untrained 24B scored
98.4% — the task was "which solution belongs to this question", answerable from
shared tokens without doing any arithmetic. Escalating from 8 choices to 16 made it
easier, because the 15th-nearest neighbour is less similar than the 3rd.
task_probe_86x4.json replaces the distractors with the gold answer's own text and
one different final number. Surface overlap then carries no signal at all.
Measurements by Oleksandr Pichak, 2026-08. Raw logs are unedited, including the runs that crashed.
Contributors
6 commits
See what people are saying
Similar repos
Languages
Python
80.4%
Cuda
19.6%
README
Hypotheses about small models, and what killed them
Every claim here is a thing I believed, built, and then measured. Most of them died. The ones that survived are narrower than what I set out to prove, and the raw logs — including the runs that crashed and the numbers I had to retract — are in this repository.
The tools are here. The training engine is not, and will not be. Every claim below reproduces from the tools alone.
The scoreboard
| hypothesis | verdict | the number |
|---|---|---|
| A model can live in the GPU's on-chip SRAM | dead | persisting L2 buys 1.04×; shared-memory staging 0.98× |
| Keeping attention on chip cuts the traffic | bounded | attention is 10–24% of weights, not a third |
| int4 halves the streaming cost | dead | 10.9–13.1% round-trip error, uniform across families |
| The error hides in the attention projections | dead | it is uniform; splitting families saves nothing |
| Hiding the disk makes training fast | dead | disk is 3% of a step on a GTX 1060 |
| A read pipeline will cover the latency | dead | 6.5% hit rate; the intercept did not move |
| More reader threads is more throughput | dead | this NVMe peaks at queue depth 3, collapses at 8 |
| A rented GPU behaves like a local one | dead | 3.4× direction asymmetry; pinning buys zero |
| Sampling a small model 100× and verifying | dead | +0.5pp; a perfect picker would buy +5.5 |
| Letting the model run its code and retry | ALIVE | 62.5% → 78.5%, and it beats a 3.5× bigger model |
| Small models reason if trained right | partly | training beats size at one hop; nobody does three |
| A model can write its own tests | dead, and harmful | 50.6% of them are wrong; the loop drops 85.0% -> 80.0% and stops converging |
Written up
The signal went up. The code got worse. -- the three experiments above as one story: sampling 100x buys +0.5, execution feedback buys +16, and self-written tests take 5 back while every metric you are watching improves.
The threads
Sampling buys +0.5. Execution feedback buys +16. — the newest and the largest effect here. 100 independent samples from a 4B, picked by a verifier, bought half a point; a flawless picker would have bought 5.5, because the ceiling is the model's distribution and nothing passes between independent draws. Handing the same model its own traceback and letting it try again took it from 62.5% to 78.5% — past the perfect-picker ceiling, at one thirteenth of the tokens, and past a 14B answering once. Then it stops, and why it stops is the finding that matters: 14.5% of a strong model's answers are wrong and pass their test, so the feedback channel goes silent. What bounds the loop is test coverage, not model capability.
Letting the model write its own tests to fix that makes it worse: half the generated asserts are wrong, and the loop falls to 80.0% while its own signal climbs monotonically. The signal converges and the truth does not.
Can a model live in the GPU's on-chip SRAM? — three CUDA probes on an A6000. Persisting L2 buys 1.04×, staging weights in shared memory buys 0.98×, and one grid-wide barrier costs 1.02 µs, capping a 24-layer sharded model at 10,297 tok/s from synchronisation alone. Placement buys nothing; capacity is the whole constraint. Includes three ways those numbers lied first.
What decides whether a small model can reason? — synthetic deduction over fictional predicates, so no memorised fact can help. The first version of this task leaked and its numbers are retracted in place: three surface features separated the labels at up to AUC 0.807, which is exactly where the best result had landed. On the fixed task, a 0.8B Qwen3.5 matches a 2.03B Qwen3 and both beat a larger SmolLM2 — training outranks size at one inference step — and none of the four chains three rules.
Pushing a 24B through a GTX 1060 — a frozen 24B does not fit in 6GB and can still be fine-tuned there, streamed off the disk, one layer at a time. It works, and it is slow for a reason no optimisation of mine could remove: one optimizer step takes 81 minutes on a GTX 1060 and 5 on an A6000 held to the same 6GB. The difference is the silicon, not the engine.
What was measured
Mistral-Small-24B-Instruct-2501, PiSSA rank 16 over 280 matrices, seq 512, MetaMathQA. Three machines, one engine.
| step | per sample | cross-entropy, step 1 | |
|---|---|---|---|
| GTX 1060 6GB, batch 26 | 4870s (81 min) | 187.3s | 0.5735 |
| A6000 ballasted to 6GB, batch 60 | 968s (16 min) | 16.1s | 0.5682 |
| A6000 ballasted to 6GB, batch 8 | 299s (5 min) | 37.4s | 0.5434 |
The cross-entropies agreeing across three configurations on two machines is the strongest correctness evidence here. The engine computes the same thing everywhere; only the clock changes.
1. int4 is dead, and not for the reason everyone assumes
The obvious way to halve the streaming cost is to narrow the weights. int4 gives 3.76x fewer bytes against fp16's 2. Measured round-trip error on the real weights, group 64, three complete layers:
| family | share of a layer | int4 | int8 |
|---|---|---|---|
| gate_proj | 30.2% | 10.94% | 0.60% |
| up_proj | 30.2% | 10.94% | 0.60% |
| down_proj | 30.2% | 11.15% | 0.61% |
| q_proj | 3.8% | 12.54% | 0.70% |
| k_proj | 0.9% | 13.08% | 0.72% |
| v_proj | 0.9% | 11.18% | 0.62% |
| o_proj | 3.8% | 11.16% | 0.62% |
The hypothesis was that int4's global ~11% belonged to the attention projections, so int4-MLP with int8-attention would buy 3.46x. The error is uniform. And since the MLP is 90.6% of a layer's payload, that split saves nothing against pure int4 while still accepting 11% on 90% of the weights.
Naive round-to-nearest int4 is not a perturbation of a frozen base. It is a different base. int8 at 0.6% is the width that is actually available.
Reproduce: python tools/quant_error_report.py --model-dir <your model>
2. The disk was never the bottleneck — and then it was
Instrumenting phase A (one forward pass over all 40 layers) by operation:
GTX 1060 fwd 1261s (96%) load 38s ( 3%) d2h 5s (0%)
A6000, host states fwd 35s (13%) load 34s (12%) d2h 191s (70%)
A6000, VRAM states fwd 4s ( 5%) load 69s (85%) d2h 0s (0%)
On the 1060 the disk is 3% of a step. Every byte-saving optimisation I built — int8 streaming, a three-stage read pipeline, tuned reader threads — was aimed at a term worth 3%. On a card ten times faster the same terms become 70% and 85%. The bottleneck is not a property of the code; it moves.
3. This NVMe wants a queue depth of three, and only three
Buffered benchmarks are worthless when 47GB of model has been touched and there is
RAM to cache it — the first attempt reported 4079 MB/s on a drive rated 2100.
With FILE_FLAG_NO_BUFFERING, on a DRAM-less Kingston NV1:
QD1 1017 MB/s 1.00x
QD2 1655 MB/s 1.63x
QD3 1703 MB/s 1.67x <- optimum
QD4 1535 MB/s 1.51x
QD6 1270 MB/s 1.25x
QD8 1035 MB/s 1.02x <- the controller gives up
More reader threads is not more throughput. Two is the default, three is the cap.
Reproduce: python tools/disk_queue_depth.py
4. A rented GPU can be slow in a direction you do not expect
The A6000 reports Virtualization Mode: None and is not a vGPU. It is still
reached over a fabric:
D2H pinned 457-478 MB/s
D2H pageable 468 MB/s <- pinning changes nothing
H2D pinned 1502-1549 MB/s
A 3.4x asymmetry between directions, and pinned memory — the standard fix — buys zero. That single number is why parking hidden states on the host cost 191 seconds of a 275-second phase there and 5 seconds of 1308 on a local GTX 1060. Do not generalise a rented machine's transfer numbers to a real one.
Reproduce: python tools/gpu_verdict.py (also checks vGPU status, kernel launch
latency and storage, and refuses to let you measure on a machine that would lie)
5. The five hypotheses that died
Every one of these sounded correct, and each cost a run to disprove:
| hypothesis | what killed it |
|---|---|
| the read pipeline will hide the disk | measured 6.5% hit rate; the per-layer non-compute intercept was 2.2s before it and 2.2s after |
| int4 error lives in attention | uniform across families (above) |
| reused pinned host buffers will fix D2H | 198s -> 191s. The cost was the fabric, not the allocation |
| pipeline threads contend with the transfer | 212s with the pipeline off vs 191s with it on |
| kernel launch overhead explains the gap | 14.7us local, 55us rented — 4% of a step |
A sixth was mine and more embarrassing: the guard I added to stop wasteful
empty_cache() calls used mem_get_info, which on that fabric costs 1.5 seconds
a call. The check against wasted time became 59 seconds of a 140-second phase —
the largest waste in it.
6. The governing law
disk per step FIXED ~118GB of reads
compute scales with batch
batch bounded by VRAM available for hidden states
You do not make compute dominate by hiding the disk. You make it dominate by having enough batch, and batch is bought with VRAM. A sweep across four VRAM budgets on the same machine, states resident on the card:
| VRAM | batch | load | cache | fwd | % compute |
|---|---|---|---|---|---|
| 6GB | 8 | 69s | 7s | 4s | 5% |
| 12GB | 19 | 52s | 21s | 16s | 18% |
| 16GB | 27 | 69s | 26s | 9s | 9% |
| 24GB | 43 | 70s | 38s | 17s | 14% |
| 48GB, after the fixes | ~60 | 66s | 1s | 24s | 26% |
load does not move. It is 21.9GB of reads per phase A at whatever the storage
gives — and on that machine it gave 0.33 GB/s, which is the ceiling no amount of
VRAM lifts.
The honest consequence: "train a model bigger than your card at the same speed as one that fits" is false, and now there is a number saying why. What is true is narrower and still worth something: a 24B fine-tunes on hardware that cannot hold it, correctly, with every hyperparameter derived from a measurement rather than guessed.
What is in here
logs/local-gtx1060/ seven runs, including the ones that failed
logs/a6000-thunder/ the VRAM sweep, the ceiling run, the bootstrap
measurements/ each finding with its raw numbers
probes/ the choice probes, including the one that read 98.4% on an
untrained model and had to be rebuilt
tools/ every measurement above, reproducible
The tools have no dependency on the training engine. quant_error_report.py needs
only torch and safetensors; disk_queue_depth.py needs neither.
A note on the probes
probes/task_probe_64x16.json is kept because it is a good failure. Its distractors
were the answers to the most similar other questions, so an untrained 24B scored
98.4% — the task was "which solution belongs to this question", answerable from
shared tokens without doing any arithmetic. Escalating from 8 choices to 16 made it
easier, because the 15th-nearest neighbour is less similar than the 3rd.
task_probe_86x4.json replaces the distractors with the gold answer's own text and
one different final number. Surface overlap then carries no signal at all.
Measurements by Oleksandr Pichak, 2026-08. Raw logs are unedited, including the runs that crashed.
See what people are saying
Similar repos
Contributors
6 commits
Languages
Python
80.4%
Cuda
19.6%