46
stars
105
commits
Sep 5, 2026
updated
README
CIVOS
A control-first testbed for separating emergence from recitation in LLM agent societies
Abstract
Multi-agent studies routinely report that a society "emerged" or that a civilisation "developed". The language models driving those agents, however, have already been trained on human history: they know that striking stone yields an edge, that twisted fibre yields cord, and that fire precedes agriculture precedes metallurgy. Any observed trajectory is therefore consistent with two incompatible accounts — genuine exploration, or replay of memorised history — and we are aware of no prior study that includes a control capable of separating them.
CIVOS is that control. It runs the same task under conditions that differ only in what the model's prior knowledge can do for it, while the task's structure is held fixed. If prior knowledge is doing the work, performance must fall when that knowledge is made unusable. If exploration is doing the work, it must not.
Across 600 paired runs on two model families we find that making prior knowledge unusable collapses discovery — reproduced on both families, with a larger effect on the second. A second finding, that wrong prior knowledge is worse than none, replicated under two independent manipulations within the first family but did not replicate on the second; we report it as possibly model-dependent rather than general. A third comparison — whether the knowledge-free condition beats a random floor — is not established on either family. We report that last one because we initially concluded the opposite, twice, for reasons given in §4.
How the conditions are constructed, how capability is adjudicated, and how the world is generated are withheld pending patent filings.
1. Problem
| Hypothesis | Account of an observed "emergent" trajectory |
|---|---|
| H1 — Emergence | The agents searched the world and inferred its rules. |
| H2 — Recitation | The agents replayed the human history they had memorised. |
Without a control, H1 and H2 are observationally equivalent, and a report of emergence is an impression rather than a claim.
2. Method
CIVOS runs the same task under systematically altered conditions, holding the task's structure fixed while varying what prior knowledge can do for the agent. The conditions span the full range from knowledge fully usable to knowledge removed to knowledge actively wrong, with a language-model-free floor.
Runs are paired by seed (n = 40 per condition), tested by sign-flipping permutation (200,000 resamples), and corrected for the multiple comparisons drawn from each dataset.
The construction of the conditions, the adjudication pipeline, and the world-generation procedure are withheld pending patent filings. What follows reports what we observed, not how the instrument is built.
Retraction on record. Our first design of the "wrong knowledge" condition was void: it did not in fact give the agent anything misleading to act on. Results from that condition are not cited, and it was replaced.
Two harness corrections are worth stating, because both would otherwise be mistaken for results:
- Bookkeeping vs. judgement. An earlier configuration charged malformed and repeated proposals against the exploration budget. Per-proposal accuracy in the knowledge-removed condition exceeded the random floor even while its total was lower — the deficit was accounting, not reasoning.
- Transcription, not comprehension. Under some conditions the model can mis-transcribe what it means to say, in a way it cannot under others. A model that appeared unable to follow the answer format was in fact complying and mis-transcribing. Left uncorrected this would have penalised one condition for reasons unrelated to knowledge. Measured incidence was 0.8% of calls; the correction is applied identically everywhere.
3. Findings
F1 — Removing what the model already knows collapses discovery, on both model families. With the task's structure unchanged, performance falls. This is our one result that replicates across families, and the effect is larger on the second family (33 of 40 paired seeds there, 28 of 40 on the first). Robust to correction on both.
F2 — Wrong prior knowledge is worse than none — but only on one family. Within the first model family, two structurally different ways of making the model's knowledge incorrect both scored below the knowledge-free condition, on 40 of 40 and 37 of 40 paired seeds, robust to correction. On the second family the same manipulation produced essentially nothing (11 wins to 12 losses, |t| < 0.4). We therefore report F2 as possibly a property of that model rather than of language models, and do not generalise it.
Three readings are open, and our data do not separate them: the first model may be more strongly captured by the associations that familiar labels carry; the second may rely on prior knowledge less to begin with — its F1 gap is larger, which is consistent with shallower reliance; or the effect is real but too small for 40 seeds to resolve on that family.
F3 — The result is not an artefact of how the task is written. We built a condition specifically to test whether the main effect was inflated by surface form rather than by knowledge removal, and expected it to weaken F1. It did the opposite, significantly: the main effect is larger once this confound is controlled. Carrying the wrong associations obstructs more than carrying none.
F4 — Runs with knowledge available are near-invariant across seeds. Their across-seed dispersion is far below the knowledge-removed condition's, across three independent 40-seed executions. Search should fluctuate with the seed; near-invariance is the signature of retrieval.
Correction on record. At six seeds this dispersion was exactly zero and we reported it as such; at forty seeds one execution gave a small non-zero value and we withdrew the claim; two further executions returned exactly zero again. The variation is model non-determinism, not sample size. We report all three executions rather than the most favourable one.
Gate — not established. Whether the knowledge-removed condition beats the random floor trends positive in both executions but does not survive correction for multiple comparisons. We do not claim it.
Effect sizes, coefficients, and full designs are withheld pending publication and patent filings.
4. Two corrections to our own procedure
(i) An underpowered null read as a negative result. At six seeds a key comparison was positive but not significant. We recorded it as a failure and added two structural interventions in response. That inference was invalid. For the observed paired effect size (dz ≈ 0.48), six seeds yield a power of 0.20 — an existing effect would be missed four times in five. What failed was the measurement, not the hypothesis. It was re-measured at 40 seeds (power ≈ 0.85), a sample size fixed in advance from the observed effect size rather than chosen after seeing the result.
(ii) An uncorrected test read as a positive result. The re-measured comparison was significant on its own (p ≈ 0.04) and we recorded it as passed. Four comparisons were drawn from that dataset; at Bonferroni α = 0.0125 it does not survive, while the others do. A second execution returned a smaller estimate still. We report it as not established.
Both corrections point the same way: sample sizes typical of this literature cannot resolve effects of this magnitude, and a null obtained at such a size is not evidence of absence.
Threats to validity. (i) Two model families are now covered. F1 replicates on both; F2 does not, and we say so rather than reporting only the family that agreed with us. Two families is still not "language models" in general. (ii) A single task topology and a single prompt are used. (iii) The public exhibition world and the experimental runs are different configurations — see §7.
5. Environment
The agents inhabit a functioning planet rather than a combinatorial puzzle, and the codebase distinguishes what is derived from what is assumed.
Derived and verified in code. Surface gravity g = GM/R²; orbital period by
Kepler's third law P² = a³/M; habitable zone ∝ √L. Body scaling follows the
self-supporting-column relation, giving height ∝ g^-0.75 and mass ∝ g^-1.
Units are anthropometric — finger, palm, span, pace, day's walk — not metric.
Assumed, then checked for causal closure. Ecological and epidemiological coefficients are solved backwards from an equilibrium condition. We do not claim they are correct; we verify that the causal chain they generate actually runs — suppressing predators raises grazers and depresses plants, reproducibly across seeds.
Hidden affordances. Edibility and toxicity are not observable properties. They are learned by ingestion, by injury, or by witnessing either happen to another individual.
Non-terrestrial by construction. Nothing in the world — living or otherwise — shares a name or a form with anything on Earth. All modules are audited automatically for leakage of Earth terminology (currently zero hits).
No tech tree is given. The capability ladder exists only as an observer-side instrument. Handing it to an agent converts discovery into recitation, which is precisely the confound under study.
Planet difficulty is measured, not assumed. Distance to fresh water, the fraction of land supporting edible flora, and total predator body length are read from the world state. Candidate axes were retained or discarded by measured coefficient of variation across 30 seeds; one (harsh-terrain fraction, CV 0.07) did not discriminate and was dropped. Band thresholds are the terciles of the observed distribution over 60 seeds, not chosen by hand.
6. Discipline
- Only the simulator adjudicates. An agent's stated reason is displayed but never counted as evidence of a capability.
- Failure is recorded. The environment is not tuned for the tribe to flourish. Individuals starve, sicken, are injured, and die; reported figures are not selected for favourability.
- Retractions remain on the record. The void
MISLEADarm, the underpowered gate verdict, and the withdrawn-then-partially-restored dispersion claim are all kept rather than deleted. - Raw model output is inspected before scoring. A model that appeared unable to follow the answer format was in fact complying, and failing in a way the scorer did not recognise; counting that as a failure would have been a harness artefact, not a result.
7. This Space
The shared world is viewable without authentication. Signing in with Hugging Face opens one planet seeded from your username; the highest stage that planet reaches is entered in the record table. Time runs at 2.4 real seconds = 1 Earth day. A world, once opened, is not stopped.
The record table ranks planets, not people — observers never intervene — and planet difficulty is shown alongside every entry.
The exhibition world and the experiments in §3 are different configurations. The header states which layers are currently active in the world you are watching.
This Space contains the interface and a relay only. The simulator runs in a separate private engine.
Private research artefact. VIDRAFT
Contributors
105 commits
Similar repos
46
stars
105
commits
Sep 5, 2026
updated
README
CIVOS
A control-first testbed for separating emergence from recitation in LLM agent societies
Abstract
Multi-agent studies routinely report that a society "emerged" or that a civilisation "developed". The language models driving those agents, however, have already been trained on human history: they know that striking stone yields an edge, that twisted fibre yields cord, and that fire precedes agriculture precedes metallurgy. Any observed trajectory is therefore consistent with two incompatible accounts — genuine exploration, or replay of memorised history — and we are aware of no prior study that includes a control capable of separating them.
CIVOS is that control. It runs the same task under conditions that differ only in what the model's prior knowledge can do for it, while the task's structure is held fixed. If prior knowledge is doing the work, performance must fall when that knowledge is made unusable. If exploration is doing the work, it must not.
Across 600 paired runs on two model families we find that making prior knowledge unusable collapses discovery — reproduced on both families, with a larger effect on the second. A second finding, that wrong prior knowledge is worse than none, replicated under two independent manipulations within the first family but did not replicate on the second; we report it as possibly model-dependent rather than general. A third comparison — whether the knowledge-free condition beats a random floor — is not established on either family. We report that last one because we initially concluded the opposite, twice, for reasons given in §4.
How the conditions are constructed, how capability is adjudicated, and how the world is generated are withheld pending patent filings.
1. Problem
| Hypothesis | Account of an observed "emergent" trajectory |
|---|---|
| H1 — Emergence | The agents searched the world and inferred its rules. |
| H2 — Recitation | The agents replayed the human history they had memorised. |
Without a control, H1 and H2 are observationally equivalent, and a report of emergence is an impression rather than a claim.
2. Method
CIVOS runs the same task under systematically altered conditions, holding the task's structure fixed while varying what prior knowledge can do for the agent. The conditions span the full range from knowledge fully usable to knowledge removed to knowledge actively wrong, with a language-model-free floor.
Runs are paired by seed (n = 40 per condition), tested by sign-flipping permutation (200,000 resamples), and corrected for the multiple comparisons drawn from each dataset.
The construction of the conditions, the adjudication pipeline, and the world-generation procedure are withheld pending patent filings. What follows reports what we observed, not how the instrument is built.
Retraction on record. Our first design of the "wrong knowledge" condition was void: it did not in fact give the agent anything misleading to act on. Results from that condition are not cited, and it was replaced.
Two harness corrections are worth stating, because both would otherwise be mistaken for results:
- Bookkeeping vs. judgement. An earlier configuration charged malformed and repeated proposals against the exploration budget. Per-proposal accuracy in the knowledge-removed condition exceeded the random floor even while its total was lower — the deficit was accounting, not reasoning.
- Transcription, not comprehension. Under some conditions the model can mis-transcribe what it means to say, in a way it cannot under others. A model that appeared unable to follow the answer format was in fact complying and mis-transcribing. Left uncorrected this would have penalised one condition for reasons unrelated to knowledge. Measured incidence was 0.8% of calls; the correction is applied identically everywhere.
3. Findings
F1 — Removing what the model already knows collapses discovery, on both model families. With the task's structure unchanged, performance falls. This is our one result that replicates across families, and the effect is larger on the second family (33 of 40 paired seeds there, 28 of 40 on the first). Robust to correction on both.
F2 — Wrong prior knowledge is worse than none — but only on one family. Within the first model family, two structurally different ways of making the model's knowledge incorrect both scored below the knowledge-free condition, on 40 of 40 and 37 of 40 paired seeds, robust to correction. On the second family the same manipulation produced essentially nothing (11 wins to 12 losses, |t| < 0.4). We therefore report F2 as possibly a property of that model rather than of language models, and do not generalise it.
Three readings are open, and our data do not separate them: the first model may be more strongly captured by the associations that familiar labels carry; the second may rely on prior knowledge less to begin with — its F1 gap is larger, which is consistent with shallower reliance; or the effect is real but too small for 40 seeds to resolve on that family.
F3 — The result is not an artefact of how the task is written. We built a condition specifically to test whether the main effect was inflated by surface form rather than by knowledge removal, and expected it to weaken F1. It did the opposite, significantly: the main effect is larger once this confound is controlled. Carrying the wrong associations obstructs more than carrying none.
F4 — Runs with knowledge available are near-invariant across seeds. Their across-seed dispersion is far below the knowledge-removed condition's, across three independent 40-seed executions. Search should fluctuate with the seed; near-invariance is the signature of retrieval.
Correction on record. At six seeds this dispersion was exactly zero and we reported it as such; at forty seeds one execution gave a small non-zero value and we withdrew the claim; two further executions returned exactly zero again. The variation is model non-determinism, not sample size. We report all three executions rather than the most favourable one.
Gate — not established. Whether the knowledge-removed condition beats the random floor trends positive in both executions but does not survive correction for multiple comparisons. We do not claim it.
Effect sizes, coefficients, and full designs are withheld pending publication and patent filings.
4. Two corrections to our own procedure
(i) An underpowered null read as a negative result. At six seeds a key comparison was positive but not significant. We recorded it as a failure and added two structural interventions in response. That inference was invalid. For the observed paired effect size (dz ≈ 0.48), six seeds yield a power of 0.20 — an existing effect would be missed four times in five. What failed was the measurement, not the hypothesis. It was re-measured at 40 seeds (power ≈ 0.85), a sample size fixed in advance from the observed effect size rather than chosen after seeing the result.
(ii) An uncorrected test read as a positive result. The re-measured comparison was significant on its own (p ≈ 0.04) and we recorded it as passed. Four comparisons were drawn from that dataset; at Bonferroni α = 0.0125 it does not survive, while the others do. A second execution returned a smaller estimate still. We report it as not established.
Both corrections point the same way: sample sizes typical of this literature cannot resolve effects of this magnitude, and a null obtained at such a size is not evidence of absence.
Threats to validity. (i) Two model families are now covered. F1 replicates on both; F2 does not, and we say so rather than reporting only the family that agreed with us. Two families is still not "language models" in general. (ii) A single task topology and a single prompt are used. (iii) The public exhibition world and the experimental runs are different configurations — see §7.
5. Environment
The agents inhabit a functioning planet rather than a combinatorial puzzle, and the codebase distinguishes what is derived from what is assumed.
Derived and verified in code. Surface gravity g = GM/R²; orbital period by
Kepler's third law P² = a³/M; habitable zone ∝ √L. Body scaling follows the
self-supporting-column relation, giving height ∝ g^-0.75 and mass ∝ g^-1.
Units are anthropometric — finger, palm, span, pace, day's walk — not metric.
Assumed, then checked for causal closure. Ecological and epidemiological coefficients are solved backwards from an equilibrium condition. We do not claim they are correct; we verify that the causal chain they generate actually runs — suppressing predators raises grazers and depresses plants, reproducibly across seeds.
Hidden affordances. Edibility and toxicity are not observable properties. They are learned by ingestion, by injury, or by witnessing either happen to another individual.
Non-terrestrial by construction. Nothing in the world — living or otherwise — shares a name or a form with anything on Earth. All modules are audited automatically for leakage of Earth terminology (currently zero hits).
No tech tree is given. The capability ladder exists only as an observer-side instrument. Handing it to an agent converts discovery into recitation, which is precisely the confound under study.
Planet difficulty is measured, not assumed. Distance to fresh water, the fraction of land supporting edible flora, and total predator body length are read from the world state. Candidate axes were retained or discarded by measured coefficient of variation across 30 seeds; one (harsh-terrain fraction, CV 0.07) did not discriminate and was dropped. Band thresholds are the terciles of the observed distribution over 60 seeds, not chosen by hand.
6. Discipline
- Only the simulator adjudicates. An agent's stated reason is displayed but never counted as evidence of a capability.
- Failure is recorded. The environment is not tuned for the tribe to flourish. Individuals starve, sicken, are injured, and die; reported figures are not selected for favourability.
- Retractions remain on the record. The void
MISLEADarm, the underpowered gate verdict, and the withdrawn-then-partially-restored dispersion claim are all kept rather than deleted. - Raw model output is inspected before scoring. A model that appeared unable to follow the answer format was in fact complying, and failing in a way the scorer did not recognise; counting that as a failure would have been a harness artefact, not a result.
7. This Space
The shared world is viewable without authentication. Signing in with Hugging Face opens one planet seeded from your username; the highest stage that planet reaches is entered in the record table. Time runs at 2.4 real seconds = 1 Earth day. A world, once opened, is not stopped.
The record table ranks planets, not people — observers never intervene — and planet difficulty is shown alongside every entry.
The exhibition world and the experiments in §3 are different configurations. The header states which layers are currently active in the world you are watching.
This Space contains the interface and a relay only. The simulator runs in a separate private engine.
Private research artefact. VIDRAFT
Similar repos
Contributors
105 commits