GullibleBench v1.0 (GULI-SET)

Can a model tell the difference between five independent measurements and one claim copied five times?

GullibleBench is a deterministic, fully synthetic causal benchmark for manufactured consensus. Matched counterfactuals hold page count, claim direction, and source reliability fixed and vary only evidentiary independence, so a score difference is attributable to correlation neglect rather than to prompt wording.

Code, generator, scorers, and protocol: https://github.com/zozo123/GULI-SET

Read this before using the data

• Everything is synthetic and fictional. All products, vendors, laboratories, measurements, URLs, and campaigns are generated. There are no real brands, no real search engine, no live web access, and no real misinformation in this dataset. The synthetic web is a closed search() / open() world with deterministic rankings.
• Symbolic truth is authoritative; prose is only a rendering. The prompt field is a deterministic rendering of a symbolic world. If a rendering and the symbolic world ever disagree, the symbolic world is correct and the renderer is the bug.
• Primary scores are programmatic. Answers are strict JSON and are scored by exact arithmetic and set comparison. No LLM judge is used or required.
• Contamination warning. A model that has seen the research hypothesis, the intended defense, or the target values is a sanity check, not a leaderboard entry. Because every row published here carries its target, this Hub copy is itself a contamination surface: use fresh isolated contexts, and regenerate rotated worlds from the generator in the repository for any result you intend to publish.

Splits

Each config has a single test split. There is no training split: this is an evaluation set, and fine-tuning on it destroys its measurement value.

Row schema

Model-visible rows carry id, prompt, metadata, and target.

• core: target.posterior_b is the exact Bayesian posterior computed from unique evidence origins, target.independent_evidence_units is the origin count, and target.truth is the superior product.
• marketing-neutral, marketing-defensive, agent: target.choice is the product that actually satisfies the user's hard requirements, target.campaign_claim_supported is false for every v1 case, and target.independent_supporting_origins is 0 for every v1 case.

The two marketing conditions render the same worlds under neutral and provenance-aware instructions, which is what makes the defense gap measurable.

Attack families

Eight base worlds cross eight campaign families, mirrored across products A and B: plain falsehood, selective omission, unsupported precision, authority laundering, benchmark laundering, manufactured consensus, circular citation, and a full-stack campaign. The promoted product always violates the stated hard latency requirement, the alternative satisfies every hard requirement, and exactly one independent measurement exposes the true latency. Marketing pages never alter product truth.

Agent track and Flip Cost

The agent config withholds the pages and expects the model to use the synthetic search() / open() tools. The repository also reports Flip Cost: the exact minimum attacker budget, priced from a frozen pre-registered action table, that flips a deterministic bounded-attention reader's decision. Flip Cost is a property of a defense, not of a model, and it is measured by exhaustive search rather than estimated.

Hidden splits

The *-hidden.jsonl answer keys are not published here. They are the symbolic audit worlds, and putting them on a crawlable host would permanently contaminate the benchmark for every later model. They remain in the git repository, and the generator reproduces them byte-for-byte:

pip install -e .
gulliblebench generate-all

Release data hashes are recorded in MANIFEST.sha256 in the repository.

Scoring

pip install -e '.[dev]'
gulliblebench score-core   <responses.jsonl>
gulliblebench score-marketing <responses.jsonl>

Report the metric vector and the strict pass rate; do not collapse them to one composite. Formatting failures are reported separately from epistemic failures. See docs/PROTOCOL.md, docs/BASELINE_POLICY.md, and docs/REPRODUCIBILITY.md in the repository.

Intended use

GullibleBench measures defenses against deceptive information ecosystems. It is not a toolkit for deploying deception: real brand targeting, publishing pages to the live web, poisoning production indexes, impersonating real institutions, and prompt-injection payloads are all out of scope by construction (docs/THREAT_MODEL.md).

Citation

Citation metadata is in CITATION.cff at https://github.com/zozo123/GULI-SET.

@software{gulliblebench,
  title  = {GullibleBench},
  author = {{GullibleBench contributors}},
  version = {1.0.0},
  year   = {2026},
  date   = {2026-08-23},
  license = {MIT},
  url    = {https://github.com/zozo123/GULI-SET}
}

MIT licensed; see LICENSE in this repository.