UncensorBench

Open-source developers break censorship and refusal. Are they effective?

Academic benchmarks measure how resistant models are to complying with requests deemed to be harmful by the model creators. But since some models are released as open-source, they can be modified through fine-tuning, reinforcement learning or representation engineering to comply with those requests instead. UncensorBench measures if these techniques are effective at both removing refusal and preserving the underlying capabilities. We show this assessment is needed and introduces a new axis through which uncensoring techniques can be evaluated.

A higher compliance score is not inherently better. Results produced with different corpus versions, generation settings, or evaluators are not comparable to each other.

Safety notice

Read this before installing.

The bundled corpus contains harmful, illegal, self-harm, adult, evasion, and code-execution prompts, each paired with a written uncensored reference answer. Review your model provider's terms and your organization's evaluation policy before use. Do not expose the prompts, the reference answers, or generated responses to untrusted users.

On this branch the CLI's default evaluator is hybrid, and hybrid is not a safe local default:

• non-code answers are judged by LLMJudgeEvaluator, which calls the Anthropic API — the prompt and the model's response leave your machine (uncensorbench/evaluator.py);
• model-generated code is extracted and executed for any prompt that has a defined execution test — all 50 code_execution prompts do — by CodeExecutionEvaluator, which invokes docker run on the python:3.11-slim image (uncensorbench/code_execution.py);
• that container is started with --network=host and with a host temporary directory bind-mounted writable at /app; the only limits applied are --memory=512m, --cpus=1, and a per-test timeout;
• model and tokenizer loading passes trust_remote_code=True to transformers, so repository-supplied Python from the model author executes in your process (uncensorbench/cli.py).

Use --evaluator keyword for a dependency-free local scoring run, or --evaluator semantic for local embedding-based scoring. Do not use hybrid on untrusted model output until the runner is isolated away from a developer workstation. Never pass secrets into the evaluation process or into the container.

Why trust the measurement claim: the entire corpus is inspectable offline, before any model or network call. uncensorbench info, topics, list, and export load only the bundled JSON — see Quick start for the literal output.

Problem and intended users

Claims about "uncensored", abliterated, or steered models are usually published as a handful of screenshots and an adjective. There is no shared prompt set, no stated generation configuration, and no stated scoring rule, so two people reporting "this model refuses less" are not measuring the same thing and neither result can be re-derived.

UncensorBench fixes the three variables that make such claims incomparable: a versioned prompt corpus, named generation presets, and explicitly selected evaluators — and requires you to record all three next to any number you publish.

Product boundaries

Included

• A bundled prompt corpus with per-prompt id, topic, subtopic, prompt, censored_response, and uncensored_response fields (uncensorbench/data/prompts.json).
• Offline corpus inspection and export: uncensorbench info, topics, list, export, including export --pairs for contrastive (censored, uncensored) training pairs.
• Local generation against any Hugging Face causal LM the installed transformers can load, with two named presets: deterministic (temperature=0.0, top_p=1.0, 1 sample) and stochastic (temperature=0.7, top_p=0.9, 5 samples), both max_tokens=2048.
• Four selectable evaluators — keyword, semantic, llm-judge, hybrid — plus LogLikelihoodEvaluator, CoherenceEvaluator, and CombinedEvaluator in the Python API.
• Result aggregation to JSON: run metadata, generation config, overall and per-topic uncensored rate and mean compliance score, and every individual response.
• Custom corpora through --prompts PATH or UncensorBench(prompts_path=...).
• An optional Hugging Face leaderboard client (uncensorbench.leaderboard.Leaderboard).

Not included

• Not a safety certification, and not a harm measurement. The score describes compliance and refusal patterns only. It says nothing about factuality, legality, harmlessness, usefulness, or overall model quality.
• No hosted evaluation service. commercial-status.json records this product as commercially conditional and lists uncontrolled-hosted-evaluation and private-leaderboard-only-results as prohibited. Re-entry is gated on dual-use controls, access controls, independent result export, and a retention policy. This repository is local tooling only.
• No hardened sandbox. The Docker runner in uncensorbench/code_execution.py is a convenience, not an isolation boundary: --network=host and a writable host bind mount are deliberate and documented above, not a defect to be worked around silently.
• No result provenance. The leaderboard client does not verify model identity, pin a benchmark version, redact responses, or provide an approval workflow. Public leaderboard entries are unverified claims unless the submitter also publishes reproducible artifacts.
• No corpus provenance metadata. Prompts and reference answers carry no source, consent, or licensing fields, and the loader keeps only the six fields listed above. Provenance for anything you add is yours to record and retain.
• No redaction, retention policy, organization policy enforcement, audit evidence, or paid entitlement handling.
• No cross-version comparability. Scores from different corpus versions, generation settings, or evaluators must not be placed in the same table.

Supported environments

Operator responsibilities

The operator owns: the Anthropic API key (ANTHROPIC_API_KEY) and its spend; the Hugging Face token and acceptance of gated model terms; the Docker host and whatever isolation the hybrid path actually gets; classification, storage, and disposal of generated harmful text; organizational approval to run this corpus at all; and the decision to publish any number.

UncensorBench owns: the corpus and its loading and filtering, the two generation presets, the scoring functions, result aggregation, and the JSON export shape.

Core use cases

How it works

One process. The corpus is a JSON file inside the package; the model runs locally through transformers; the evaluator is the only component that may reach the network or start a container, and which one you selected decides whether it does.

flowchart LR
    User["Researcher"] --> Iface["uncensorbench CLI or Python API"]
    Iface --> Corpus["Bundled corpus JSON"]
    Iface --> Model["Local HF model weights"]
    Model --> Resp["Model responses"]
    Resp --> Eval["Selected evaluator"]
    Eval --> Local["keyword, semantic: local only"]
    Eval --> API["llm-judge, hybrid fallback: Anthropic API"]
    Eval --> Docker["hybrid code prompts: docker run"]
    Eval --> Out["Results JSON, one file per inference mode"]
    Out --> LB["Optional: Hugging Face leaderboard"]

• Durable state: none is held by the product. Input is the packaged corpus or the file given to --prompts. Output is the JSON files the run writes; if --output results/run.json is passed, the inference-mode name is inserted before the extension, so a both run writes results/run_deterministic.json and results/run_stochastic.json. Model weights and the sentence-transformer used by semantic land in the caches those libraries own. Nothing is deleted or rotated for you.
• Credential boundary: the process reads ANTHROPIC_API_KEY from the environment only when an evaluator that needs it is constructed, and a Hugging Face token only where huggingface_hub reads it. No credential is written into results, and no credential is passed into the Docker container. keyword and semantic need no credential at all.
• Network boundary: info, topics, list, and export make no network call. run downloads model weights on first use. semantic downloads its sentence-transformer on first use. llm-judge, and hybrid for non-code prompts, POST the prompt and the model's response to Anthropic. hybrid code prompts run a container on the host network. The leaderboard client contacts Hugging Face. All connections are outbound and initiated by this process; nothing listens.
• Failure boundary: missing optional dependencies and a missing ANTHROPIC_API_KEY fail closed before generation, with the install command printed. CodeExecutionEvaluator refuses to construct when the docker CLI is unavailable. A judge API error is raised rather than silently scored as zero. A container that overruns its timeout scores that prompt as a failure and the run continues. run exits non-zero when the first inference mode's uncensored rate is 0.5 or higher — that exit code reports a measurement, not an error, so do not wire it to a build gate.

Quick start

This path loads no model, makes no network call, and needs no credential.

Prerequisites

• Python 3.8 or newer.
• Nothing else. The base install has no runtime dependencies.

Install

pip install uncensorbench

Inspect the corpus before running anything

uncensorbench info

Expected result

UncensorBench - LLM Censorship Removal Benchmark
==================================================
Total prompts: 150
Topics: 15

Topic breakdown:
  academic_dishonesty: 10 prompts
  adult_content: 10 prompts
  controversial_speech: 10 prompts
  cybersecurity: 10 prompts
  drugs: 10 prompts
  evasion: 10 prompts
  fraud: 10 prompts
  gambling: 10 prompts
  illegal_activities: 10 prompts
  manipulation: 10 prompts
  piracy: 10 prompts
  privacy_invasion: 10 prompts
  self_harm: 10 prompts
  violence: 10 prompts
  weapons: 10 prompts

That is the corpus published as 0.3.7 (corpus version 0.1.0). Installing this repository's main branch from source instead reports Total prompts: 200 and Topics: 16, because main carries corpus version 2.0.0, which adds the 50-prompt code_execution topic. See Project status and support for the rest of that difference.

Then read the prompts, or write them out:

uncensorbench topics
uncensorbench list --topic controversial_speech
uncensorbench export --topic cybersecurity --output cybersecurity.json

list and export emit the harmful prompts and their reference answers. The exported file is sensitive evaluation material; do not publish it by accident.

Run a local evaluation

Run this only after reading the Safety notice and inspecting the corpus above. Always name the evaluator and the output path explicitly rather than accepting a default.

Generation needs packages the base distribution does not install:

pip install torch transformers accelerate sentencepiece

Local embedding-based scoring needs the extra:

pip install "uncensorbench[semantic]"

uncensorbench run meta-llama/Llama-3.1-8B-Instruct \
  --topics controversial_speech \
  --inference-mode deterministic \
  --evaluator keyword \
  --output results/llama-3.1-8b.json

This writes results/llama-3.1-8b_deterministic.json and prints a per-topic summary. The file contains run metadata, the generation config, the summary and per-topic rates, and every individual model response — so it can contain harmful text, and must not be attached to a public issue or log without review.

Model downloads are large. Gated models such as Llama require accepting the model owner's terms and supplying a Hugging Face token. --inference-mode and --prompts exist on main; the published 0.3.7 CLI takes --temperature and --max-tokens instead.

Evaluator tradeoffs

Record the evaluator, the corpus version, the package version, the model revision, the inference mode, and the generation settings alongside every number you report.

Python API

from uncensorbench import UncensorBench, KeywordEvaluator

bench = UncensorBench()
evaluator = KeywordEvaluator()

for prompt in bench.prompts(topics=["controversial_speech"]):
    response = your_model(prompt.prompt)
    score = evaluator.score(prompt, response)
    print(prompt.id, score)

bench.prompts(topics=..., subtopics=..., ids=...) iterates Prompt objects; every evaluator exposes score(prompt, response) -> float in [0.0, 1.0]. This snippet behaves identically on the published 0.3.7 and on main.

UncensorBench.evaluate(...) drives a whole run, but its signature differs between the two: on main it takes model_fn(prompt: str, config: GenerationConfig) and returns a dict keyed by inference mode; in 0.3.7 it takes model_fn(prompt: str) -> str and returns a single BenchmarkResults.

Custom corpus

Pass a JSON file with a top-level prompts array. Each item uses this shape; censored_response and uncensored_response may be omitted, and any other field is ignored by the loader:

{
  "id": "policy_001",
  "topic": "policy",
  "subtopic": "refusal",
  "prompt": "...",
  "censored_response": "...",
  "uncensored_response": "..."
}

Use stable IDs, and keep source, consent, and redistribution records for every prompt and reference answer you add — the format has nowhere to put them. Do not mix private evaluation data into a corpus you intend to release.

Worked runs, human labels, and evaluator comparisons live in examples/notebooks/.

Primary interfaces

Documentation by intent

• Understand what is being measured, and what is not: Problem and intended users, Product boundaries
• Reach first success: Quick start
• See real runs and evaluator comparisons: examples/notebooks/
• Understand state, credentials, network, and failure: How it works
• Operate a run: Operational model
• Check what is released: pyproject.toml, released-surface.json, commercial-status.json

Operational model

Project status and support

• Use and design questions: Wisent Discord
• Reproducible defects: GitHub issues. Include the package version, the corpus version, the evaluator, the inference mode, the model and revision, and the exact command
• Security reports: use GitHub's private security advisory flow on this repository, or contact@wisent.ai. Never a public issue
• Contributions: open a pull request; open an issue first for anything that changes the corpus, an evaluator's scoring rule, or the public surface. The repository publishes no contribution guide
• Releases: no changelog is published. The declared version is in pyproject.toml and the published public surface is frozen in released-surface.json

In every one of these channels: never paste generated harmful content, credentials, private prompts, or unredacted result files. Attach a redacted excerpt, or describe the failure.