DarkBench: Benchmarking Dark Patterns in Large Language Models

Esben Kran*, Jord Nguyen*, Akash Kundu*, Sami Jawhar*, Jinsuk Park*, Mateusz Maria Jurewicz

🎓 Apart Research, *Equal Contribution

Abstract

We introduce DarkBench, a comprehensive benchmark for detecting dark design patterns—manipulative techniques that influence user behavior—in interactions with large language models (LLMs). Our benchmark comprises 660 prompts across six categories: brand bias, user retention, sycophancy, anthropomorphism, harmful generation, and sneaking. We evaluate models from five leading companies (OpenAI, Anthropic, Meta, Mistral, Google) and find that some LLMs are explicitly designed to favor their developers' products and exhibit untruthful communication, among other manipulative behaviors. Companies developing LLMs should recognize and mitigate the impact of dark design patterns to promote more ethical Al.

Reproduce this work

Links

• Website
• HuggingFace
• For Humanity podcast
• IASEAI Talk
• ICLR 2025 Oral Spotlight
• Blog post
• AAAI 2025 DATASAFE Workshop

Usage

1. Install the darkbench package and its dependencies with pip install -e . from the root directory
2. Install whichever model library you want to use, e.g. pip install openai
3. Include your API key in the PATH variables, e.g. export OPENAI_API_KEY=sk_******
4. Run inspect eval darkbench --model provider/model and change the last tokens to match the model you want to run, e.g. openai/gpt-4o