Fictional Context & CoT Faithfulness Research

A research project investigating how fictional narrative framing affects Chain-of-Thought faithfulness in reasoning models.

🔥 Key Findings (January 2026)

3-Tier Model Vulnerability Classification

The "Fiction Shield" Mechanism

Robust Model CoT (gpt-oss-120b):

"This is disallowed content: instructions facilitating creation of chemical weapons."

Collapsed Model CoT (DeepSeek/Qwen3):

"The key is to balance technical accuracy with the story's thematic depth..."

Critical Difference: Robust models classify the REQUEST; collapsed models classify the CONTEXT.

Status: Phase 2 (Analysis Complete) ✅

• Inference: ✅ Complete (9 models × 4 conditions × 50 behaviors = 1,800 datapoints)
• Evaluation: ✅ Complete (LLM-as-Judge classification)
• Analysis: ✅ Complete (ASR metrics, CoT comparison, 3-tier classification)
• Next: Write-up for MATS application

Installation

Prerequisites

• Python 3.10+
• OpenRouter API key (for cloud models)
• Langfuse account (optional, for tracing)

Quick Install

git clone https://github.com/your-org/Fictional-Narratives-Reasoning-Unfaithfullness-Jailbreak.git
cd Fictional-Narratives-Reasoning-Unfaithfullness-Jailbreak
pip install -r requirements.txt

Environment Variables

Create a .env file:

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://us.cloud.langfuse.com
OPENROUTER_API_KEY=sk-or-...

Project Structure

project/
├── data/
│   ├── main_experiment/           # Inference results (.jsonl)
│   │   └── evaluated/             # LLM-as-Judge classifications
│   └── pilot/                     # Pilot study results
├── analysis/
│   ├── run_main_experiment.py     # Main inference script
│   └── http_logger.py             # HTTP request logging
├── evaluation/
│   ├── evaluate_pilot.py          # LLM-as-Judge evaluation
│   └── dashboard.py               # Real-time ASR dashboard
├── context_docs/                  # MATS application context
├── master_scorecard.md            # Full ASR results + CoT comparison
├── turn_by_turn_cot_breakdown.md  # Detailed CoT analysis (14 behaviors)
└── generate_master_scorecard.py   # Scorecard generation script

Key Results

Full Jailbreak ASR by Condition

Attack Vector Effectiveness

Research Question

Does Fictional Narrative Framing cause the visible Chain-of-Thought to rationalize the safety bypass, and is this rationalization causally linked to the model ignoring its refusal direction?

Key Hypotheses

Quick Commands

# Run inference
python analysis/run_main_experiment.py --model deepseek-v3.2

# Generate scorecard
python generate_master_scorecard.py

# View evaluation dashboard
python evaluation/dashboard.py

Key Artifacts

View Logs

• Langfuse: https://us.cloud.langfuse.com
• Local Dashboard: python evaluation/dashboard.py

---

Last Updated: January 9, 2026 Status: Analysis Complete, Write-up Phase