🕵️ Lies We Can See — Dataset

Data for the paper Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions.

MINEAMONGUS is a 3D multimodal Among Us sandbox in Minecraft where imposter agents deceive crewmates through both what they say and what they do — stalking a target, checking for witnesses, fleeing an unreported body, then accusing the crewmate who found it. ARIA is the configurable VLM-agent harness that runs them. This repository holds the RQ2 artifacts: the cross-VLM tournament game data, the crewmate behavior-cloning training set, and the LLM-as-a-judge deception-atom scoring.

Code / environment: https://github.com/JunseoKim0103/Lies-We-Can-See Private / research use.

📂 Contents

lies-we-can-see/
├── RQ2_train/                     # cross-VLM tournament games + crewmate SFT data
│   ├── case1A.tar … case2B.tar    #   576 raw game runs (logs, traces, frames)
│   ├── all_runs.csv / .json       #   per-run aggregate results
│   └── sft_used/                  #   crewmate detection SFT data (ShareGPT)
└── RQ2_llm_as_a_judge/            # deception-atom scoring of top-3/worst-3 imposters
    ├── input.tar                  #   576 judged games (judge inputs) + _index.csv
    └── output/                    #   per-game findings + analysis / paper results

---

RQ2_train/

The RQ2 cross-VLM tournament: every VLM backbone plays imposter against every other across four ARIA configurations (Case 1A/1B/2A/2B).

Raw games — caseXX.tar

One tar per case, 144 matchups each (576 games total). Each game has 8 players and full per-game artifacts:

caseXX/trial1/matchupMM/<YYYY-MM-DD_HH-MM-SS>/
    game.log, main.log, <Player>.log, agents.yaml, token_usage.txt
    <Player>/trace.json                 # structured per-decision ARIA trace
    <Player>/images/*.jpeg | *.png      # first-person frames

Detection training data — sft_used/

Supervised fine-tuning samples for the crewmate deception detector (behavior cloning): every ARIA crewmate call from games the crewmate wins. Each call is an (X, S, Y) example — text prompt X, optional egocentric RGB state S, VLM response Y — spanning all crewmate decision surfaces (planning, memory, reflection, skill memory, and the REPORT / SURVEILLANCE / EMERGENCY / MEETING / VOTE / MOVE / MISSION modules). A subset carry an image; the rest are text-only.

Images. Paths in the images field are repo-relative — extract the case tars inside RQ2_train/ and they resolve:

cd RQ2_train && for t in case*.tar; do tar -xf "$t"; done

Recipe. Base model Qwen/Qwen3.5-9B, LoRA (rank 16, α 32, dropout 0.05, vision tower frozen), AdamW lr 1e-4 cosine (0.05 warmup), batch 8, 2 epochs, bf16, qwen2-vl chat template, 4,096-token cutoff, trained in LLaMA-Factory.

⚠️ Evaluation — avoid identity contamination

In this data the impostors are always James and Olivia (fixed name ↔ color ↔ role pairings), so a model fine-tuned on it can learn a name/color shortcut instead of reading behavior. When you evaluate such a model, change the impostor identities in the environment setup (e.g. make Steve and Jason the impostors) so the shortcut cannot fire.

---

RQ2_llm_as_a_judge/

LLM-as-a-judge scoring of deception atoms for the imposters with the highest / lowest win rates. Self-contained — it carries its own judge input logs.

• 576 judged games = 6 imposter models × 96 games, case-balanced (Case 1A/1B/2A/2B × 144; each case trial1/trial2 × 72). File prefixes top- / worst-.
• Judge: qwen3.6-27b-thinking, two-pass (pass1 non-verbal atoms, pass2 verbal / meeting transcript), noise-filtered, imposter-only.
• top-3 = gemini-3-flash-preview, kimi-k2.5, gemini-3.1-flash-lite-preview
• worst-3 = qwen3.5-9b, gemma4-26b-a4b, gemini-2.5-flash

RQ2_llm_as_a_judge/
├── input.tar          # judge inputs (game logs) — tar -xf → input/
├── _index.csv         # one row per judged game (case, trial, matchup, rank, models, …)
└── output/
    ├── findings_per_run.tar          # 576 × 2-pass *.findings.json (atom labels)
    ├── paper_rq2_quantitative.md     # the paper's RQ2 numbers
    ├── atom_top3_vs_worst3.md, atom_correlation_with_imp_wr.md, atom_diversity.md
    └── atom_*.csv, schedule.json, runner.log, console.log

Do not confuse the judge model (qwen3.6-27b-thinking) with the players' models that appear in the logs.

---

Intended use

For research on agent deception and alignment. The deceptive behaviors here are the object of study, not a capability to deploy.

Citation

@misc{mineamongus2026,
  title  = {Lies We Can See: Joint Verbal and Non-Verbal Deception
            by VLM Agents in Embodied Social Interactions},
  note   = {Under review},
  year   = {2026}
}