Results Directory

All evaluation outputs, reorganized by benchmark × run mode × model.

Layout

results/
├── ehr_bench/              # text-only EHR-Bench (1800 rows, 45 tasks)
│   ├── agentic/            # multi-turn tool-calling via deploy_agent.py
│   │   ├── smoke20/<model>/
│   │   ├── full1800/<model>/      ← rollout outputs (results.jsonl, run.log, per-region shards)
│   │   └── scored/                ← evaluate_results.py output (WIP slot)
│   └── oneshot/            # single Bedrock invoke via deploy_reasoning_model.py
│       ├── smoke20/<model>/
│       └── full1800/<model>/
│
├── agent_ehr_bench/        # AgentEHR-Bench (600 rows, 6 MIMIC tasks) ← symlinks into DeepMed-eval
│   └── agentic/
│       └── subset600/<model>/     ← symlinks to /fsx-shared/juncheng/DeepMed-eval/openresearcher_ehr/subsets_600_*
│
├── mm_bench/               # multimodal EHR-Bench (2703 rows, 6 tasks)
│   ├── agentic/            # multi-turn via deploy_agent_mm.py (+ image MCP, + EHR MCP)
│   │   ├── full2703/<model>/      ← rollout outputs (Opus=real dir, others=symlinks into _multi_eval_raw/)
│   │   ├── full2703/_multi_eval_raw/  ← raw 5-model fan-out run (multi_eval/full5_*)
│   │   └── scored/<model>/        ← scorer_mm.py --vocab output
│   └── oneshot/            # single Bedrock invoke via deploy_reasoning_model_mm.py
│       ├── smoke20/<model>/
│       ├── full2703/<model>/
│       └── scored/                ← full2703 scored (model=<model>), smoke20 scored (prefix smoke20_*)
│
└── _archive/               # superseded / half-finished / early-pilot runs kept for audit

Model slug convention

All leaf directories use snake-cased slugs:

Canonical "where are the numbers"

• EHR-Bench agentic full: ehr_bench/agentic/full1800/<model>/results.jsonl
• EHR-Bench one-shot full: ehr_bench/oneshot/full1800/<model>/results.jsonl
• AgentEHR-Bench agentic: agent_ehr_bench/agentic/subset600/<model>/results.jsonl
• MM agentic full: mm_bench/agentic/full2703/<model>/ (Opus: results.jsonl; others: merged_unique.jsonl)
• MM agentic scored (vocab): mm_bench/agentic/scored/<model>/summary_vocab.{json,md}
• MM one-shot full: mm_bench/oneshot/full2703/<model>/results.jsonl
• MM one-shot scored (vocab): mm_bench/oneshot/scored/<model>/summary.{json,md}

Notes on the MM agentic symlinks

The five non-Anthropic MM agentic runs were produced by run_multi_model_eval.py on 2026-04-21, which uses a different folder layout (<run>/scored_vocab/<model>/ with hyphenated model names). For browsability we symlink underscore-slugged peer directories pointing into _multi_eval_raw/:

mm_bench/agentic/full2703/claude_sonnet_4_6  ->  _multi_eval_raw/full5_20260421T091517Z/claude-sonnet-4-6
mm_bench/agentic/scored/claude_sonnet_4_6    ->  ../full2703/_multi_eval_raw/full5_20260421T091517Z/scored_vocab/claude-sonnet-4-6

The Opus MM agentic (mm_bench/agentic/full2703/claude_opus_4_6) is a real directory because it was produced by a separate Opus-solo run on 2026-04-20.

Archive

_archive/ contains:

• Half-completed runs (ehr_bench_opus46_1800_pre_ssl_retry, ehr_bench_opus46_retry693_ssl_fix)
• Duplicate/salvage-patched re-smokes (ehr_bench_smoke20_salvage_patched_*)
• Early pilot MM runs (mm_agentic_smoke5_early, mm_agentic_mix6_early, mm_agentic_images_pilot_{a,b}, etc.)
• Pre-merge retry outputs (ehr_bench_oneshot_full1800_kimi_k2_5_pre_retry, ehr_bench_oneshot_retry_{kimi,minimax}_*)
• Earlier non-vocab MM scored outputs (mm_agentic_opus_scored_novocab, mm_agentic_prepared_scored_novocab)

Safe to delete after ~30 days if no one has needed to reference them.