ABM_based_LLMS

Generative agent-based model (GABM) of depression contagion on social networks, plus the PHQ-9 assessment pipeline it depends on. LLM agents (Qwen3.5-27B via vLLM) write social-media posts from a persona + PHQ-9 profile; depression is re-assessed from post histories by either an optimized LLM prompt or a MentalBERT+MLP regressor; prompts are optimized with TextGrad. The repo grew across two MSc theses (CS = generative/assessment model performance + sensitivity analyses; GABM/CLS = network simulation) and the paper LLM_agent_Depression__PNAS_Nexus.

Layout

Documentation

• docs/ROADMAP.md — narrative code roadmap: the pipeline stages, tagged with which manuscript (CS thesis / GABM thesis / PNAS paper) uses each
• docs/STRUCTURE.md — folder-by-folder "what is used where" (live / historical / frozen / binned) + known quirks
• docs/SCRIPTS.md — every driver script and SLURM job (inputs, outputs, GPU/walltime)
• docs/prompt_optimizer.md — TextGrad prompt optimization: method, file outline, how to run
• docs/PAPER_MAP_PNAS.md — PNAS paper figure/table provenance
• docs/THESIS_MAP_CS.md — CS thesis chapter → code map
• docs/THESIS_MAP_GABM.md — Computational Science (GABM) thesis chapter → code map
• src/README.md — module map + the hand-run analysis CLIs
• data/README.md — data provenance map: where every dataset comes from (real vs hand-made vs generated, and by which script) and what it is used for
• per-folder NOTES.md files (under data/, bin/) — file-level status/cleanup details for the data trees

Getting started

uv venv .venv_vllm
source .venv_vllm/bin/activate
uv pip install -r requirements_vllm.txt

Python modules run from the repo root with src on the path:

PYTHONPATH=src python -m utils.<module> [...]

GPU work goes through SLURM: sbatch jobs/<name>.job (see docs/SCRIPTS.md).

Provenance

• PNAS paper figures/tables → docs/PAPER_MAP_PNAS.md
• CS thesis (Computer_Science_Transformer/) → docs/THESIS_MAP_CS.md
• GABM thesis (Computational_Science_GABM/) → docs/THESIS_MAP_GABM.md

Known caveat — assessment and generation results come from different datasets

Three generation prompts produced the synthetic post data:

Order actually run: high-fidelity set first → TextGrad optimization of the PHQ-9 assessment prompt on that set → TextGrad + human optimization of the generation prompt, which was never fed back into a regenerated high-fidelity set (no compute/time budget once the focus moved to the GABM thesis). The clean order would have been: optimize generation → regenerate high-fidelity → optimize assessment.

Consequences per pipeline:

• LLM assessment prompt (data/test_post/optimized_phq9/) — optimized and evaluated on the base set only; on human-optimized data it is evaluated (shift table) but never re-optimized.
• MentalBERT+MLP — trained on the base set (bert_regression/), then fine-tuned on human-optimized data (bert_regression_finetuned/, via scripts/assessment/run_finetune.sh). The fine-tuned regressor is the one the simulation uses, so GABM-thesis results are internally consistent.
• Per-band MAE/bias figures come from the base set; the S-BERT adjacent-band and class-similarity figures come from minimal + human-optimized SA runs (data/sensitivity/). Any claim linking assessment error to linguistic overlap crosses distributions unless it uses the base-set version.

Contained, not fatal: the bias direction (over-estimate mild, under-estimate severe) and the rising adjacent-band similarity reproduce on every distribution and in both assessor families, and the mismatch makes the shift table (scripts/assessment/run_eval_comparison.sh) a fair symmetric OOD test rather than a self-test. A within-distribution version of the linguistics↔error link already exists: the S-BERT class-similarity matrix computed on the base-set BERT test split (CS thesis App. B, fig:phq9_confusion_cosim).

Manuscript-side TODOs (fix there, not here) — agreed with D. Roy 2026-08-26 to stay a background limitation, provided each result set names its dataset:

• Both Methods sections currently imply the high-fidelity set came from the optimized prompts; name its actual generating prompt where it is introduced.
• State the dataset in the assessment figure/table captions (CS thesis Figs 4.6–4.8 + tab:phq9-estimators; paper tab:estimators, fig:prompt_comparison) and in Figs 4.3b / 4.4.
• Limitations: one sentence that both assessors were calibrated on the base set and only MentalBERT+MLP was re-fit and re-evaluated on the human-optimized one.
• Use one dataset name throughout; the CS thesis currently mixes "high fidelity", "non-optimized" and "the synthetic dataset" for the same data.
• CS thesis Methods (§ Datasets and Splits) states the base set as "≈1,200 post-blocks (≈12,000 posts)" — a 10× slip. On-disk counts and the split figures both manuscripts use (9,638+1,237+1,125 = 12,000 blocks) give 12,000 blocks / 120,000 posts; the PNAS manuscript's numbers are correct.