Official repository for Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues.

Structure

SCRIPTS/
├── dataset/           # Dataset files
│   ├── english_combined.csv    # English dialogue dataset
│   └── korean_combined.csv     # Korean dialogue dataset
│
├── experiment/        # Experiment code (see experiment/README.md)
│   ├── core/         # Evaluation, prompts, utilities
│   ├── scoring/      # Scoring and parsing
│   ├── analysis/     # Analysis scripts
│   ├── models/       # LLM model wrappers
│   └── config.py     # Configuration
│
└── results/          # Generated results (auto-created)

Usage

See experiment/README.md for detailed documentation, API reference, and usage examples.

License

CC-BY-NC-ND

Citation

@misc{kim2025loversfriendsevaluatingllms,
      title={Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues}, 
      author={Eunsu Kim and Junyeong Park and Juhyun Oh and Kiwoong Park and Seyoung Song and A. Seza Dogruoz and Najoung Kim and Alice Oh},
      year={2025},
      eprint={2510.19028},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.19028}, 
}