Reading List: Advances in Machine Theory of Mind

This illustration is generated using DALL·E 3

Overview

Citation

This is a curated list of related literature and resources for machine theory of mind (ToM) research. Last Update: Dec 30th, 2024.

If you find our work useful, please give us credit by citing:

@inproceedings{ma2023towards,
  title={Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models},
  author={Ma, Ziqiao and Sansom, Jacob and Peng, Run and Chai, Joyce},
  booktitle={Findings of the Association for Computational Linguistics: EMNLP 2023},
  year={2023}
}

Contributors

• Main Contributors: Martin Ziqiao Ma
• Active Contributors: X. Angelo Huang, Karahan Sarıtaş, Run Peng
• Previous Contributors: Jacob Sansom, Pony Zhang

How To Contribute

Welcome to contribute to our paper list or be a collaborator!

• To add missing papers: Please create an issue or pull request, so the team can make the update.
• To become a contributor: Please drop an email to Martin.

Table of Contents

• Reading List: Advances in Machine Theory of Mind
  • Overview
    • Citation
    • Contributors
    • How To Contribute
  • Table of Contents
  • 1. ToM Community Resources
    • 1.1 Workshops
    • 1.2 Talks and Tutorials
    • 1.3 Tools
  • 2. Machine ToM Surveys and Position Papers
  • 3. Cognitive Underpinnings of ToM
    • 3.1 Definition and Importance of ToM in Human Cognition (Selected)
    • 3.2 Taxonomies of ToM and Mental States
  • 4. Computational Inquiry to ToM in Foundation Models
    • 4.1 Probing Intrinsic Mental States
    • 4.1 Evidence for Understanding Extrinsic Mental States
    • 4.2 Counter-Evidence for Understanding Extrinsic Mental States
  • 4. ToM Benchmarks and Platforms
    • 4.1 Benchmarks from original Survey
    • 4.2 New Benchmarks
  • 5. Computational Modeling of ToM
    • 5.1 Learning Latent Representation for ToM
    • 5.2 Learning (Neural-)Symbolic Representation for ToM
    • 5.3 Prompting and In-Context Learning for ToM in LLMs
    • 5.4 Bayesian and (Inverse) Reinforcement Learning Based ToM Modeling
    • 5.5 Other ToM Modeling
  • 6. ToM Application
    • 6.1 Pragmatics and Instruction Generation/Following
    • 6.2 Dialogue Processing and Generation
    • 6.3 Language Acquisition
    • 6.4 Human-AI Interactions
    • 6.5 Explainable AI
    • 6.6 Healthcare
    • 6.7 Privacy

1. ToM Community Resources

1.1 Workshops

• (ToM 2024) 2nd Workshop on Theory-of-Mind @ ICML 2024. [Web]
• (ToM 2023) 1st Workshop on Theory-of-Mind @ ICML 2023. [Web]

1.2 Talks and Tutorials

• To be updated

1.3 Tools

• (ToM 2023) The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agents. [Paper][Web]
• (Preprint 2023) SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents. [Paper][Web]

2. Machine ToM Surveys and Position Papers

• (Preprint 2025) A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks [Paper]
• (EMNLP Main 2024) Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding [Paper]
• (EMNLP Findings 2023) Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models. [Paper][Data]
• (Preprint 2023) A Review on Machine Theory of Mind. [Paper]
• (EMNLP Findings 2022) Language Models as Agent Models. [Paper]
• (RO-MAN 2022) Understanding Intention for Machine Theory of Mind: A Position Paper. [Paper]
• (Psychological Medicine 2020) Knowing Me, Knowing you: Theory of Mind in AI. [Paper]
• (Neuropsychologia 2020) Theory of Mind and Decision Science: Towards a Typology of Tasks and Computational Models. [Paper]
• (AI 2018) Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems. [Paper]
• (Preprint 2017) It Takes Two to Tango: Towards Theory of AI's Mind. [Paper]
• (AI 2016) Integrating Social Power Into The Decision-making of Cognitive Agents. [Paper]

3. Cognitive Underpinnings of ToM

3.1 Definition and Importance of ToM in Human Cognition (Selected)

• (Premack et al., 1978) Does the Chimpanzee Have a Theory of Mind? [Paper]
• (Dennett, 1988) Précis of The Intentional Stance. [Paper]
• (Gopnik et al., 1992) Why the Child's Theory of Mind Really Is a Theory. [Paper]
• (Baron-Cohen, 1992) Mindblindness: An Essay on Autism and Theory of Mind. [Book]
• (Blakemore et al,. 2001) From the Perception of Action to the Understanding of Intention. [Paper]
• (Ho et al,. 2022) Planning With Theory Of Mind. [Paper]

3.2 Taxonomies of ToM and Mental States

• (ToM 2023) EPITOME: Experimental Protocol Inventory for Theory Of Mind Evaluation. [Paper]
• (Stack et al., 2022) Framework for a Multi-dimensional Test of Theory of Mind for Humans and AI Systems. [Paper]
• (Osterhaus et al., 2022) Looking for the Lighthouse: A Systematic Review of Advanced Theory-of-mind Tests beyond Preschool. [Paper]
• (Beaudoin et al., 2020) Systematic Review and Inventory of Theory of Mind Measures for Young Children. [Paper]

4. Computational Inquiry to ToM in Foundation Models

4.1 Probing Intrinsic Mental States

• (EACL 2023) Methods for Measuring, Updating, and Visualizing Factual Beliefs in Language Models. [Paper][Code]
• (EMNLP Findings 2021) Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding. [Paper][Data]
• (ACL 2021) Implicit Representations of Meaning in Neural Language Models. [Paper][Code]

4.1 Evidence for Understanding Extrinsic Mental States

• (Preprint 2023) Unveiling Theory of Mind in Large Language Models: A Parallel to Single Neurons in the Human Brain. [Paper]
• (Preprint 2023) Sparks of Artificial General Intelligence: Early experiments with GPT-4. [Paper]
• (Preprint 2023) Theory of Mind Might Have Spontaneously Emerged in Large Language Models. [Paper][Web][Code]
• (EMNLP Findings 2021) Effectiveness of Pre-training for Few-shot Intent Classification. [Paper][Code]

4.2 Counter-Evidence for Understanding Extrinsic Mental States

• (CONLL 2023) Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Tests [Paper]
• (EMNLP 2023) FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions. [Paper][Code]
• (EACL 2024) Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models. [Paper]
• (Preprint 2023) Limitation of Theory of Mind In Large Language Model: Anthropomorphize Religous Figure. [Paper]
• (Preprint 2023) Does ChatGPT have Theory of Mind? [Paper]
• (Preprint 2023) Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. [Paper]
• (AI Review 2023) Mind the Gap: Challenges of Deep Learning Approaches to Theory of Mind. [Paper]
• (Preprint 2022) Do Large Language Models Know what Humans Know? [Paper]
• (Preprint 2022) Large Language Models Are Not Zero-shot Communicators. [Paper][Web][Code][Data]
• (EMNLP 2022) Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs. [Paper]

4. ToM Benchmarks and Platforms

4.1 Benchmarks from original Survey

A taxonomized review of existing benchmarks for machine ToM and their settings under ATOMS. We further break beliefs into first-order beliefs (1st) and second-order beliefs or beyond (2nd+); and break intentions into Action intentions and Communicative intentions. Tasks are divided into Inference, Question Answering, Natural Language Generation, MultiAgent Collaboration, and MultiAgent Competition. Input modalities consist of Text (Human, AI, or Template) and Nonlinguistic ones. The latter further breaks into Cartoon, Natural Images, Chess, 2D Grid World, and 3D Simulation. The Situatedness is divided into None, Passive Perceiver, and Active Interactor. Symmetricity refers to whether the tested agent is co-situated and engaged in mutual interactions with other ToM agents.

4.2 New Benchmarks

• (ACL 2024) ToMBench: Benchmarking Theory of Mind in Large Language Models [Paper]
• (ACL 2024) OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models [Paper]
• (AAAI 2025) ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind [Paper][Data]
• (Preprint 2026) PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting [Paper][Data]

5. Computational Modeling of ToM

5.1 Learning Latent Representation for ToM

• (IJCAI 2023) Towards Collaborative Plan Acquisition through Theory of Mind Modeling in Situated Dialogue. [Paper]
• (EMNLP 2021) MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks. [Paper][Code]
• (RO-MAN 2021) Deep Interpretable Models of Theory of Mind. [Paper]
• (EMNLP 2020) RMM: A Recursive Mental Model for Dialog Navigation. [Paper][Code]
• (ICML 2018) Machine Theory of Mind. [Paper][Code]

5.2 Learning (Neural-)Symbolic Representation for ToM

• (ACL 2023) Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker. [Paper][Code]
• (ToM 2023) The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputs. [Paper][Code]

5.3 Prompting and In-Context Learning for ToM in LLMs

• (EMNLP 2024) A Notion of Complexity for Theory of Mind via Discrete World Models [Paper]
• (EMNLP 2023) Theory of Mind for Multi-Agent Collaboration via Large Language Models. [Paper]
• (Preprint 2023) How FaR Are Large Language Models From Agents with Theory-of-Mind? [Paper]
• (Preprint 2023) Violation of Expectation via Metacognitive Prompting Reduces Theory of Mind Prediction Error in Large Language Models. [Paper][Code]
• (Preprint 2023) CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society. [Paper][Code]
• (Preprint 2023) Boosting Theory-of-Mind Performance in Large Language Models via Prompting. [Paper][Data]
• (Preprint 2023) Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4. [Paper][Code]

5.4 Bayesian and (Inverse) Reinforcement Learning Based ToM Modeling

• (ToM 2023) Theory of Mind as Intrinsic Motivation for Multi-Agent Reinforcement Learning. [Paper]
• (ToM 2023) Iterative Machine Teaching for Black-box Markov Learners. [Paper]
• (ToM 2023) Between Prudence and Paranoia: Theory of Mind Gone Right, and Wrong. [Paper]
• (ToM 2023) Emergent Deception and Skepticism via Theory of Mind. [Paper]
• (ToM 2023) How To Make Social Decisions in a Heterogeneous Society? [Paper]
• (ICML 2022) Symmetric Machine Theory of Mind. [Paper][Code]
• (ICML 2021) Few-shot Language Coordination by Modeling Theory of Mind. [Paper][Code]
• (CogSci 2020) Improving Multi-Agent Cooperation using Theory of Mind. [Paper][Code]
• (Preprint 2019) Modeling Theory of Mind in Multi-Agent Games Using Adaptive Feedback Control. [Paper]
• (EmeComm 2019) Emergence of Theory of Mind Collaboration in Multiagent Systems. [Paper][Code]
• (Current Opinion in Behavioral Sciences 2019) Theory of Mind as Inverse Reinforcement Learning. [Paper]

5.5 Other ToM Modeling

• (ToM 2023) Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective. [Paper]
• (ToM 2023) Inferring the Future by Imagining the Past. [Paper]
• (ToM 2023) Inferring the Goals of Communicating Agents from Actions and Instructions. [Paper]
• (RSS 2015) Grounding English Commands to Reward Functions. [Paper]
• (CogSci 2011) Bayesian Theory of Mind: Modeling Joint Belief-Desire Attribution. [Paper][Web]

6. ToM Application

6.1 Pragmatics and Instruction Generation/Following

• (ToM 2023) Towards a Better Rational Speech Act Framework for Context-aware Modeling of Metaphor Understanding. [Paper]
• (ACL Findings 2023) Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models. [Paper]
• (ACL 2022) Learning to Mediate Disparities Towards Pragmatic Communication. [Paper][Code]
• (ICML 2021) Few-shot Language Coordination by Modeling Theory of Mind. [Paper][Code]
• (Science 2012) Predicting Pragmatic Reasoning in Language Games. [Paper]

6.2 Dialogue Processing and Generation

• (ToM 2023) MindDial: Belief Dynamics Tracking with Theory-of-Mind Modeling for Neural Dialogue Generation. [Paper]
• (ACL Findings 2023) Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind. [Paper][Code]
• (SIGDIAL 2022) Towards Socially Intelligent Agents with Mental State Transition and Human Utility. [Paper]
• (EMNLP 2020) RMM: A Recursive Mental Model for Dialog Navigation. [Paper][Code]

6.3 Language Acquisition

• (ICLR 2023) Computational Language Acquisition with Theory of Mind. [Paper][Code]
• (Preprint 2023) Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Theory of Mind. [Paper][Code]

6.4 Human-AI Interactions

• (ToM 2023) Preference Proxies: Evaluating Large Language Models in Capturing Human Preferences in Human-AI Tasks. [Paper]
• (CHI 2021) Towards Mutual Theory of Mind in Human-AI Interaction: How Language Reflects What Students Perceive About a Virtual Teaching Assistant. [Paper]
• (Springer 2002) Theory of Mind for a Humanoid Robot [Paper]

6.5 Explainable AI

• (iScience 2021) CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models. [Paper]

6.6 Healthcare

• (ToM 2023) Discovering User Types: Mapping User Traits by Task-Specific Behaviors in Reinforcement Learning. [Paper]

6.7 Privacy

• (Preprint 2023) Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory. [Paper][Code]