RealFin

RealFin is a benchmark for evaluating whether large language models can reason about finance when a user leaves a crucial premise unstated.

The benchmark focuses on logical underdetermination in financial reasoning: instead of asking models to directly solve fully specified finance questions, RealFin asks models to identify the missing condition that is necessary to make the problem answerable.

Overview

Financial questions often look answerable on the surface while silently depending on an unstated assumption, condition, or institutional rule. RealFin converts finance exam-style questions into condition-missing multiple-choice questions. Each item contains:

• an underspecified finance question;
• four candidate missing conditions;
• one correct condition that is necessary for resolving the question.

This setup evaluates whether a model can recognize what information is missing before attempting to answer.

Dataset

The released benchmark files are in datasets/:

Each CSV file uses UTF-8 encoding and contains the following columns:

Task Format

Given the Question and four candidate options, a model should select the option that identifies the critical missing premise. The intended output is one of A, B, C, or D.

Example prompt template:

You are given a financial reasoning question that may be underspecified.
Select the option that provides the most critical missing condition required to answer it.

Question:
{Question}

A. {Option A}
B. {Option B}
C. {Option C}
D. {Option D}

Answer with only one letter: A, B, C, or D.

Code

Model evaluation scripts are provided in models/. The normal_* scripts evaluate models on fully specified questions, while the mask_* scripts evaluate models on the condition-missing setting.

The scripts are intended as reference implementations and may require local model paths, API keys, or environment-specific configuration before running.

License

This repository is released under the license in LICENSE.

Citation

If you use these datasets and scripts, please cite:

@inproceedings{dai2026realfin,
  title     = {RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?},
  author    = {Dai*, Yuyang and Lin*, Yan and Xie, Zhuohan and Wang, Yuxia},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year      = {2026}
}