README
FinFIRST: Financial Information Retrieval, Sourcing and Traceability
Released alongside Ling-3.0-flash-Fin, FinFIRST is an open benchmark for evaluating whether financial search agents can produce answers that are not only correct, but also supported by authoritative, timely, and verifiable evidence. It was developed by Ant Group, with professional support from the investment banking team at China International Capital Corporation Limited (CICC).
Financial research requires more than finding a number: an agent must also verify the entity, reporting period, currency, unit, definition, and data version. FinFIRST therefore uses atomic rubrics to evaluate the full research process rather than relying solely on final-answer matching.
Dataset
FinFIRST V1 contains 123 tasks authored and verified by more than 50 finance professionals. The tasks cover China, the United States, and other markets, with research targets spanning companies, industries, regional economies, and financial instruments. All tasks use publicly accessible sources.
| Dimension | Value |
|---|---|
| Chinese / English | 74 (60.2%) / 49 (39.8%) |
| Multiple related subquestions | More than 80% |
| Explicit computation required | 61.8% |
| Multiple sources required | 32.5% |
| No prescribed source | 75.6% |
| Atomic criteria | 701 |
| Total rubric weight | 12,300 points |
Schema
Each record contains 24 string-or-null fields:
| Group | Fields | Description |
|---|---|---|
| Core task data | original_id, id, query, answer | Identifiers, research question, and reference answer |
| Evaluation | rubric, rubric_annotated | Atomic criteria and their capability annotations |
| Task structure | question_decomposition, relation_type, conditional_filtering | Subquestion decomposition and structural metadata |
| Research target | subject_category, metric, measurement_scope, region | Subject, measure, scope, and market |
| Time and verification | time_constraint_type, time_description, verification_method | Time constraints and verification approach |
| Output requirements | calculation_type, output_format_requirement, language | Calculation, format, and language requirements |
| Sources and intent | source_specification, source_accessibility, source_mainstreamness, source_count, intent_category | Source requirements and task intent |
The answer field is stored as a string. Missing values are represented as null, while the literal category value None remains a string. Percentages imported from Excel are converted to the requested display format; for example, 0.0083 is stored as 0.83%.
Evaluation Design
Each expected solution is decomposed into independently assessable criteria across three dimensions:
- Raw-information acquisition: retrieving the correct facts and values for the required entity, period, unit, definition, and data version.
- Source verification: using sources that are authoritative, relevant, and valid for the required time period and version.
- Computation and answer formation: calculating, reasoning, and synthesizing correctly, with conclusions supported by the retrieved evidence.
This design gives credit for valid intermediate work, identifies correct answers with incomplete evidence, and attributes failures to specific stages of the research process.
Atomic Rubric Example
The example below requires an agent to combine Meta's official disclosures with an IAB/PwC industry report. The reference answer is 72.45%, while the rubric separately evaluates source identification, data extraction, aggregation, calculation, precision, and units.
Protocol and Metrics
Fifteen model configurations were evaluated using the same ReAct-style agent framework and the same Web Search, Visit, and Python tools. GLM-5.1 served as the automated judge. On 50 randomly sampled evaluation instances annotated and adjudicated by eight finance professionals, the judge achieved a Cohen's κ of 0.816 against the human reference labels at the atomic-criterion level.
| Metric | Definition |
|---|---|
| Atomic | Unweighted pass rate across all atomic criteria |
| Loose Pass | Partial-credit score based on expert-assigned criterion weights |
| Strict Pass | Percentage of tasks for which every atomic criterion passes |
| Unsupported-Correct Rate (UCR) | Among correct answers, the share that does not satisfy the complete evidence requirements; lower is better |
Dataset Construction
FinFIRST was created through three stages:
- Scenario-driven preprocessing: Real financial use cases were translated into an 18-field task taxonomy, a six-axis coverage plan, and a source map covering 138 types of financial sources.
- Expert task construction: Finance professionals authored candidate tasks and assembled reference packages containing answers, sources, key data, definitions, calculations, valid alternatives, and numerical tolerances. Each solution was then decomposed into atomic criteria.
- Quality control: Candidates underwent value-and-scope review, independent re-solving, cross-validation and adjudication, rubric audit, model stress testing, and consistency checking. Financial experts made all revision and inclusion decisions.
The final acceptance rate was 9.78%, yielding 123 tasks.
Results
Overall Performance
GPT-5.6-Sol records the highest Strict Pass score at 71.54%. Claude-Opus-5 achieves 87.61% on Loose Pass but 69.11% on Strict Pass, showing that strong partial performance can still conceal missed requirements. Across all evaluated models, computation and answer formation remain weaker than raw-information acquisition.
Unsupported-Correct Rate
Of 1,150 instances with a correct final answer, 201 lack complete supporting evidence, resulting in a micro-averaged UCR of 17.48%. Among the evaluated open-weight models, Ling-3.0-flash-Fin achieves both the lowest UCR and the highest source-verification score.
Slice Analysis
Performance declines from easy to hard tasks, and multi-source tasks are consistently more challenging than single-source tasks. Results also vary across languages and markets.
License
FinFIRST is released under the Apache License 2.0.
Citation
Paper and citation information will be added when available.
Contributors
15 commits
Similar repos
README
FinFIRST: Financial Information Retrieval, Sourcing and Traceability
Released alongside Ling-3.0-flash-Fin, FinFIRST is an open benchmark for evaluating whether financial search agents can produce answers that are not only correct, but also supported by authoritative, timely, and verifiable evidence. It was developed by Ant Group, with professional support from the investment banking team at China International Capital Corporation Limited (CICC).
Financial research requires more than finding a number: an agent must also verify the entity, reporting period, currency, unit, definition, and data version. FinFIRST therefore uses atomic rubrics to evaluate the full research process rather than relying solely on final-answer matching.
Dataset
FinFIRST V1 contains 123 tasks authored and verified by more than 50 finance professionals. The tasks cover China, the United States, and other markets, with research targets spanning companies, industries, regional economies, and financial instruments. All tasks use publicly accessible sources.
| Dimension | Value |
|---|---|
| Chinese / English | 74 (60.2%) / 49 (39.8%) |
| Multiple related subquestions | More than 80% |
| Explicit computation required | 61.8% |
| Multiple sources required | 32.5% |
| No prescribed source | 75.6% |
| Atomic criteria | 701 |
| Total rubric weight | 12,300 points |
Schema
Each record contains 24 string-or-null fields:
| Group | Fields | Description |
|---|---|---|
| Core task data | original_id, id, query, answer | Identifiers, research question, and reference answer |
| Evaluation | rubric, rubric_annotated | Atomic criteria and their capability annotations |
| Task structure | question_decomposition, relation_type, conditional_filtering | Subquestion decomposition and structural metadata |
| Research target | subject_category, metric, measurement_scope, region | Subject, measure, scope, and market |
| Time and verification | time_constraint_type, time_description, verification_method | Time constraints and verification approach |
| Output requirements | calculation_type, output_format_requirement, language | Calculation, format, and language requirements |
| Sources and intent | source_specification, source_accessibility, source_mainstreamness, source_count, intent_category | Source requirements and task intent |
The answer field is stored as a string. Missing values are represented as null, while the literal category value None remains a string. Percentages imported from Excel are converted to the requested display format; for example, 0.0083 is stored as 0.83%.
Evaluation Design
Each expected solution is decomposed into independently assessable criteria across three dimensions:
- Raw-information acquisition: retrieving the correct facts and values for the required entity, period, unit, definition, and data version.
- Source verification: using sources that are authoritative, relevant, and valid for the required time period and version.
- Computation and answer formation: calculating, reasoning, and synthesizing correctly, with conclusions supported by the retrieved evidence.
This design gives credit for valid intermediate work, identifies correct answers with incomplete evidence, and attributes failures to specific stages of the research process.
Atomic Rubric Example
The example below requires an agent to combine Meta's official disclosures with an IAB/PwC industry report. The reference answer is 72.45%, while the rubric separately evaluates source identification, data extraction, aggregation, calculation, precision, and units.
Protocol and Metrics
Fifteen model configurations were evaluated using the same ReAct-style agent framework and the same Web Search, Visit, and Python tools. GLM-5.1 served as the automated judge. On 50 randomly sampled evaluation instances annotated and adjudicated by eight finance professionals, the judge achieved a Cohen's κ of 0.816 against the human reference labels at the atomic-criterion level.
| Metric | Definition |
|---|---|
| Atomic | Unweighted pass rate across all atomic criteria |
| Loose Pass | Partial-credit score based on expert-assigned criterion weights |
| Strict Pass | Percentage of tasks for which every atomic criterion passes |
| Unsupported-Correct Rate (UCR) | Among correct answers, the share that does not satisfy the complete evidence requirements; lower is better |
Dataset Construction
FinFIRST was created through three stages:
- Scenario-driven preprocessing: Real financial use cases were translated into an 18-field task taxonomy, a six-axis coverage plan, and a source map covering 138 types of financial sources.
- Expert task construction: Finance professionals authored candidate tasks and assembled reference packages containing answers, sources, key data, definitions, calculations, valid alternatives, and numerical tolerances. Each solution was then decomposed into atomic criteria.
- Quality control: Candidates underwent value-and-scope review, independent re-solving, cross-validation and adjudication, rubric audit, model stress testing, and consistency checking. Financial experts made all revision and inclusion decisions.
The final acceptance rate was 9.78%, yielding 123 tasks.
Results
Overall Performance
GPT-5.6-Sol records the highest Strict Pass score at 71.54%. Claude-Opus-5 achieves 87.61% on Loose Pass but 69.11% on Strict Pass, showing that strong partial performance can still conceal missed requirements. Across all evaluated models, computation and answer formation remain weaker than raw-information acquisition.
Unsupported-Correct Rate
Of 1,150 instances with a correct final answer, 201 lack complete supporting evidence, resulting in a micro-averaged UCR of 17.48%. Among the evaluated open-weight models, Ling-3.0-flash-Fin achieves both the lowest UCR and the highest source-verification score.
Slice Analysis
Performance declines from easy to hard tasks, and multi-source tasks are consistently more challenging than single-source tasks. Results also vary across languages and markets.
License
FinFIRST is released under the Apache License 2.0.
Citation
Paper and citation information will be added when available.
Similar repos
Contributors
15 commits