Dataset Card for FinBen-FiNER-ORD

Table of Contents

• Table of Contents
• Dataset Description
  • Dataset Summary
  • Supported Tasks and Leaderboards
  • Languages
• Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
• Dataset Creation
  • Curation Rationale
  • Source Data
  • Annotations
  • Personal and Sensitive Information
• Considerations for Using the Data
  • Social Impact of Dataset
  • Discussion of Biases
  • Other Known Limitations
• Additional Information
  • Dataset Curators
  • Licensing Information
  • Citation Information
  • Contributions

Dataset Description

• Homepage: https://huggingface.co/datasets/TheFinAI/finben-finer-ord
• Repository: https://huggingface.co/datasets/TheFinAI/finben-finer-ord/edit/main/README.md
• Paper: FinBen: A Holistic Financial Benchmark for Large Language Models
• Leaderboard: https://huggingface.co/spaces/finosfoundation/Open-Financial-LLM-Leaderboard

Dataset Summary

FinBen-FiNER-ORD is a financial named entity recognition (NER) dataset adapted from FiNER-ORD (Shah et al., 2023b). The dataset is designed for training and evaluating large language models (LLMs) on financial text entity recognition tasks. The dataset includes necessary label columns and instructions to enhance its usability for LLM-based training and evaluation.

Supported Tasks and Leaderboards

• Task: Named Entity Recognition (NER)
• Evaluation Metric: Entity F1 Score
• Test Size: 1080 instances

Languages

• English

Dataset Structure

Data Instances

Each instance consists of a list of tokens along with their corresponding entity labels. The annotation follows the BIO tagging format:

• B-PER, B-LOC, B-ORG: Indicates the beginning of an entity (Person, Location, Organization).
• I-PER, I-LOC, I-ORG: Indicates the continuation of an entity.
• O: Indicates a token that does not belong to any named entity category.

Data Fields

• id: A unique identifier for each data instance.
• query: The input text that the model processes.
• answer: The expected response or annotation.
• label: The sequence of labels for each token.
• token: The tokenized version of the query text.

Data Splits

The dataset is split into:

• Test: 1080 instances

Dataset Creation

Curation Rationale

The dataset is adapted from FiNER-ORD (Shah et al., 2023b) to improve its suitability for LLM-based NER tasks by adding instruction and label columns for better training and evaluation.

Source Data

Initial Data Collection and Normalization

The dataset originates from financial documents and articles containing named entities relevant to financial contexts.

Who are the source language producers?

Financial analysts, researchers, and automated data extraction systems.

Annotations

Annotation Process

Annotations follow the BIO tagging scheme, where entities are labeled manually and reviewed for accuracy.

Who are the annotators?

Trained annotators with expertise in financial document analysis.

Personal and Sensitive Information

No personally identifiable information (PII) is included.

Considerations for Using the Data

Social Impact of Dataset

This dataset enhances financial NLP capabilities, allowing more accurate extraction of named entities in financial texts.

Discussion of Biases

Potential biases may exist due to:

• Overrepresentation of specific financial sectors.
• Linguistic biases in the original dataset.

Other Known Limitations

• May require domain-specific fine-tuning.
• Lacks multilingual support.

Additional Information

Dataset Curators

• The Fin AI Community

Licensing Information

• License: CC BY-NC 4.0

Citation Information

@article{shah2023finer,
  title={FiNER: Financial Named Entity Recognition Dataset and Weak-Supervision Model},
  author={Shah, Agam and Vithani, Ruchit and Gullapalli, Abhinav and Chava, Sudheer},
  journal={arXiv preprint arXiv:2302.11157},
  year={2023}
}

Adapted Version (FinBen-FiNER-ORD):

@article{xie2024finben,
  title={FinBen: A Holistic Financial Benchmark for Large Language Models},
  author={Xie, Qianqian and others},
  journal={arXiv preprint arXiv:2402.12659},
  year={2024}
}