TEAM 6552

Features

• 1. Running with full Data Processing Pipeline from PDF Files:

  • This pipeline includes:
    • PDF Text Extraction: Extracts text from PDFs using pdfPlumber, with OCR-enabled extraction specifically for finance documents.
    • Data Cleaning: Cleans insurance and finance text according to domain-specific rules for improved accuracy.
    • Chunking: Splits large texts into smaller, manageable chunks to enhance retrieval performance.
    • Title Extraction: Uses a LLaMA model to extract document titles for more precise retrieval (pre-built titles can also be used).
    • Semantic Retrieval: Implements a language model-based reranker to identify and return the most relevant file in response to a query.

• 2. Running from Pre-Built Dataset:

  • A pre-built dataset is available, generated by the same data processing pipeline as described in Step 1.

Recommendation: For efficiency, it’s recommended to use the pre-built dataset to save processing time.

Prerequisites

• Python 3.11.8
• Require pytesseract system package
• Required packages listed in requirements.txt, including:
  • pymupdf
  • pytesseract
  • pdf2image
  • pdfplumber
  • torch==2.3.1
  • (Maybe it works on newer versions but mine is 2.3.1 and I have no time to test it out)
  • transformers
  • tqdm

Usage

Command-Line Arguments

Run the script by providing the following arguments:

Running the Script

Again, it is recommended to use -use_pre_built_dataset to save time or it would probably spend extra 30+ minutes for processing data.

Example:

python main.py --question_path ...questions.json --source_path .../reference --output_path ....pred_retrieve.json --use_pre_built_dataset