🌟 This repo contains the code and datasets for the paper "M-LongDoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework" accepted by EMNLP 2025.

🎉 Updates

• [2025-08] Our paper is accepted by EMNLP 2025.
• [2024-11] Check out our paper on arXiv.
• [2024-11] Visit our official project website for more information.

Overview

• M-LongDoc is a challenging benchmark designed to evaluate large multimodal models on super-long document understanding, especially for real-world documents containing interleaved text, figures, and tables.
• Unlike existing document understanding benchmarks that mostly focus on short documents or extractive QA, M-LongDoc contains 851 samples over documents with more than 200 pages on average, requiring models to generate open-ended, in-depth answers rather than simply extracting short spans.
• The benchmark covers diverse real-world domains, including academic papers, financial reports, and product manuals, and evaluates whether models can reason over different evidence types such as text, figures, and tables in long multimodal documents.
• Experiments show that existing models still struggle with multimodal long-document QA, particularly on figure- and table-based questions, and can be distracted by irrelevant retrieved content even under retrieval-augmented generation settings.
• The paper further proposes a retrieval-aware multimodal tuning framework, which trains models to use relevant retrieved pages while ignoring distracting multimodal content, achieving a 4.6% relative improvement in answer correctness over the baseline open-source model.

• Our automated evaluation framework can reliably and scalably assess the correctness of open-ended solutions for multimodal document question answering.

Dataset Overview

The 851 validated test questions are the official benchmark set, filtered through automated verification and expert human validation (pass rate: 80.9%). The remaining 200 questions were excluded during annotation.

Dataset Statistics

Each question is associated with:

• A source document (PDF processed into JSON)
• An evidence page number
• A content category: text (271), figure (283), or table (297)
• A domain: Academic Paper, Financial Report, or Technical Manual

Data Format

Document JSON (data/{train,test}/*.json)

Each document is stored as JSONL (one MultimodalPage per line):

{
  "number": 1,
  "objects": [
    {
      "page": 1,
      "text": "",
      "image_string": "<base64 PNG>",
      "category": "Table",
      "score": 0.95,
      "source": "data/test/NYSE_CI_2023.pdf"
    }
  ],
  "text": "Extracted page text...",
  "image_string": "<base64 PNG of full page>",
  "source": "data/test/NYSE_CI_2023.pdf"
}

• objects: Detected tables and figures (YOLO), each with a cropped sub-image
• text: Plain text extracted via PyMuPDF
• image_string: Base64-encoded PNG of the full page at 150 DPI

Question JSON (data/questions/*.json)

Each question is a JSONL line (MultimodalSample):

{
  "question": "What does the revenue trend indicate...?",
  "answer": "",
  "category": "figures or diagrams or charts",
  "evidence_pages": [12],
  "source": "data/test/NYSE_CI_2023.json",
  "annotator": "gemini-1.5-pro-002",
  "retrieved_pages": [],
  "judgements": []
}

• category: One of texts, figures or diagrams or charts, tables
• evidence_pages: The page where the answer can be found
• source: Path to the processed document JSON
• answer: Gold reference answer (present in train, empty in test for benchmarking)

Annotation Files (data/annotation/)

• valid_questions.json: Set of 851 human-validated test questions
• *_cq.xlsx, *_ma.xlsx, *_mj.xlsx: Annotator-specific quality check sheets
  • Each row checks 4 criteria: content contains category, question requires category, clear/answerable, reasonable difficulty
• score_checking_100_hp.xlsx: 100-sample subset with human correctness scores

Setup

conda create -n docs python=3.10 setuptools=69.5.1 -y
conda activate docs
pip install -r requirements.txt

Data Download

The source PDFs (~4.5 GB total) are hosted externally due to their size. Download and place them in the corresponding directories:

After downloading, place the downloaded folders in data/train and data/test respectively.

Pre-processing Status

Questions and annotations are included. PDFs must be downloaded from external links and document parsing must be run to generate the processed JSONs and page images (~85 GB output).

Step	Status	Details
PDF Download (480 files)	Required	Download from Data Download and place in data/{train,test}/
Document Parsing (text + images + YOLO)	Required	Run Step 2 to generate 480 JSONs + page images (~85 GB)
Question Generation	Done	1,051 test (851 validated) + 10,070 train questions in data/questions/
Human Annotation Validation	Done	851 validated questions in data/annotation/valid_questions.json

Quick Start: Inference & Evaluation

Before running inference, you must download the PDFs and process them into JSON (see Step 2 below). Once processed:

# Set up API keys (required for LLM-based answer generation and judging)
cp .env.example .env
# Edit .env to add your API keys

# Run the end-to-end demo
python demo.py main

# Evaluate retrieval methods (e.g., ColPali)
python evaluation.py test_retriever data/questions/test_academic.json \
  --retriever_name colpali --path outputs/retrieve/test/colpali.json

# Generate answers and judge quality
python evaluation.py generate_answers_and_judgements \
  --data_path outputs/retrieve/test/colpali.json \
  --retriever_name colpali --generator_name azure

Full Data Processing Pipeline (from scratch)

Note: Questions and annotations are already included. Steps 1 and 2 must be run by the user: first download the PDFs from the Data Download links, then process them into JSONs. Step 3 has already been completed.

1. Download PDFs [Required]

Download the train and test PDFs from the Data Download section above and place the downdloaded folders in data/train and data/test. Alternatively, you can re-download them programmatically:

python data_loading.py download_pdfs data/train/metadata.csv data/train
python data_loading.py download_pdfs data/test/metadata.csv data/test

2. Process PDFs into JSON [Required]

Extracts text (PyMuPDF), renders page images (150 DPI), and detects tables/figures (YOLO). This produces ~85 GB of output (480 JSONs + 86K page images) and may take several hours depending on your hardware. A GPU is strongly recommended — YOLOv8x inference on ~75,000 pages on CPU may take days instead of hours.

export HF_ENDPOINT=https://hf-mirror.com  # if HuggingFace is blocked
python data_loading.py process_documents data/train/*.pdf --skip_exist
python data_loading.py process_documents data/test/*.pdf --skip_exist

3. Generate Questions (requires LLM API keys) [Done]

python question_generation.py generate_questions data/test/NYSE*.json \
  --path_out data/questions/test_finance.json --questions_per_doc 6
python question_generation.py generate_questions data/test/24*.json \
  --path_out data/questions/test_academic.json --questions_per_doc 6
python question_generation.py generate_questions data/test/*.json \
  --exclude "24,NYSE" --path_out data/questions/test_product.json --questions_per_doc 6

4. Run Inference Demo

python demo.py main

5. Evaluate Retrieval and QA

# Evaluate retrieval methods
python evaluation.py test_retriever data/questions/test_academic.json \
  --retriever_name colpali --path outputs/retrieve/test/colpali.json

# Generate answers and judge quality
python evaluation.py generate_answers_and_judgements \
  --data_path outputs/retrieve/test/colpali.json \
  --retriever_name colpali --generator_name azure

Repository Structure

├── data_loading.py         # Core data models and PDF processing
├── question_generation.py  # LLM-based question generation with verification
├── retrieval.py            # Page retrieval (BM25, CLIP, BGE-M3, ColPali)
├── evaluation.py           # Answer generation and LLM-as-judge scoring
├── modeling.py             # Multimodal LLM wrappers (30+ models)
├── analysis.py             # Statistical analysis and visualization
├── demo.py                 # End-to-end inference demo
├── download_data.py        # Parallel PDF downloader with retry logic
├── retry_downloads.py      # Retry failed downloads with curl + archive URL fix
├── process_light.py        # Lightweight PDF processing without YOLO
├── prepare_release.py      # Dataset assembly and validation
├── run_pipeline.sh         # One-command full pipeline script
├── detection.py            # YOLO-based document layout detection
├── onevision.py            # LLaVA-OneVision model implementation
├── parsing.py              # Document parsing (page images to markdown)
├── crawler.py              # Web crawler for manualslib.com
├── custom_judge.py         # Custom LLM judge training data creation
├── reading.py              # PDF image extraction utility
├── training.py             # PaliGemma fine-tuning trainer
├── data/
│   ├── train/              # 300 training PDFs + metadata.csv (JSONs generated via Step 2)
│   ├── test/               # 180 test PDFs + metadata.csv (JSONs generated via Step 2)
│   ├── questions/          # QA pairs (JSONL format, 9 files)
│   ├── annotation/         # Human validation spreadsheets + valid_questions.json
│   ├── crawl/              # Crawled brand/manual listings from manualslib.com
│   ├── detect_train/       # YOLO object detection training data (images + XML)
│   ├── mllm_demo_data/     # Demo multimodal conversation data
│   └── demo/               # Demo documents
└── scripts/                # Evaluation shell scripts (5 files)

Key Results (from paper)

Retrieval MRR (851 validated test questions):

QA Correctness (1-5, 851 validated test questions):

Model identifiers: GPT-4o (gpt-4o-2024-05-13), Claude 3.5 Sonnet (claude-3-5-sonnet-20240620), Gemini 1.5 Pro (gemini-1.5-pro-002), Qwen2-VL-7B fine-tuned with LoRA rank 64 on 10,070 training QA pairs.

Citation

@inproceedings{chia-etal-2025-longdoc,
    title = "{M}-{L}ong{D}oc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework",
    author = "Chia, Yew Ken  and
      Cheng, Liying  and
      Chan, Hou Pong  and
      Song, Maojia  and
      Liu, Chaoqun  and
      Aljunied, Mahani  and
      Poria, Soujanya  and
      Bing, Lidong",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing {(EMNLP)}",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.469/",
    doi = "10.18653/v1/2025.emnlp-main.469",
    pages = "9233--9250",
    ISBN = "979-8-89176-332-6"
}