ST-Raptor

✨ Project Introduction

ST-Raptor is a tool for answering questions over tables with diverse semi-structured layouts. It takes only an Excel-formatted table and a natural language question as input, and produces precise answers.

Unlike many existing approaches, ST-Raptor requires no additional fine-tuning. It combines a vision-language model (VLM) with a tree-construction algorithm (HO-Tree) and flexibly integrates with different LLMs. ST-Raptor employs a two-stage validation mechanism to ensure reliable results.

❓ What Tables Can ST-Raptor Handle?

📣 Updates

• Main functionss

  • Support both local deployment or API calls for LLM, VLM, and Embedding models.
  • Support diverse input formats: HTML, CSV, MARKDOWN, ...
  • Support Image input.
  • Expand the table extraction module to support table types beyond problem definition.

• Benchmark

  • Update both english and chinese version of SSTQA Benchmark.
  • The SSTQAv2 is on the way!!!

• Visualization

  • Support web visualization platform (FastAPI + HTML).
  • Support hyper-parameter settings (WIP).
  • Support the visualization of HO-Tree structure.
  • Support the HO-Tree manual correction function.

Semi-structures tables like personal information form, academic tables, financial tables... from Excel, websites (HTML), Markdown, csv files...

💻 SSTQA Benchmark

The 102 tables and 764 questions in SSTQA are carefully curated from over 2031 real-world tables by considering $(i)$ tables featuring semi-structured formats, such as nested cells, multi-row/column headers, irregular layouts and $(ii)$ coverage across 19 representative real scenarios.

We list out 10 representative real scenarios as below:

Human Resources, Corporate Management, Financial Management, Marketing, Warehouse Management, Academic, Schedule Management, Application Forms, Education-related, and Sales Management.

You can find the SSTQA benchmark in ./data directory: SSTQA-en SSTQA-ch

📊 Performance

The following table demonstrates the answering accuracy (%) and ROUGE-L score of different methods over our collected SSTQA benchmark and other two benchmarks.

Note that the required question answering is highly dependent on both the semi-structured table complexity and the question complexity.

Baselines

NL2SQL methods: OpenSearch-SQL

Fine-tuning based methods: TableLLaMA TableLLM

Agent based methods: ReAcTable TAT-LLM

Vision Language Model based methods: TableLLaVA mPLUG-DocOwl1.5

Foudation Models: GPT-4o DeepSeekV3

Experiment Results

🕹 Quick Start

1. Clone Repository

git clone git@github.com:weAIDB/ST-Raptor.git
cd ST-Raptor

Please download this repository.

2. Environment & Benchmark & Model

Environment.

1. Use the following command to install the conda environment.

# create virtual environment
conda create -n straptor python=3.10
conda activate straptor
# install required packages
pip install -r requirements.txt

2. Install the HTML rendering plugin wkhtmltox and font package.

wget https://github.com/wkhtmltopdf/packaging/releases/download/0.12.6.1-2/wkhtmltox_0.12.6.1-2.jammy_amd64.deb
sudo apt-get install -f ./wkhtmltox_0.12.6.1-2.jammy_amd64.deb
sudo apt-get install -y fonts-noto-cjk fonts-wqy-microhei

Benchmark

1. You can find the SSTQA benchmark in ./data directory: SSTQA-en SSTQA-ch
   • You can also find the SSTQA benchmark on hugging face SSTQA Huggingface
2. Change the settings in ./main.py

# You need to change this
input_jsonl = 'PATH_TO_YOUR_INPUT_JSONL'      # The QA pairs
table_dir = 'PATH_TO_YOUR_TABLE_DIR'          # The corresponding tables
pkl_dir = 'PATH_TO_YOUR_PKL_DIR'              # The directory to store HO-Tree object files 
output_jsonl = 'PATH_TO_YOUR_OUTPUT_JSONL'    # The QA results
log_dir = 'PATH_TO_YOUR_LOG_DIR'              # The directory to store log files

The Q&A data is stored in a JSONL format file, and the format of each record is as follows.

{
	"id": "XXX", 
	"table_id": "XXX", 
	"query": "XXX", 
	"label": "XXX"    // Optional when inference
}

Model Configuration. The model configuration in our paper includes Deepseek-V3 (LLM API) + InternVL2.5 26B (VLM) + Multilingual-E5-Large-Instruct (Embedding Model). This configuration requires a total of approximately 160GB of GPU memory. You can replace the model according to the hardware situation or change it to use APIs.

You need to set model configuration in ./utils/constnts.py

"""Change this for requesting LLM"""
LLM_API_URL = "YOUR_LLM_API_URL"
LLM_API_KEY = "YOUR_LLM_API_KEY"
LLM_MODEL_TYPE = "YOUR_LLM_MODEL_TYPE" 

"""Change this for requesting VLM"""
VLM_API_URL = "YOUR_VLM_API_URL"
VLM_API_KEY = "YOUR_VLM_API_KEY"
VLM_MODEL_TYPE = "YOUR_VLM_MODEL_TYPE"

"""Change this for requesting Embedding Model"""
EMBEDDING_TYPE = "api" # api / local

## If EMBEDDING_TYPE is local
EMBEDDING_MODE_PATH = "YOUR_PATH_TO_MULTILINGULE_E5"

## If EMBEDDING_TYPE is api
EMBEDDING_API_URL = "YOUR_EMBEDDING_API_URL"
EMBEDDING_API_KEY = "YOUR_EMBEDDING_API_KEY"
EMBEDDING_MODEL_TYPE = "YOUR_EMBEDDING_MODEL_TYPE"

If you want to use other format of APIs, please revise the code in ./utils/api_utils.py

Use local deployment VLM and Embedding Model with LLM API as an example.

First to Download InternVL2.5 and Download Multilingual-E5

1. Install the vllm package.

pip install vllm

2. Denote the GPU and deploy the VLM.

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m vllm.entrypoints.openai.api_server \
--model=PATH_TO_INTERNVL \
--served-model-name internvl
--port 8138 \
--trust-remote-code \
--max-num-batched-tokens 8192 \
--seed 42 \
--tensor-parallel-size 4

3. Set API configs in ./utils/constnts.py

"""Change this for requesting LLM"""
LLM_API_URL = "YOUR_LLM_API_URL"        # [Change This]
LLM_API_KEY = "YOUR_LLM_API_KEY"        # [Change This]
LLM_MODEL_TYPE = "YOUR_LLM_MODEL_TYPE"  # [Change This]

"""Change this for requesting VLM"""
VLM_API_URL = "http://localhost:8000/v1/"
VLM_API_KEY = "Empty"
VLM_MODEL_TYPE = "internvl"

"""Change this for requesting Embedding Model"""
EMBEDDING_TYPE = "local" # api / local

## If EMBEDDING_TYPE is local
EMBEDDING_MODE_PATH = "YOUR_PATH_TO_MULTILINGULE_E5"  # [Change This]

Question Answering !

If you have completed all the above settings, use the following command to start execution.

python ./main.py

Web Frontend

Use the unified web entry to start the frontend:

python ./start_web.py

Open in browser: http://localhost:7860/

On this interface, you can upload a table, view the HO-Tree structure, and ask our model questions about it!

💡 Semi-Structured Table QA Examples

Note: The "" cell in the table indicate that the baseline fails to generate an answer of that question.

The full result please refer to the file: baseline_output.jsonl

📍 Citation

If you like this project, please cite our paper link:

@article{tang2026straptor,
  author       = {Zirui Tang and Boyu Niu and Xuanhe Zhou and Boxiu Li and Wei Zhou and Jiannan Wang and Guoliang Li and Xinyi Zhang and Fan Wu},
  title        = {ST-Raptor: LLM-Powered Semi-Structured Table Question Answering},
  journal      = {Proc. {ACM} Manag. Data},
  year         = {2026}
}

👨‍🏫 Join us !

ST-Raptor@Complex Semi-Structured Table Analysis Community (Please contact the following WeChat account)

📝 License

This project is licensed under the MIT License - see the LICENSE file for details