1. Introduction of this repository

Official Repository of "Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models". NeurIPS 2024

• Paper Link: (https://arxiv.org/abs/2409.19667/)
• GitHub Repository: (https://github.com/BUPT-GAMMA/ProGraph)

2. Pipelines and Experimental Results

The pipeline of ProGraph benchmark construction

The pipeline of LLM4Graph dataset construction and corresponding model enhancement.

The pass rate (left) and accuracy (right) of open-source models with instruction tuning.

Compilation error statistics for open source models.

Performance (%) of open-source models regarding different question types.

3. How to Use

Here give some examples of how to use our models.

Chat Model Inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, StoppingCriteria, StoppingCriteriaList
from peft import PeftModel

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model_name_or_path = '../models/deepseek-ai/deepseek-coder-7b-instruct-v1.5'
# You can use Llama-3-8B by 'meta-llama/Meta-Llama-3-8B-Instruct'.
# You can also use your local path.
peft_model_path = 'https://huggingface.co/lixin4sky/ProGraph/tree/main/deepseek-code-only' 
# Or other models in the repository.

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
model = AutoModelForCausalLM.from_pretrained(model_name_or_path).to(device)
peft_model = PeftModel.from_pretrained(model, peft_model_path).to(device)

input_text = '' # the question.

message = [
    {"role": "user", "content": f"{input_text}"},
]

input_ids = tokenizer.apply_chat_template(conversation=message,
                                        tokenize=True,
                                        add_generation_prompt=False,
                                        return_tensors='pt')

input_ids = input_ids.to("cuda:0" if torch.cuda.is_available() else "cpu")
with torch.inference_mode():
    output_ids = model.generate(input_ids=input_ids[:, :-3], max_new_tokens=4096, do_sample=False, pad_token_id=2)
response = tokenizer.batch_decode(output_ids.detach().cpu().numpy(), skip_special_tokens = True)

print(response)

You can find more tutorials in our GitHub repository: (https://github.com/BUPT-GAMMA/ProGraph)

4. Next Level

• GraphTeam: (https://arxiv.org/abs/2410.18032)
• Github Repository: (https://github.com/BUPT-GAMMA/GraphTeam)