document_summary

네이버뉴스를 요약하기위한 챗봇입니다. KoAlpaca-Polyglot-5.8B 모델을 AIhub 문서 요약 데이터셋에서 뉴스에 대한 데이터만을 fine-tuning 하였습니다.

data preprocessing

import json
json_file_path = "Training/article_train_original.json"
save_path = "preprocessing/"
with open(json_file_path,'r') as json_file:
    json_dict = json.load(json_file)
    documents = json_dict['documents']


with open(f"{save_path}/article_train_docs.json",'w') as json_file:
    json.dump(documents, json_file)

Model Path

Requirement

• pytorch > 1.13.x
• transformers
• peft
• accelerate
• datasets

Training

python main.py --dataset_path ./article_train_docs.json

Web Demo