AuditWen

An Open-source Large Language Model for Audit

Introduction

The first Chinese open source audit model is based on Qwen-7B-chat and fine-tunes a large amount of supervisory audit related data. The model has a wide range of audit knowledge and intelligent analysis capabilities, aiming to provide comprehensive and effective problem analysis and audit foundation recommendations for the audit field.In the future, we will closely follow the update of qwen version and launch qwen 2.5 version. Stay tuned.

Explain

1. This project is still constantly iterating and updating. Considering the wide coverage of the audit field and the serious shortage of domain dataset resources, we have only released fine-tuning models on the audit laws, regulations, and qualitative question and answer datasets;
2. The data and models provided in this project are for scientific research purposes only and are strictly prohibited from being used for commercial purposes;

Application Scenarios of LLM in Audit Domain

Audit issue summary and Laws Recommendation

The primary task of audit is to identify any potential audit issues within a project and determine which laws and regulations can serve as the audit basis. From this perspective, auditors are seeking LLMs to assist in summarizing audit issues based on audit working papers and recommending suitable laws and regulations as both qualitative and punishment basis. The primary challenge in the application is that an internal auditor may have a divergent qualitative basis for an audit issue compared to a social auditor based on the case description in the audit working paper. For example, an internal auditor may use items from enterprise internal control manual as qualitative basis without any penalty provision, while a social auditor may refer to items in Accounting Law and Criminal Law for punishment. To address this challenge, we propose an audit issue schema that summarizes audit issue from case description and aligns them with the clauses of laws and regulations simultaneously。We hope to bridge a gap between the clause of laws and regulations and the audit issue:

Audit Relevant Question and Answer

The secondary task of LLM used in audit is to answer question related to audit, such as questions list in Table 1. These questions pertain to defining an audit concept, understanding the specifics of a particular clause of a law, determining the methods for investigating and verifying audit issues, and identifying the necessary data to be collected. These diverse questions prompt us to gather relevant audit documents pertaining to audit cases, audit criteria, audit guidelines, and so on. When assessing the quality of answers generated by LLM, it is crucial to minimize the occurrence of hallucination responses and ensure the retrieval of original text based on existing system documents and other relevant content:

Audit assistant

Further derive requirement of LLM applied in audit domain is LLM can act as an intelligent assistant and help auditor to extract specified phrase from audit document, do accounting relevant numerical calculation, generate an outline for an audit report and further fill content based on the given audit working papers. The possible case questions are list in Table 2. Audit assistant usually need to execute fine-grained NLP task step by step, such as information extraction, multi-documents summarization and document generation. Additionally, audit assistants must achieve collaborative work between humans and machines with the guidance of human-provided knowledge:

Modality and Prompts

On the basis of collecting and organizing structured data for various audit tasks, write instruction data. Taking audit procedures as an example, the table below shows some structured data：

Next, construct the corresponding instructions through the following scheme：
Question：在[审计类型]中，[审计事项]的审计程序是什么？
Answer:[审计程序]
（For more types of tasks, refer to the file Raw structured data.xlsx in the directory/corpus/Raw structured data）
Based on the above template, the converted instruction data is as follows:

The table below summarizes the different tasks, template and examples of an instruction data:

Tasks

The overall performance of different LLMs on audit evaluation benchmark, * denotes 5-shot evaluation for the task.

GPT_Q&A

Construct a Q&A session on legal and regulatory content through GPT4.0
The gpt4_law_result.xlsx in the directory/AuditWen/GPT_Q&A/data displays regulatory content and content generated by GPT4.

Quick Start

Directory Structure

---/src/ evaluate related packages
----/finetune/ Qwen's finetune Project -- Instruction fine-tuning
----/PIXIU/ PIXIU Project -- NLP Task Inference and Evaluation
----/qa_eval/ Q&A Task Reasoning Reasoning and Evaluation
-----/quick_interference.py/ Immediate inference code
-----/interence.py/ task-based inference code
-----/evaluation.py/ evaluation code
---/pics/ Related pictures
---/GPT_Q&A/
----/code/ GPT4 generates the code for the Q&A
----/data/ partial data
---/corpus/
----/benchmark datatset/ Test set for various tasks
----/result/ Inference results of various task test sets on various pedestals
----/Raw structured data/ Raw Structured data
----/train/ Training instruction set

Inference Model

To reason with AuditWen, you simply enter a few lines of code, as shown below. Remember to pass the correct model name or path, such as "/model/AuditWen". However, make sure you are using the latest code. (This project also provides quick_interference.py with proxy code located in the directory /AuditWen/src/qa_eval)

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig

# Model names: "/model/AuditWen"
tokenizer = AutoTokenizer.from_pretrained( "/model/AuditWen", trust_remote_code=True)

# use auto mode, automatically select precision based on the device.
model = AutoModelForCausalLM.from_pretrained(
    " /model/AuditWen",
    device_map="auto",
    trust_remote_code=True
).eval()
#You can specify different generation length, top p and other related superparameters
model.generation_config = GenerationConfig.from_pretrained(" /model/AuditWen",, trust_remote_code=True, )  

# 1st dialogue turn
response, history = model.chat(tokenizer, "请问什么是审计范围?", history=None)
print(response)
#审计范围是指审计机构和审计人员在一定的审计目的和审计计划指导下，为完成审计任务所进行的审查所有事项。
#它包括审计对象、审计期间、以及对这些审计对象各个方面所进行的审查程度。

# 2nd dialogue turn
response, history = model.chat(tokenizer, "请问什么是财务报表审计？", history=history)
print(response)
#财务报表审计是指注册会计师对被审计单位编制的财务报表的真实性、合法性、完整性作出审计结论，提出审计意见，形成审计报告的一项审计业务。
#财务报表审计是注册会计师接受委托，按照独立审计准则的要求，对企事业单位的财务报表进行审査，并据此发表审计意见，出具审计报告。所发行的审计报告，为报表使用者做出正确决策提供可靠的参考资料

Fine tuning

Model fine-tuning related content reference https://github.com/QwenLM/Qwen
To prepare the training data, you need to put all the samples into a list and save them to a json file. Each sample is a dictionary, consisting of an id and a list of conversations. Here is a list of simple examples with 1 sample:

[
  {
    "id": "identity_0",
    "conversations": [
      {
        "from": "user",
        "value": "请问审计事项'工程设计审计'的审计类型是什么？"
      },
      {
        "from": "assistant",
        "value": "工程设计审计的审计类型是工程项目竣工决算审计。"
      }
    ]
  }
]

In addition, this project also provides a training set containing 2000 entries located in the directory/AuditWen/corpus for users to refer to.
After the data is ready, you can use the shell script provided in the directory/AuditWen/src/finetune/Qwen main/finetune to run finetuning. Please remember to specify the path to the data file. For detailed usage, please refer to the introduction of the Qwen project.

Evaluation-PIXIU

The NLP tasks, namely NER, RE, and NL tasks, are evaluated based on PIXIU. This project has optimized the evaluation method to ensure the accuracy of the evaluation results.

Environmental preparation

cd PIXIU
pip install -r requirements.txt
cd PIXIU/src/financial-evaluation
pip install -e .[multilingual]

Data path setting

In the init.py file under path \PIXIU\src\tasks, set the name of the task and the corresponding method of data processing，Taking "flare_zh_auditner": flare.AuditNER, "as an example, the method for auditing the data loading and processing of the named entity identification task is AuditNER under flare.py.Next, the parameter DATASET_PATH in the AuditNER method is set to the data set path.

Evaluation

To evaluate a model hosted on the HuggingFace Hub (for instance, AuditWen), use this command:

python eval.py \
    --model
    "hf-causal-vllm"
    --model_args
    "pretrained=/model/AuditWen,tokenizer=/model/AuditWen,trust_remote_code=True"
    --tasks
    "flare_zh_auditner"

Commercial APIs：

export OPENAI_API_SECRET_KEY=YOUR_KEY_HERE
python eval.py \
    --model gpt-4 \
    --tasks "flare_zh_auditner"

Evaluation-QA

Model inference is run inference.py under the qa_eval directory, while content evaluation is run evaluation.py under the qa_eval directory

Inference

Reasoning data in the form of a visible directory/AuditWen corpus/benchmark datatset q&a type task under json files in the directory.

Evaluation

The evaluation method of Q&A tasks is optimized based on the evaluation method of PIXIU Q&A tasks. rouge_chinese is used for evaluation, and the word segmentation "Audit Word segmentation.txt" located in the directory /AuditWen/src/qa_eval/src is loaded for evaluation.Loading through jieba's load_userdict method.The evaluation of bart and bert is also replaced with the Chinese version, and the evaluation effect is somewhat improved compared with PIXIU.
This project provides download links of bart and bert models, users can download to the local, and set the bart path in the corresponding place of the code. Due to the project being deployed Autodl platform, Bert model file location for/root/cache/huggingface/hub/models - Bert - base - Chinese/snapshots/main.
Note: Since the lm_eval and bart_score packages used in the evaluation.py code are all self-contained packages of the project, you need to manually add the package path in the interpreter to ensure the normal operation of the code.

Model download

Qwen-7B-chat：https://huggingface.co/Qwen/Qwen-7B-Chat
ChatGLM3-6B:https://huggingface.co/THUDM/chatglm3-6b
bart-base-chinese:https://huggingface.co/fnlp/bart-base-chinese
bert-base-chinese:https://huggingface.co/google-bert/bert-base-chinese
AuditWen:https://huggingface.co/HooRin/AuditWen

Citation

If you use AuditWen in your work, please cite our paper.

@misc{2024AuditWen,
      title={AuditWen:An Open-source Large Language Model for Audit }, 
      author={Jiajia Huang, Haoran Zhu, Chao Xu, Tianming Zhan, Qianqian Xie, Jimin Huang},
      year={2024},
      booktitle = {China National Conference on Chinese Computational Linguistics},
      pages = {505–521},
      url = {https://aclanthology.org/2024.emnlp-main.28},<
      Cite = {Huang J, Zhu H, Xu C, et al. AuditWen: An Open-Source Large Language Model for Audit[C]//China National Conference on Chinese Computational Linguistics. Singapore: Springer Nature Singapore, 2024: 505-521.}
}

Thanks

This project is based on the secondary development of existing open-source projects. We would like to express our gratitude to the relevant projects and R&D personnel.
1.https://github.com/The-FinAI/PIXIU
2.https://github.com/QwenLM/Qwen

Contact

E-mail: