LogBench

LogBench is a benchmark for evaluating logging statement generation.

Logging statements are imperative in modern software. They serve important role in reflecting developer's intention, recording system behavior, and guiding failure diagnosis procedure. LogBench provides a benchmark and toolkit, allowing you to measure your own models and conveniently compare them with existing baseline models.

If you find our paper benefit your research, please kindly cite our following paper:

• Yichen Li, Yintong Huo, Zhihan Jiang, Renyi Zhong, Pinjia He, Yuxin Su, Lionel C. Briand, and Michael R. Lyu. Exploring the Effectiveness of LLMs in Automated Logging Generation: An Empirical Study, IEEE Transactions on Software Engineering(TSE), 2024.

Study overview

The study is fully described in this paper. LogBench comprises two subsets for evaluating the model's effectiveness and generalizability, respectively:

1. Effectiveness: LogBench-O contains a collection of high-quality logging statements and their associated code contexts.
2. Generalizability: LogBench-T is an unseen code dataset, after semantically-equivalent code transformation from LogBench-O.

Additionally, LogBench offers various variants to support different settings in logging statement generation, including:

• Method-level
• File-level
• Comment-included
• Comment-free

Repository organization

We currently provide part of the code in the folder /src. We will release the full source code after the paper has been accepted.

• LogBench-O: The /LogBench-O folder contains the files for LogBench-O.
• LogBench-T: The /LogBench-T folder contains the files for LogBench-T.
• Cases: Please refer to the cases folder for the generated cases.

├── LICENSE
├── LogBench-O
│   ├── LogBench-O_prefix_1point.zip
│   ├── LogBench-O_prefix_1point_file_level.zip
│   └── LogBench-O_prefix_1point_wo_comments.zip
├── LogBench-T
│   ├── LogBench-T_prefix_1point.zip
│   └── LogBench-T_prefix_1point_file_level.zip
├── README.md
├── build
│   └── code-transformer.jar
├── cases
│   └── generated_cases.csv
├── img
│   ├── overview.pdf
│   └── overview.png
└── src
    ├── Baselines
    │   ├── DeepLV
    │   ├── WhichVar
    │   ├── LogenText-Plus
    │   ├── StarCoder
    │   └── Lance
    │   └── InCoder
    │   └── ...
    ├── CodeTransformer
    │   └── README.md
    └── DataCollector
        ├── ...

Study subjects

For each baseline utilized, we kindly request that please ensure to cite the relevant paper while using the code.

Download original crawling logging dataset

For further logging-related research, as GitHub does not hold large datasets, you can download the whole collected logging dataset Fullsize at here (zip: 252M; unzip: 786M).

Code transformation tool

The folder /build contains the built tranformation tool. It will conduct the code tranformation automatically with its eight code transformers.

• To conduct the code transformation in batch:

java -jar code-transformer.jar -f ./javafiles/