Preferred Generation Benchmark

pfgen-benchmark is a benchmark designed to evaluate Japanese text generation, specifically for pretrained models. Unlike conventional benchmarks that use templates containing instructions, this benchmark relies solely on numerous examples. By conveying expectations such as the question-answering nature of the task, responses of approximately 100 characters, and outputs resembling formal public documents purely through examples, it minimizes the influence of differences in instructions or templates. Additionally, output evaluation is conducted using n-gram-based methods, enabling quick, cost-effective, and deterministic evaluations, unlike the LLM as a Judge approach.

To enable comparisons across as many models as possible, the leaderboard actively includes a wide range of models. These include openly accessible models, models cited in academic papers, and those announced by companies through press releases. Contributions of model outputs are encouraged, and results can be submitted via pull requests. For detailed instructions on how to contribute, please refer to the "How to Contribute" section.

See more details: arXiv:2502.09316

pfgen-benchmark は事前学習モデル向けに設計された日本語の生成文を評価するベンチマークです。通常のベンチマークでは指示文を含むテンプレートを使いますが、このベンチマークでは多数の例示のみを行います。質問応答タスクであることや、約100字の回答、公用文に近い出力を期待していることを例示のみで伝えることで、指示文やテンプレートの差異による影響を小さくしています。また、出力文の評価は n-gram を用いた方法を用いており、LLM as a Judge の手法と異なり、短時間、低コストでかつ決定的な評価を可能にしています。

詳しくはこちら： Jxiv preprint

できる限り多くのモデルを同じ軸で比較できるように、リーダーボードには積極的に多くのモデル掲載しています。オープンにアクセス可能なモデル、論文で言及されているモデル、企業がプレスリリースを出しているモデルなど、比較の価値があると思われるモデルについては、是非プルリクエストで出力を追加してください。追加方法については「How to contribute」を参照ください。

License of LLM Output

The license for parts of this repository, except for LLM-generated outputs, is Apache License Version 2.0. The license for LLM-generated outputs depends on the license of each model.

How to Evaluate a Model

You can evaluate the model using either run-hf.py (which uses transformers) or run-vllm.py (which uses vLLM). For detailed parameters, refer to --help. The --num-trials parameter, which determines the number of patterns for which the model will generate answers, should be decided considering the trade-off between execution time and required accuracy.

For pretrained models:

# Run a model using Huggingface library or vLLM.
python ./run-hf.py --model=llm-jp/llm-jp-3-150m --num-trials=5

# Evaluate output and update leaderboard.
make

For instruction models:

# Run a model using Huggingface library or vLLM with three templates.
python ./run-hf.py --model=llm-jp/llm-jp-3-150m-instruct3 --num-trials=5
python ./run-hf.py --model=llm-jp/llm-jp-3-150m-instruct3 --num-trials=5 --mode=qa
python ./run-hf.py --model=llm-jp/llm-jp-3-150m-instruct3 --num-trials=5 --mode=chat

# Evaluate output and update leaderboard.
make

Command-line Arguments

• --model={{model name}} ... The model name. (Required)
• --path={{path to model directory}} ... The path to a local model directory. (Default: None)
• --num-trials={{number of trials}} ... The number of trials. (Default: 10)
• --mode={{mode}} ... Must be one of completion, qa, and chat. (Default: completion)
  • qa and chat can be used only when the model has a chat template.
  • The instruction message will be included in a user message for qa and in a system message for chat.

How to Contribute

Follow the instructions in the "How to Evaluate a Model" section to run the evaluation. This process will generate config.json and trials.jsonl.xz files under the result directory. Please create a pull request containing only these two files.

To ensure more accurate ranking among models, the number of executions (--num-trials) should be as many as possible, within the limit of 100 trials.

Leaderboard

🟢 ... completion mode, 💬 ... qa/chat mode.

FAQ

What is the difference between the modes?

pfgen-bench provides three types of templates: completion, qa, and chat.

• completion: No instruction is provided. It consists solely of question-answer pairs.
• qa: An instruction is included at the beginning of the user message.
• chat: An instruction is placed in a system message.

Should we control the temperature?

pfgen-bench recommends setting the temperature to 1.0.

Some tasks (e.g., generating dice rolls) require a temperature of 1.0, and setting a lower temperature often leads to unnatural repetition.

Citation

If you use this repository, please cite the following paper:

@preprint{Imos2024-pre-pfgen,
  title={{pfgen-bench: 日本語事前学習モデルのための文章生成性能評価ベンチマーク}},
  author={今城, 健太郎 and 平野, 正徳 and 鈴木, 脩司 and 三上, 裕明},
  doi={10.51094/jxiv.1008},
  year={2024}
}
@preprint{Imos2025-judge-free,
  title={{A Judge-free LLM Open-ended Generation Benchmark Based on the Distributional Hypothesis}},
  author={Kentaro Imajo and Masanori Hirano and Shuji Suzuki and Hiroaki Mikami},
  year={2025},
  eprint={2502.09316},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2502.09316},
  doi={10.48550/arXiv.2502.09316}
}

Or cite directory this repository:

@misc{imajo2024-pfgen
    title={{Preferred Generation Benchmark}},
    author={Kentaro Imajo and Masanori Hirano and Shuji Suzuki and Hiroaki Mikami},
    year={2024},
    url = {https://github.com/pfnet-research/pfgen-bench}
}