🌐 Website ｜ 🤗 Hugging Face • 🤖️ ModelScope
English | 中文

CS-Eval is a comprehensive evaluation toolkit for fundamental cybersecurity models or large language models' cybersecurity ability, encompassing 11 major cybersecurity categories, 42 subdomains, featuring 4,369 assessment items across multiple-choice, true/false, and knowledge extraction questions. It delivers a balanced mix of knowledge-oriented and practice-focused evaluation tasks. The platform empowers users to conduct self-assessments and offers leaderboards across various subdomains, fostering competitive benchmarking and performance insights.

News

• [2024.05.31] CS-Eval has been released, and users are now able to submit evaluations on the website independently. 🎉🎉🎉
• [2024.03.29] The CS-Eval dataset has been jointly constructed and completed. ✅✅✅

Table of Contents

• Leaderboard

• Data

• CS-Eval Usage Steps

• How to Submit

• Licenses

• Citation

• Disclaimer

Leaderboard

Here are the accuracies obtained when evaluating industry-leading models upon our initial release. Please refer to our official platform's Leaderboard for the latest community rankings and also pay attention to rankings within different subdomains. Note that subtle differences may exist in the results for the same model due to variations in its generation config.

Data

Download

• Method 1: Download or load on Hugging Face:

  • Download data directly:

    wget https://huggingface.co/datasets/cseval/cs-eval/resolve/main/cs-eval-questions.zip
  

  • Load dataset using Hugging Face datasets:

  from datasets import load_dataset
  dataset=load_dataset(r"cseval/cs-eval")
  
  print(dataset['{test}'][0])
  

• Method 2: Download on ModelScope:

  • Download data directly:

  git clone https://www.modelscope.cn/datasets/cseval/cs-eval.git
  

  • Load dataset using Model Scope SDK:

  from modelscope.msdatasets import MsDataset
  ds =  MsDataset.load('cseval/cs-eval')
  

CS-Eval Usage Steps

1. Download the CS-Eval evaluation data from either Hugging Face or ModelScope.
2. Adapt the model inference format.
3. Conduct model inference on the CS-Eval evaluation dataset.
4. Format the inference results according to the specified submission guidelines.
5. Submit the model results to the CS-Eval platform.
6. Obtain the evaluation results on the platform.
7. Decide whether to opt into public leaderboard.

How to Submit

You need to convert the organized model inference results into a JSON file encoded in UTF-8 and format it according to the following example.

## Example
[
    {
      "question_id": "1",
      "answer": "A"
    },
    {
      "question_id": "123",
      "answer": "对"
    },
    {
      "question_id": "1234",
      "answer": "是否涉及漏洞：是\n漏洞号：CVE-2024-22891\n影响的产品及版本：Nteract v.0.28.0"
    }
]

In this example, question_id refers to the question number, and answer contains the processed model output.

Please note:

• For multiple-choice questions, the correct answer option(s) can typically be extracted directly from the model's generated result using regular expressions.
• In the case of multiple-answer questions, regular expressions can similarly be used to extract multiple correct answer options from the model's output.
• For true/false questions, if the question instructions require a specific answer format, the judgment result is usually taken from the beginning or end of the model's output.
• For knowledge extraction tasks, where the question specifies a particular response format, use the raw text output from the model inference directly.

When you regularize multiple-choice questions, you can quickly locate multiple-choice questions by filtering the following keywords in the dataset prompt.

"单选题："
"多选题："
"Single-choice question:"

Licenses

This project adheres to the MIT License.

The CS-Eval dataset adheres to the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Citation

If you utilize our dataset in your research or technical reports, please ensure proper citation.

@inproceedings{Yu2024CSEvalAC,
  title={CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity},
  author={Zhengmin Yu and Jiutian Zeng and Siyi Chen and Wenhan Xu and Dandan Xu and Xiangyu Liu and Zonghao Ying and Nan Wang and Yuan Zhang and Min Yang},
  year={2024},
  url={https://api.semanticscholar.org/CorpusID:274234403}
}

Disclaimer

Our platform and its affiliated entities consistently adhere to principles of legality, compliance, positivity, and health, dedicated to promoting the research and application of large language models in the field of cybersecurity to enhance protective capabilities. To prevent any potential misunderstanding of the content on this platform, we hereby issue the following statement:

1. Legitimate Purposes: All information, resources, tools, and services we provide are intended to facilitate lawful and beneficial activities in the cybersecurity sector, including scientific research, technological innovation, risk assessment, and the formulation of defensive strategies using large models. We firmly oppose any utilization of large language models for illegal activities, infringement, or compromising cybersecurity.
2. Non-Inducement: This platform strictly prohibits any content that incites or encourages others to engage in cyberattacks, intrusions, disruptions, or unauthorized data acquisition. We emphasize that all content related to large language model cybersecurity evaluation sets aims to advance industry development, provide cybersecurity system evaluations, and facilitate technical exchanges, devoid of any elements that induce, encourage, or imply malicious attacks.
3. Non-Malicious Attack Education: We explicitly state that our provided content does not involve teaching, demonstrating, or guiding techniques for malicious cyberattacks. All discussions involving offensive actions are strictly confined within the realms of legitimate cybersecurity drills, vulnerability research, and risk assessments, aimed at enhancing defensive capabilities rather than offensive purposes.
4. User Responsibility: Users must strictly abide by relevant laws and regulations when using our platform's services and are prohibited from utilizing platform resources for any illegal, infringing, or cybersecurity-compromising activities. In the event of user violations, our platform reserves the right to take measures including but not limited to warnings, service suspension, account banning, and the pursuit of legal responsibility.
5. Disclaimer: While we strive to ensure the accuracy, legality, and appropriateness of our platform's content, users are solely responsible for any direct or indirect losses incurred from their own actions during usage, including legal disputes, property loss, data breaches, reputational harm, etc. Neither our platform nor its affiliated entities assume any legal liability. Users should assess and bear all risks associated with using platform resources themselves.

We sincerely call upon all users to jointly maintain a sound order in the cybersecurity domain, employing large model technologies and related resources legally, rationally, and responsibly. The final interpretation right of this disclaimer resides with our platform, and changes, if any, will not be separately notified.