tum-nlp/IDMGSP

Dataset

Dataset Card for A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era

51 commits

2 linked in READMEs

updated Apr 8, 2024

See the code

README

Dataset Card for A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era

Dataset Description

Repository: https://github.com/qwenzo/-IDMGSP
Paper: https://aclanthology.org/2023.trustnlp-1.17/

Dataset Summary

A benchmark for detecting machine-generated scientific papers based on their abstract, introduction and conclusion sections.

Supported Tasks and Leaderboards

current benchmark results in terms of accuracy:

Model	Train Dataset	TEST	OOD-GPT3	OOD-REAL	TECG	TEST-CC
LR-1gram (tf-idf) (our)	TRAIN	95.3%	4.0%	94.6%	96.1%	7.8%
LR-1gram (tf-idf) (our)	TRAIN+GPT3	94.6%	86.5%	86.2%	97.8%	13.7%
LR-1gram (tf-idf) (our)	TRAIN-CG	86.6%	0.8%	97.8%	32.6%	1.2%
RF-1gram (tf-idf) (our)	TRAIN	94.8%	24.7%	87.3%	100.0%	8.1%
RF-1gram (tf-idf) (our)	TRAIN+GPT3	91.7%	95.0%	69.3%	100.0%	15.1%
RF-1gram (tf-idf) (our)	TRAIN-CG	97.6%	7.0%	95.0%	57.0%	1.7%
IDMGSP-Galactica-TRAIN (our)	TRAIN	98.4%	25.9%	95.5%	84.0%	6.8%
IDMGSP-Galactica-TRAIN_GPT3 (our)	TRAIN+GPT3	98.5%	71.2%	95.1%	84.0%	12.0%
IDMGSP-Galactica-TRAIN-CG (our)	TRAIN-CG	96.4%	12.4%	97.6%	61.3%	2.4%
IDMGSP-RoBERTa-TRAIN-ABSTRACT + IDMGSP-RoBERTa-TRAIN-INTRODUCTION + IDMGSP-RoBERTa-TRAIN-CONCLUSION (our)	TRAIN	72.3%	55.5%	50.0%	100.0%	63.5%
IDMGSP-RoBERTa-TRAIN_GPT3-ABSTRACT + IDMGSP-RoBERTa-TRAIN_GPT3-INTRODUCTION + IDMGSP-RoBERTa-TRAIN_GPT3-CONCLUSION (our)	TRAIN+GPT3	65.7%	100.0%	29.1%	100.0%	75.0%
IDMGSP-RoBERTa-TRAIN-CG-ABSTRACT + IDMGSP-RoBERTa-TRAIN-CG-INTRODUCTION + IDMGSP-RoBERTa-TRAIN-CG-CONCLUSION (our)	TRAIN-CG	86.0%	2.0%	92.5%	76.5%	9.2%
GPT-3 (our)	TRAIN-SUB	100.0%	25.9%	99.0%	100.0%	N/A
DetectGPT	-	61.5%	0.0%	99.9%	68.7%	N/A
ChatGPT-IO (our)*	-	69.0%	49.0%	89.0%	0.0%	3.0%
LLMFE (our)*	TRAIN+GPT3	80.0%	62.0%	70.0%	90.0%	33.0%

Languages

English

Dataset Structure

Data Instances

Each instance in the dataset corresponds to a row in a CSV file, encompassing the features of a paper, its label, and the paper's source.

Data Fields

classifier_input

name: id description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real". dtype: string
name: year description: year of the publication as given by the arXiv database. dtype: string
name: title description: title of the paper given by the arXiv database. dtype: string
name: abstract description: abstract of the paper given by the arXiv database. dtype: string
name: introduction description: introduction section of the paper. extracted by the PDF parser. dtype: string
name: conclusion description: conclusion section of the paper. extracted by the PDF parser. dtype: string
name: categories description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real". dtype: string
name: src description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica". dtype: string
name: label description: 0 for real/human-written papers and 1 for fake/machine-generated papers. dtype: int64

train+gpt3

name: id description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real". dtype: string
name: year description: year of the publication as given by the arXiv database. dtype: string
name: title description: title of the paper given by the arXiv database. dtype: string
name: abstract description: abstract of the paper given by the arXiv database. dtype: string
name: introduction description: introduction section of the paper. extracted by the PDF parser. dtype: string
name: conclusion description: conclusion section of the paper. extracted by the PDF parser. dtype: string
name: categories description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real". dtype: string
name: src description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica", "gpt3". dtype: string
name: label description: 0 for real/human-written papers and 1 for fake/machine-generated papers. dtype: int64

tecg

name: id description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real". dtype: string
name: year description: year of the publication as given by the arXiv database. dtype: string
name: title description: title of the paper given by the arXiv database. dtype: string
name: abstract description: abstract of the paper given by the arXiv database. dtype: string
name: introduction description: introduction section of the paper. extracted by the PDF parser. dtype: string
name: conclusion description: conclusion section of the paper. extracted by the PDF parser. dtype: string
name: categories description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real". dtype: string
name: src description: indicator of the source of the paper. Always has the value "chatgpt". dtype: string
name: label description: always having the value 1. dtype: int64

train-cg

name: id description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real". dtype: string
name: year description: year of the publication as given by the arXiv database. dtype: string
name: title description: title of the paper given by the arXiv database. dtype: string
name: abstract description: abstract of the paper given by the arXiv database. dtype: string
name: introduction description: introduction section of the paper. extracted by the PDF parser. dtype: string
name: conclusion description: conclusion section of the paper. extracted by the PDF parser. dtype: string
name: categories description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real". dtype: string
name: src description: indicator of the source of the paper. This can have the values "gpt2", "real", "scigen" or "galactica". dtype: string
name: label description: 0 for real/human-written papers and 1 for fake/machine-generated papers. dtype: int64

ood_gpt3

name: title description: title of the paper given by the arXiv database. dtype: string
name: abstract description: abstract of the paper given by the arXiv database. dtype: string
name: introduction description: introduction section of the paper. extracted by the PDF parser. dtype: string
name: conclusion description: conclusion section of the paper. extracted by the PDF parser. dtype: string
name: src description: indicator of the source of the paper. Has the value "gpt3". dtype: string
name: label description: always having the value 1. dtype: int64

ood_real

dtype: string

name: abstract description: abstract of the paper given by the arXiv database. dtype: string
name: introduction description: introduction section of the paper. extracted by the PDF parser. dtype: string
name: conclusion description: conclusion section of the paper. extracted by the PDF parser. dtype: string
name: src description: indicator of the source of the paper. Has the value "ood_real". dtype: string
name: label description: always having the value 0. dtype: int64

test-cc

name: id description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real". dtype: string
name: year description: year of the publication as given by the arXiv database. dtype: string
name: title description: title of the paper given by the arXiv database. dtype: string
name: abstract description: abstract of the paper given by the arXiv database. dtype: string
name: introduction description: introduction section of the paper. extracted by the PDF parser. dtype: string
name: conclusion description: conclusion section of the paper. extracted by the PDF parser. dtype: string
name: categories description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real". dtype: string
name: src description: indicator of the source of the paper. Always has the value "chatgpt-paraphrased". dtype: string
name: paraphrased_sections description: indicator of which sections are paraphrased. Can have the values "introduction", "conclusion", "introduction, conclusion", "abstract, introduction, conclusion". dtype: string
name: label description: 0 for real/human-written papers and 1 for fake/machine-generated papers. Always has the value 1. dtype: int64

Data Splits

Table: Overview of the datasets used to train and evaluate the classifiers.

Dataset	arXiv	ChatGPT	GPT-2	SCIgen	Galactica	GPT-3	ChatGPT (co-created)
Standard train (TRAIN)	8k	2k	2k	2k	2k	-	-
Standard train subset (TRAIN-SUB)	4k	1k	1k	1k	1k	-	-
TRAIN without ChatGPT (TRAIN-CG)	8k	-	2k	2k	2k	-	-
TRAIN plus GPT-3 (TRAIN+GPT3)	8k	2k	2k	2k	2k	1.2k	-
Standard test (TEST)	4k	1k	1k	1k	1k	-	-
Out-of-domain GPT-3 only (OOD-GPT3)	-	-	-	-	-	1k	-
Out-of-domain real (OOD-REAL)	4k (parsing 2)	-	-	-	-	-	-
ChatGPT only (TECG)	-	1k	-	-	-	-	-
Co-created test (TEST-CC)	-	-	-	-	-	-	4k

[More Information Needed]

Dataset Creation

Curation Rationale

[More Information Needed]

Source Data

Initial Data Collection and Normalization

[More Information Needed]

Who are the source language producers?

[More Information Needed]

Annotations

Annotation process

[More Information Needed]

Who are the annotators?

[More Information Needed]

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

[More Information Needed]

Citation Information

[More Information Needed]

Contributions

[More Information Needed]

fake papers

science

scientific paper

scientific text

Similar repos

qwenzo/-IDMGSP

yafuly/MAGE

Machine-generated text detection in the wild (ACL 2024)

232

baixianghuang/survey-authorship

Paper list for the paper "Authorship Attribution in the Era of Large Language Models: Problems,…

junchaoIU/LLM-generated-Text-Detection

A survey and reflection on the latest research breakthroughs in LLM-generated Text detection,…

andreas122001/roberta-mixed-detector

Model

Machine-generated text-detection by fine-tuning of language models

yaful/MAGE

Dataset

🚀 Introduction

avduarte333/arXivTection

Dataset

📄 arXivTection Dataset

ByteDance-BandAI/ReportBench

Dataset

ReportBench — Dataset Card

Linked in READMEs

ICTMCG/Awesome-Machine-Generated-Text

Continuously updated list of related resources for generative LLMs like GPT and their analysis and…

231

README badge

qwenzo/-IDMGSP

README badge

tum-nlp/IDMGSP

Dataset

Dataset Card for A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era

51 commits

2 linked in READMEs

updated Apr 8, 2024

See the code

README

Dataset Card for A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era