Skip to content

无法复现论文中的结果 Unable to reproduce the results in the paper #23

@ohgodaaaa

Description

@ohgodaaaa

我尝试复现中论文中的结果,现在我是直接导入hugging face上的chatgpt-detector-roberta作为model和tokenizer,根据页面上的描述这是由mixed数据集训练的,在论文中对raw-full的F1 score应该为99.44,但我没办法得到这个数据,我使用的数据集是在hc3中readme中的谷歌网盘下载的,以下是我得到的结果
{'0': {'precision': 0.9994103425909546, 'recall': 0.9951852504256943, 'f1-score': 0.9972933215651661, 'support': 17031.0}, '1': {'precision': 0.9898640296662546, 'recall': 0.9987528061860813, 'f1-score': 0.994288552272163, 'support': 8018.0}, 'accuracy': 0.9963271986905665, 'macro avg': {'precision': 0.9946371861286046, 'recall': 0.9969690283058878, 'f1-score': 0.9957909369186646, 'support': 25049.0}, 'weighted avg': {'precision': 0.9963546382901745, 'recall': 0.9963271986905665, 'f1-score': 0.9963315170942771, 'support': 25049.0}}

I am trying to reproduce the results of a paper. Currently, I am using the 'chatgpt-detector-roberta' model and tokenizer directly imported from Hugging Face. According to the information on their website, this model was trained on a mixed dataset. In the paper, the F1 score for 'raw-full' should be 99.44. However, I am unable to achieve this result, The dataset used was downloaded from the Google Drive link provided in the README of hc3, and here are the results I obtained.
{'0': {'precision': 0.9994103425909546, 'recall': 0.9951852504256943, 'f1-score': 0.9972933215651661, 'support': 17031.0}, '1': {'precision': 0.9898640296662546, 'recall': 0.9987528061860813, 'f1-score': 0.994288552272163, 'support': 8018.0}, 'accuracy': 0.9963271986905665, 'macro avg': {'precision': 0.9946371861286046, 'recall': 0.9969690283058878, 'f1-score': 0.9957909369186646, 'support': 25049.0}, 'weighted avg': {'precision': 0.9963546382901745, 'recall': 0.9963271986905665, 'f1-score': 0.9963315170942771, 'support': 25049.0}}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions