The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction
幽灵标注者:通过共形预测探索内容审核中人类标签变异的框架
机构 * Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University(生物与社会技术系统建模实验室,东北大学) ; Heriot-Watt University(赫瑞-沃顿大学) ; aequa-tech ; Università del Piemonte Orientale(皮埃蒙特东方大学) ; Università degli Studi di Torino(托斯卡纳大学)
AI总结 提出结合共形预测与协同过滤式标注者表征的框架,通过幽灵预测度量和幽灵标注者表征量化模型预测与所有人类标注的分歧,并发现模型在标注者分歧时不确定性增加,但大型模型对无人类对齐文本更自信,且存在结构性人口统计偏差。
Comments The publishing of this preprint is contextual with the ACL ARR cycle system. After an encouraging review in January we revised and submit the paper on Arxiv. However, a new batch of reviewers raised additional issues that will lead to significant revisions of the experimental setting. Therefore, we decide to withdraw the manuscript