Annotation of Positive vs Negative User Interactions for Social Sign Prediction
社交关系符号预测中积极与消极用户交互的标注
Biancamaria Bombino, Chiara Boldrini, Andrea Passarella, Marco Conti
AI总结 提出利用大语言模型零样本识别交互中的个人赞扬和攻击作为关系信号,以改进社交关系符号预测。
详情
从在线交互中推断社交关系的符号是社交网络分析中的一个基本挑战。现有方法通常依赖情感分析将单个交互标记为积极或消极,然后聚合这些标签以赋予关系符号。然而,情感分析捕捉的是所讨论内容的效价,而非关系交换本身的性质,这种混淆可能导致系统性误分类。在本文中,我们提出了一种方法,通过利用大语言模型(LLMs)在零样本设置中识别交互层面的关系信号(具体来说,是针对对话者的个人赞扬和个人攻击),作为积极和消极社会关系的更直接指标。我们在三个复杂度递增的提示设计下,评估了涵盖开放权重和专有架构的四种模型(Qwen2.5:7b、Gemma2:9b、GPT-4o、GPT-5.4-mini),并在两个分别包含约298和340个文本的人工标注数据集上进行测试。结果表明,零样本LLMs在没有任务特定训练数据的情况下,在两个任务上都取得了良好的分类性能,为关系标注建立了实用基线。性能因任务而异:攻击检测对提示设计和模型选择具有鲁棒性,而赞扬检测对两者都更敏感,反映了积极关系姿态的更大主观性。这些发现为将基于LLM的关系标注集成到符号预测流程中奠定了基础。
Inferring the sign of social relationships from online interactions is a fundamental challenge in social network analysis. Existing approaches typically rely on sentiment analysis to label individual interactions as positive or negative, then aggregate these labels to assign a sign to the relationship. However, sentiment analysis captures the valence of the content being discussed rather than the nature of the relational exchange itself, a conflation that can lead to systematic misclassification. In this paper, we propose a methodology that addresses this limitation by leveraging Large Language Models (LLMs) in a zero-shot setting to identify interaction-level relational signals (specifically, personal praise and personal attacks directed at the interlocutor) as more direct indicators of positive and negative social ties. We evaluate four models spanning open-weight and proprietary architectures (Qwen2.5:7b, Gemma2:9b, GPT-4o, GPT-5.4-mini) across three prompt designs of increasing complexity, on two human-annotated datasets of approximately 298 and 340 texts respectively. Results show that zero-shot LLMs achieve good classification performance on both tasks without any task-specific training data, establishing a practical baseline for relational annotation. Performance differs across tasks: attack detection is robust to prompt design and model choice, while praise detection is more sensitive to both, reflecting the greater subjectivity of positive relational gestures. These findings lay the groundwork for integrating LLM-based relational annotation into sign prediction pipelines.