大模型对齐与安全 - arXivDaily 专题

2509.19199 2025-09-30 cs.CL 57%

Agentic Reinforcement Learning with Implicit Step Rewards

Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Junge Zhang, Jianbin Jiao

机构 * University of Chinese Academy of Sciences（中国科学院大学）； Tongyi Lab（通义实验室）； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

Comments 18 pages, 8 figures

2506.09736 2025-09-30 cs.CV cs.AI 57%

Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation

Yuting Li, Lai Wei, Kaipeng Zheng, Jingyuan Huang, Guilin Li, Bo Wang, Linghe Kong, Lichao Sun, Weiran Huang

机构 * School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）； Shanghai Innovation Institute（上海创新研究院）； Zhongguancun Academy（中关村学院）； Tencent（腾讯）； State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）； Lehigh University（莱斯大学）

专题命中偏好对齐：DPO(abstract)；分类 cs.AI

Comments Technical Report

2505.15963 2025-09-30 cs.CV cs.CL 57%

OViP: Online Vision-Language Preference Learning for VLM Hallucination

Shujun Liu, Siyuan Wang, Zejun Li, Jianxiang Wang, Cheng Zeng, Zhongyu Wei

机构 * Fudan University（复旦大学）； University of Southern California（南加州大学）； ByteDance（字节跳动）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

2509.22624 2025-09-29 cs.CV cs.LG 57%

SPARK: Synergistic Policy And Reward Co-Evolving Framework

Ziyu Liu, Yuhang Zang, Shengyuan Ding, Yuhang Cao, Xiaoyi Dong, Haodong Duan, Dahua Lin, Jiaqi Wang

机构 * Shanghai Jiao Tong University（上海交通大学）； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）； Fudan University（复旦大学）； The Chinese University of Hong Kong（香港中文大学）； Shanghai Innovation Institute（上海创新研究院）

专题命中偏好对齐：RLHF(abstract)；分类 cs.LG

Comments Project:https://github.com/InternLM/Spark

2509.22325 2025-09-29 cs.IR cs.CL 57%

Can Synthetic Query Rewrites Capture User Intent Better than Humans in Retrieval-Augmented Generation?

JiaYing Zheng, HaiNan Zhang, Liang Pang, YongXin Tong, ZhiMing Zheng

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

Comments 10 pages, 6 figures

2506.15220 2025-09-29 cs.CV cs.CL cs.SD 57%

video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models

Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, Chao Zhang

机构 * Tsinghua University（清华大学）； University of Cambridge（剑桥大学）； ByteDance（字节跳动）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

2509.20961 2025-09-26 cs.CV cs.AI 57%

Unlocking Financial Insights: An advanced Multimodal Summarization with Multimodal Output Framework for Financial Advisory Videos

Sarmistha Das, R E Zera Marveen Lyngkhoi, Sriparna Saha, Alka Maurya

机构 * Indian Institute of Technology Patna（印度帕纳杰大学）； CRISIL LTD（CRISIL公司）

专题命中偏好对齐：DPO(abstract)；分类 cs.AI

2509.18661 2025-09-24 cs.IR cs.CL cs.HC 57%

Agentic AutoSurvey: Let LLMs Survey LLMs

Yixin Liu, Yonghui Wu, Denghui Zhang, Lichao Sun

机构 * Lehigh University（莱文大学）； University of Florida（佛罗里达大学）； Stevens Institute of Technology（史蒂文斯理工学院）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL

Comments 29 pages, 7 figures

2509.16965 2025-09-23 cs.CL 57%

Preference Distillation via Value based Reinforcement Learning

Minchan Kwon, Junwon Ko, Kangil Kim, Junmo Kim

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国先进科学技术研究院）； Gwangju Institute of Science and Technology (GIST)（全州科学技术院）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

Comments 20 page

Journal ref NIPS 2025 Poster

2509.16679 2025-09-23 cs.CL 57%

Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

Keliang Liu, Dingkang Yang, Ziyun Qian, Weijie Yin, Yuchi Wang, Hongsheng Li, Jun Liu, Peng Zhai, Yang Liu, Lihua Zhang

机构 * Fudan University（复旦大学）； The Chinese University of Hong Kong, MMLab（香港中文大学MMLab）； Lancaster University（兰卡斯特大学）； Tongji University（同济大学）； The University of Toronto（多伦多大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments A Survey of Reinforcement Learning for Large Language Models

2506.07570 2025-09-22 cs.CV cs.AI 57%

OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization

Yixuan Yang, Zhen Luo, Tongsheng Ding, Junru Lu, Mingqi Gao, Jinyu Yang, Victor Sanchez, Feng Zheng

机构 * Southern University of Science and Technology（南方科技大学）； Shanghai Innovation Institute（上海创新研究院）； University of Warwick（沃林顿大学）

专题命中偏好对齐：DPO(abstract)；分类 cs.AI

2505.20415 2025-09-19 cs.CL 57%

Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision

Xingwei Tan, Marco Valentino, Mahmud Akhter, Maria Liakata, Nikolaos Aletras

机构 * School of Computer Science, University of Sheffield（谢菲尔德大学计算机科学学院）； School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦大学玛丽女王学院电子工程与计算机科学学院）； The Alan Turing Institute（艾伦·图灵研究所）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

Comments EMNLP 2025 (Main), 9+6 pages

2505.19176 2025-09-19 cs.CL 57%

Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge

Zhuo Liu, Moxin Li, Xun Deng, Qifan Wang, Fuli Feng

机构 * University of Science and Technology of China（中国科学技术大学）； National University of Singapore（新加坡国立大学）； Meta AI

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments Under review

2504.05154 2025-09-19 cs.CL 57%

CARE: Multilingual Human Preference Learning for Cultural Awareness

Geyang Guo, Tarek Naous, Hiromi Wakaki, Yukiko Nishimura, Yuki Mitsufuji, Alan Ritter, Wei Xu

机构 * Georgia Institute of Technology（佐治亚理工学院）； Sony Group Corporation（索尼集团）； Sony AI（索尼人工智能）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments Accepted at EMNLP 2025

2509.13869 2025-09-18 cs.CL 57%

Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs

Yang Liu, Chenhui Chu

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments 38 pages, 31 figures

2509.06652 2025-09-18 cs.CL 57%

IntrEx: A Dataset for Modeling Engagement in Educational Conversations

Xingwei Tan, Mahathi Parvatham, Chiara Gambi, Gabriele Pergola

机构 * Department of Computer Science, University of Warwick, UK（沃里克大学计算机科学系）； School of Computer Science, University of Sheffield, UK（谢菲尔德大学计算机科学学院）； Department of Psychology, University of Warwick, UK（沃里克大学心理学系）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL

Comments EMNLP 2025 Findings camera-ready, 9+7 pages

2509.12521 2025-09-17 cs.LG 57%

Phi: Preference Hijacking in Multi-modal Large Language Models at Inference Time

Yifan Lan, Yuanpu Cao, Weitong Zhang, Lu Lin, Jinghui Chen

机构 * The Pennsylvania State University（宾夕法尼亚州立大学）； The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）

专题命中偏好对齐：safety(abstract)；分类 cs.LG

2509.11287 2025-09-16 cs.CV cs.CL 57%

Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations

Yifan Lu, Ziqi Zhang, Chunfeng Yuan, Jun Gao, Congxuan Zhang, Xiaojuan Qi, Bing Li, Weiming Hu

机构 * Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information, CASIA（北京多模态信息超级智能安全重点实验室，中国科学院自动化所）； State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中国科学院自动化所）； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）； Hello Group（Hello集团）； Nanchang Hangkong University（南昌航空大学）； The University of Hong Kong（香港大学）； School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments emnlp 2025 accepted

2505.00047 2025-09-16 cs.CL 57%

Base Models Beat Aligned Models at Randomness and Creativity

Peter West, Christopher Potts

机构 * Stanford University（斯坦福大学）； University of British Columbia（不列颠哥伦比亚大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

2509.10147 2025-09-15 cs.AI 57%

Virtual Agent Economies

Nenad Tomasev, Matija Franklin, Joel Z. Leibo, Julian Jacobs, William A. Cunningham, Iason Gabriel, Simon Osindero

机构 * Google DeepMind（谷歌DeepMind）； University of Toronto（多伦多大学）

专题命中偏好对齐：safety(abstract)；分类 cs.AI

2509.09121 2025-09-12 cs.CL 57%

Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asia

Sophia Maria

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

2509.07370 2025-09-12 cs.CL 57%

PersonaFuse: A Personality Activation-Driven Framework for Enhancing Human-LLM Interactions

Yixuan Tang, Yi Yang, Ahmed Abbasi

机构 * The Hong Kong University of Science and Technology（香港科学与技术大学）； University of Notre Dame（诺丁汉大学）

专题命中偏好对齐：safety(abstract)；分类 cs.CL

2509.08500 2025-09-11 cs.AI 57%

TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making

Kechen Jiao, Zhirui Fang, Jiahao Liu, Bei Li, Qifan Wang, Xinyu Liu, Junhao Ruan, Zhongjian Qiao, Yifan Zhu, Yaxin Xu, Jingang Wang, Xiu Li

机构 * Tsinghua University（清华大学）； Meituan（美团）； Northeastern University（东北大学）； Beijing University of Posts and Telecommunications（北京邮电大学）； Meta AI ； Wuhan University（武汉大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI

2509.06286 2025-09-09 cs.LG 57%

RecMind: LLM-Enhanced Graph Neural Networks for Personalized Consumer Recommendations

Chang Xue, Youwei Lu, Chen Yang, Jinming Xing

机构 * Yeshiva University（耶鲁大学）； Oklahoma State University（俄克拉荷马州立大学）； Case Western Reserve University（凯斯西储大学）； North Carolina State University（北卡罗来纳州立大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.LG

2502.11689 2025-09-09 cs.CL 57%

Improve LLM-as-a-Judge Ability as a General Ability

Jiachen Yu, Shaoning Sun, Xiaohui Hu, Jiaxu Yan, Kaidong Yu, Xuelong Li

机构 * Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）； Institute of Artificial Intelligence, China Telecom (TeleAI)（中国电信人工智能研究院（TeleAI））； Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

2504.15477 2025-09-09 cs.LG 57%

In-context Ranking Preference Optimization

Junda Wu, Rohan Surana, Zhouhang Xie, Yiran Shen, Yu Xia, Tong Yu, Ryan A. Rossi, Prithviraj Ammanabrolu, Julian McAuley

机构 * UC San Diego（加州大学圣迭戈分校）； Adobe Research（Adobe研究）

专题命中偏好对齐：DPO(abstract)；分类 cs.LG

Comments 10 pages

2508.21496 2025-09-03 cs.CV cs.AI 57%

ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understanding

Hao Lu, Jiahao Wang, Yaolun Zhang, Ruohui Wang, Xuanyu Zheng, Yepeng Tang, Dahua Lin, Lewei Lu

机构 * Sensetime（秒氏科技）

专题命中偏好对齐：DPO(abstract)；分类 cs.AI

2509.01035 2025-09-03 cs.CL 57%

We Politely Insist: Your LLM Must Learn the Persian Art of Taarof

Nikta Gohari Sadr, Sahar Heidariasl, Karine Megerdoomian, Laleh Seyyed-Kalantari, Ali Emami

机构 * Brock University（布鲁克大学）； Zoorna AI ； York University（约克大学）； Emory University（埃默里大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments Accepted to EMNLP 2025 Main Conference

2504.03612 2025-09-03 cs.CL 57%

AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset

Bingxiang He, Wenbin Zhang, Jiaxi Song, Cheng Qian, Zixuan Fu, Bowen Sun, Ning Ding, Haiwen Hong, Longtao Huang, Hui Xue, Ganqu Cui, Wanxiang Che, Zhiyuan Liu, Maosong Sun

机构 * Tsinghua University（清华大学）； Harbin Institute of Technology（哈尔滨工业大学）； Shanghai AI Lab（上海人工智能实验室）； University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）； Alibaba Group（阿里巴巴集团）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments Accept at the Conference On Language Modeling (COLM) 2025

2508.19567 2025-08-28 cs.LG 57%

Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning

Sheryl Mathew, N Harshit

专题命中偏好对齐：RLHF(abstract)；分类 cs.LG

1. 偏好对齐 3243 篇

Agentic Reinforcement Learning with Implicit Step Rewards

Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation

OViP: Online Vision-Language Preference Learning for VLM Hallucination

SPARK: Synergistic Policy And Reward Co-Evolving Framework

Can Synthetic Query Rewrites Capture User Intent Better than Humans in Retrieval-Augmented Generation?

video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models

Unlocking Financial Insights: An advanced Multimodal Summarization with Multimodal Output Framework for Financial Advisory Videos

Agentic AutoSurvey: Let LLMs Survey LLMs

Preference Distillation via Value based Reinforcement Learning

Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization

Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision

Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge

CARE: Multilingual Human Preference Learning for Cultural Awareness

Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs

IntrEx: A Dataset for Modeling Engagement in Educational Conversations

Phi: Preference Hijacking in Multi-modal Large Language Models at Inference Time

Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations

Base Models Beat Aligned Models at Randomness and Creativity

Virtual Agent Economies

Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asia

PersonaFuse: A Personality Activation-Driven Framework for Enhancing Human-LLM Interactions

TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making

RecMind: LLM-Enhanced Graph Neural Networks for Personalized Consumer Recommendations

Improve LLM-as-a-Judge Ability as a General Ability

In-context Ranking Preference Optimization

ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understanding

We Politely Insist: Your LLM Must Learn the Persian Art of Taarof

AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset

Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning