大模型对齐与安全 - arXivDaily 专题

2511.07477 2025-11-12 cs.CY cs.AI cs.CL 75%

The Polite Liar: Epistemic Pathology in Language Models

Bentley DeVilling

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.CY

Comments 17 pages, 2 tables, Preprint - under review at AI & Society

2410.15483 2025-11-11 cs.LG cs.AI cs.CL math.OC stat.ML 75%

Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective

Heshan Fernando, Han Shen, Parikshit Ram, Yi Zhou, Horst Samulowitz, Nathalie Baracaldo, Tianyi Chen

机构 * Rensselaer Polytechnic Institute（罗切斯特理工学院）； IBM Research（IBM研究院）； Cornell University（康奈尔大学）

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2408.13518 2025-11-07 cs.CL cs.AI cs.LG 75%

Selective Preference Optimization via Token-Level Reward Function Estimation

Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, Sophia Ananiadou

机构 * National Centre for Text Mining, The University of Manchester（曼彻斯特大学文本挖掘中心）； School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）； Center for Language and Information Research, Wuhan University（武汉大学语言与信息研究中心）； The Fin AI（Fin AI公司）； Baidu Inc.（百度公司）； Archimedes Research（阿基米德研究）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by the EMNLP 2025 main conference

Journal ref https://aclanthology.org/2025.emnlp-main.359/

2502.20548 2025-10-21 cs.LG cs.AI cs.CL 75%

$Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training

Jin Peng Zhou, Kaiwen Wang, Jonathan Chang, Zhaolin Gao, Nathan Kallus, Kilian Q. Weinberger, Kianté Brantley, Wen Sun

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NeurIPS 2025

2410.23223 2025-10-15 cs.LG cs.AI cs.CL cs.GT 75%

COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences

Yixin Liu, Argyris Oikonomou, Weiqiang Zheng, Yang Cai, Arman Cohan

机构 * Yale University（耶鲁大学）； Allen Institute for AI（人工智能研究院）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2509.23024 2025-09-30 cs.LG cs.AI cs.CL 75%

Tracing the Representation Geometry of Language Models from Pretraining to Post-training

Melody Zixuan Li, Kumar Krishna Agrawal, Arna Ghosh, Komal Kumar Teru, Adam Santoro, Guillaume Lajoie, Blake A. Richards

机构 * Computer Science, McGill University（麦吉尔大学计算机科学系）； Mila - Quebec AI Institute（魁北克人工智能研究所）； UC Berkeley（加州大学伯克利分校）； G o o g l e , Paradigms of Intelligence Team（谷歌，智能范式团队）； Mathematics and Statistics, Université de Montréal（蒙特利尔大学数学与统计学系）； Neurology & Neurosurgery and Montreal Neurological Institute, McGill University（神经病学与神经外科及蒙特利尔神经研究所，麦吉尔大学）； CIFAR Learning in Machines & Brains Program（CIFAR 机器与大脑学习计划）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 33 pages, 14 figures, 9 tables

2507.05129 2025-09-19 cs.CL cs.CY cs.LG 75%

SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction

Alexander Scarlatos, Nigel Fernandez, Christopher Ormerod, Susan Lottridge, Andrew Lan

机构 * University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）； Cambium Assessment（Cambium评估）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.CY、cs.LG

Comments Published in EMNLP 2025: The 2025 Conference on Empirical Methods in Natural Language Processing

2507.07484 2025-07-11 cs.CL cs.AI cs.LG 75%

Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models

Kaiqu Liang, Haimin Hu, Xuandong Zhao, Dawn Song, Thomas L. Griffiths, Jaime Fernández Fisac

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Project page, code & data: https://machine-bullshit.github.io

2506.03637 2025-07-08 cs.CL cs.AI cs.LG 75%

RewardAnything: Generalizable Principle-Following Reward Models

Zhuohao Yu, Jiali Zeng, Weizheng Gu, Yidong Wang, Jindong Wang, Fandong Meng, Jie Zhou, Yue Zhang, Shikun Zhang, Wei Ye

机构 * Peking University（北京大学）； WeChat AI（微信AI）； William & Mary（威廉与玛丽学院）； Westlake University（西交利物浦大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 25 pages, 9 figures, Code & model weights available at: https://zhuohaoyu.github.io/RewardAnything

2507.00092 2025-07-02 cs.AI cs.CL cs.LG 75%

Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models

Basab Jha, Firoj Paudel, Ujjwal Puri, Zhang Yuting, Choi Donghyuk, Wang Junhao

机构 * SAGEA ； Tribhuwan University（特里布文大学）； Vedas College（维达学院）； Madan Bhandari Memorial College（马达恩·班达里纪念学院）； Fudan University（复旦大学）； ETH Zurich（苏黎世联邦理工学院）

专题命中偏好对齐：safety(abstract)；AI safety(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 19 pages, 2 figures, 9 tables

2407.10490 2025-07-01 cs.LG cs.AI cs.CL 75%

Learning Dynamics of LLM Finetuning

Yi Ren, Danica J. Sutherland

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2506.08712 2025-06-13 cs.CL cs.AI cs.LG 75%

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

Hee Suk Yoon, Eunseop Yoon, Mark Hasegawa-Johnson, Sungwoong Kim, Chang D. Yoo

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）； University of Illinois Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校）； Korea University（韩国大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICML 2025

2506.03056 2025-06-04 cs.AI cs.CY cs.LG 75%

Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models

Ram Potham, Max Harms

机构 * Independent Researcher（独立研究者）； Machine Intelligence Research Institute（机器智能研究院）

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.AI、cs.CY、cs.LG

Comments Preprint. This work has been submitted to the Reliable and Responsible Foundation Models Workshop at ICML 2025 for review

2505.19964 2025-05-27 cs.LG cs.AI cs.CL cs.GT 75%

The Limits of Preference Data for Post-Training

Eric Zhao, Jessica Dai, Pranjal Awasthi

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

2505.01706 2025-05-06 cs.AI cs.CL cs.LG 75%

Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

Sarvesh Shashidhar, Ritik, Nachiketa Patil, Suraj Racha, Ganesh Ramakrishnan

机构 * Centre for Machine Intelligence and Data Science, IIT Bombay（印度理工学院班加罗尔机器智能与数据科学中心）； Koita Centre for Digital Health, IIT Bombay（印度理工学院班加罗尔数字健康中心）； Department of Computer Science and Engineering, IIT Bombay（印度理工学院班加罗尔计算机科学与工程系）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Updated abstract, algorithm and experimental results

2504.14439 2025-04-22 cs.LG cs.AI cs.CL 75%

LoRe: Personalizing LLMs via Low-Rank Reward Modeling

Avinandan Bose, Zhihan Xiong, Yuejie Chi, Simon Shaolei Du, Lin Xiao, Maryam Fazel

机构 * FAIR at Meta（Meta 的 FAIR 部门）； University of Washington（华盛顿大学）； Carnegie Mellon University（卡内基梅隆大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2503.10965 2025-03-31 cs.AI cs.CL cs.LG 75%

Auditing language models for hidden objectives

Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus, Siddharth Mishra-Sharma, Daniel Ziegler, Emmanuel Ameisen, Joshua Batson, Tim Belonax, Samuel R. Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, Florian Dietz, Satvik Golechha, Akbir Khan, Jan Kirchner, Jan Leike, Austin Meek, Kei Nishimura-Gasparian, Euan Ong, Christopher Olah, Adam Pearce, Fabien Roger, Jeanne Salle, Andy Shih, Meg Tong, Drake Thomas, Kelley Rivoire, Adam Jermyn, Monte MacDiarmid, Tom Henighan, Evan Hubinger

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2412.05818 2025-03-26 cs.CV cs.AI cs.CL cs.LG cs.MM 75%

SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation

Leigang Qu, Haochuan Li, Wenjie Wang, Xiang Liu, Juncheng Li, Liqiang Nie, Tat-Seng Chua

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments CVPR 2025 Camera-ready. Project page: https://silmm.github.io/

2503.10723 2025-03-17 cs.CL cs.AI cs.IR cs.LG 75%

RankPO: Preference Optimization for Job-Talent Matching

Yafei Zhang, Murray Wang, Yu Wang, Xiaohui Wang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages, 3 figures, 7 tables

2501.00581 2025-02-25 cs.CL cs.AI cs.LG 75%

Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective

Yipeng Kang, Junqi Wang, Yexin Li, Mengmeng Wang, Wenming Tu, Quansen Wang, Hengli Li, Tingjun Wu, Xue Feng, Fangwei Zhong, Zilong Zheng

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2412.10400 2025-02-25 cs.CL cs.AI cs.LG 75%

Reinforcement Learning Enhanced LLMs: A Survey

Shuhe Wang, Shengyu Zhang, Jie Zhang, Runyi Hu, Xiaoya Li, Tianwei Zhang, Jiwei Li, Fei Wu, Guoyin Wang, Eduard Hovy

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.12206 2025-02-19 cs.AI cs.CL cs.LG 75%

Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?

Yufei He, Yuexin Li, Jiaying Wu, Yuan Sui, Yulin Chen, Bryan Hooi

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2501.07886 2025-01-15 cs.LG cs.AI cs.CL 75%

Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision

Yaowen Ye, Cassidy Laidlaw, Jacob Steinhardt

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 22 pages, 10 figures

2404.00530 2025-01-09 cs.CL cs.AI cs.LG 75%

Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization

Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, Aditya Grover

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 22 pages, 16 figures, 7 tables

2409.06691 2024-12-31 cs.LG cs.AI cs.CL 75%

Geometric-Averaged Preference Optimization for Soft Preference Labels

Hiroki Furuta, Kuang-Huei Lee, Shixiang Shane Gu, Yutaka Matsuo, Aleksandra Faust, Heiga Zen, Izzeddin Gur

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at NeurIPS 2024

2412.15244 2024-12-23 cs.CL cs.AI cs.LG 75%

MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples

Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by COLING2025

2310.13008 2024-12-20 cs.LG cs.AI cs.CL 75%

DavIR: Data Selection via Implicit Reward for Large Language Models

Haotian Zhou, Tingkai Liu, Qianli Ma, Yufeng Zhang, Jianbo Yuan, Pengfei Liu, Yang You, Hongxia Yang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.10522 2024-12-19 cs.LG cs.AI cs.CL 75%

Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

Jifan Zhang, Lalit Jain, Yang Guo, Jiayi Chen, Kuan Lok Zhou, Siddharth Suresh, Andrew Wagenmaker, Scott Sievert, Timothy Rogers, Kevin Jamieson, Robert Mankoff, Robert Nowak

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2412.02685 2024-12-04 cs.CL cs.AI cs.LG 75%

T-REG: Preference Optimization with Token-Level Reward Regularization

Wenxuan Zhou, Shujian Zhang, Lingxiao Zhao, Tao Meng

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.13846 2024-12-04 cs.LG cs.AI cs.CL 75%

Filtered Direct Preference Optimization

Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, Kaito Ariu

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments EMNLP 2024

1. 偏好对齐 3235 篇

The Polite Liar: Epistemic Pathology in Language Models

Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective

Selective Preference Optimization via Token-Level Reward Function Estimation

$Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training

COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences

Tracing the Representation Geometry of Language Models from Pretraining to Post-training

SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction

Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models

RewardAnything: Generalizable Principle-Following Reward Models

Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models

Learning Dynamics of LLM Finetuning

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models

The Limits of Preference Data for Post-Training

Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

LoRe: Personalizing LLMs via Low-Rank Reward Modeling

Auditing language models for hidden objectives

SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation

RankPO: Preference Optimization for Job-Talent Matching

Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective

Reinforcement Learning Enhanced LLMs: A Survey

Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?

Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision

Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization

Geometric-Averaged Preference Optimization for Soft Preference Labels

MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples

DavIR: Data Selection via Implicit Reward for Large Language Models

Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

T-REG: Preference Optimization with Token-Level Reward Regularization

Filtered Direct Preference Optimization