大模型对齐与安全

Preference Alignment Improves Language Model-Based TTS

Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watanabe, Dong Yu

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

What Makes and Breaks Safety Fine-tuning? A Mechanistic Study

Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H. S. Torr, Amartya Sanyal, Puneet K. Dokania

专题命中 偏好对齐 ：safety(title,abstract)；jailbreak(abstract)；分类 cs.CL、cs.LG

Comments Preprint

LIRE: listwise reward enhancement for preference alignment

Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, Zhendong Mao

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments Accepted by ACL 2024 Findings

Direct Alignment of Language Models via Quality-Aware Self-Refinement

Runsheng Yu, Yong Wang, Xiaoqi Jiao, Youzhi Zhang, James T. Kwok

专题命中 偏好对齐 ：alignment(title)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment

Yueqin Yin, Zhendong Wang, Yujia Xie, Weizhu Chen, Mingyuan Zhou

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Understanding the performance gap between online and offline alignment algorithms

Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, Will Dabney

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

MetaRM: Shifted Distributions Alignment via Meta-Learning

Shihan Dou, Yan Liu, Enyu Zhou, Tianlong Li, Haoxiang Jia, Limao Xiong, Xin Zhao, Junjie Ye, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments 11 pages, 6 figures. arXiv admin note: text overlap with arXiv:2401.06080

Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization

Swaroop Nath, Tejpalsingh Siledar, Sankara Sri Raghava Ravindra Muddu, Rupasai Rangaraju, Harshad Khadilkar, Pushpak Bhattacharyya, Suman Banerjee, Amey Patil, Sudhanshu Shekhar Singh, Muthusamy Chelliah, Nikesh Garera

专题命中 偏好对齐 ：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.LG

Comments 19 pages, 6 figures, 21 tables

Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

Anand Siththaranjan, Cassidy Laidlaw, Dylan Hadfield-Menell

专题命中 偏好对齐 ：RLHF(title,abstract)；jailbreak(abstract)；分类 cs.AI、cs.LG

Comments Presented at ICLR 2024

Configurable Safety Tuning of Language Models with Synthetic Preference Data

Victor Gallego

专题命中 偏好对齐 ：safety(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment

Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, Yuandong Tian

专题命中 偏好对齐 ：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2024

Preference Ranking Optimization for Human Alignment

Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, Houfeng Wang

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments Accepted by AAAI 2024

ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization

Feifan Song, Yuxuan Fan, Xin Zhang, Peiyi Wang, Houfeng Wang

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Aligning Large Language Models with Counterfactual DPO

Bradley Butcher

专题命中 偏好对齐 ：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning

Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, Yejin Choi

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 26 pages, 8 figures. Project website: https://allenai.github.io/re-align/

Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language

Di Jin, Shikib Mehri, Devamanyu Hazarika, Aishwarya Padmakumar, Sungjin Lee, Yang Liu, Mahdi Namazifar

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments Accepted by Workshop on Instruction Tuning and Instruction Following at NeurIPS 2023, Submitted to AAAI 2024

OpenAssistant Conversations -- Democratizing Large Language Model Alignment

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, Alexander Mattick

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments Published in NeurIPS 2023 Datasets and Benchmarks

Zephyr: Direct Distillation of LM Alignment

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, Thomas Wolf

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment

Jixiang Hong, Quan Tu, Changyu Chen, Xing Gao, Ji Zhang, Rui Yan

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, Matthieu Cord

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Representative Social Choice: From Learning Theory to AI Alignment

Tianyi Qiu

机构 * Peking University（北京大学） ； UC Berkeley（加州大学伯克利分校） ； Center for Human-Compatible AI（人类兼容人工智能中心） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.CY

Comments Journal of Artificial Intelligence Research, in press. Best Paper at NeurIPS 2024 Pluralistic Alignment Workshop

MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models

Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, Jiaqi Wang

专题命中 偏好对齐 ：DPO(title,abstract)；alignment(abstract)；分类 cs.AI

Comments Project URL: https://github.com/Liuziyu77/MIA-DPO

Learning When to Trust via Selective Context Preference Optimization

通过选择性上下文偏好优化学习何时信任

Xian Sun, Wei Chow, Yingshuo Wang, Junhao Liu, Wei Gao, Qing Wu, Lingdong Kong

机构 * Duke University（杜克大学） ； National University of Singapore（新加坡国立大学） ； UC Berkeley（加州大学伯克利分校） ； UC Irvine（加州大学欧文分校） ； Northeastern University（东北大学） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 偏好对齐 ：DPO(summary_cn,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 本研究针对语言模型易受误导性外部信号影响的问题，提出SCOPE方法，在含四种条件的MIST基准上优化DPO目标，降低SC2W的同时保持正常上下文下的准确率，主张以选择性信任评判模型。

Comments Project Page at https://worldbench.github.io/scope GitHub Repo at https://github.com/worldbench/SCOPE HF Dataset at https://huggingface.co/datasets/worldbench/MIST-Bench

Safety in Batches? Understanding and Mitigating Safety Failures in Batch Prompting

批量提示中的安全性？理解并缓解批量提示中的安全故障

Kihyun Kim, Hee-Seon Kim, Wonjun Lee, Changick Kim

专题命中 偏好对齐 ：safety(title,abstract)；alignment(abstract)；jailbreak(comments)

AI总结 该研究发现批量提示存在独特安全故障模式，可作为黑盒攻击实现高成功率，提出感知批量的偏好优化可缓解该漏洞，为大语言模型安全对齐提供了新方向。

Comments jailbreak, safety

EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent

EvoPref：多目标进化优化发现超越梯度下降的多样化大语言模型对齐

Dongxin Guo, Jikun Wu, Siu Ming Yiu

机构 * The University of Hong Kong（香港大学） ； Stellaris AI Limited（Stellaris AI有限公司） 纠错

专题命中 偏好对齐 ：DPO(abstract,abstract_cn)；alignment(abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 EvoPref通过多目标进化算法在帮助性、无害性和诚实性目标上优化LoRA适配器，发现比梯度下降更丰富的对齐方式，实验显示其在偏好覆盖和崩溃率上均有显著提升。

Comments 10 pages, 2 figures, 6 tables, 1 algorithm. Accepted to GECCO 2026

LAD-VF: LLM-Automatic Differentiation Enables Fine-Tuning-Free Robot Planning from Formal Methods Feedback

LAD-VF：LLM自动微分实现基于形式化方法反馈的无微调机器人规划

Yunhao Yang, Junyuan Hong, Gabriel Jacob Perin, Zhiwen Fan, Li Yin, Zhangyang Wang, Ufuk Topcu

机构 * The University of Texas at Austin（德克萨斯大学奥斯汀分校） ； University of São Paulo（圣保罗大学） ； Texas A&M University（德克萨斯A&M大学） ； SylphAI 纠错

专题命中 偏好对齐 ：DPO(abstract,abstract_cn)；alignment(abstract)；safety(abstract)；trustworthy(abstract)

AI总结 提出LAD-VF框架，利用形式化验证反馈和LLM自动微分自动优化提示词，无需微调即可提升机器人规划任务中规范符合率，成功率从60%提升至90%以上。

Comments Presented at ICRA 2026

Towards Anatomically Plausible Human Image Generation via Synthetic Localized Preferences

通过合成局部偏好实现解剖学合理的人体图像生成

Bao Li, Yuliang Xiu, Zhen Liu

机构 * Westlake University（西湖大学） ； The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）） 纠错

专题命中 偏好对齐 ：DPO(summary_cn,abstract)；alignment(abstract)

AI总结 提出 ASAP 框架，利用局部退化机制构建受控偏好对，并结合局部有界 DPO 变体，在保持整体图像质量的同时减少解剖学错误。

StepAudio 2.5 Technical Report

StepAudio 2.5 技术报告

Bin Lin, Bo Zhao, Boyong Wu, Chao Yan, Chen Wu, Cheng Yi, Chengyuan Yao, Daijiao Liu, Fei Tian, Feng Tian, Haiyang Sun, Haoyang Zhang, Jiangjie Zhen, Jinglan Gong, Jun Chen, Li Xie, Peilin Li, Peng Yang, Pengfei Tan, Qingjian Lin, Runze Li, Shenghua Hu, Siyi Zhou, Wenwen Qu, Xiangyu Li, Xiangyu Tony Zhang, Xuerui Yang, Yang Yang, Yechang Huang, Yu Fu, Yuchu Luo, Yuxin Li, Yuxin Zhang, Zhengyan Sheng, Brian Li, Chang Zeng, Changlin Zhang, Chen Geng, Chenghao Dong, Chengli Feng, Dan Zhou, Danni Wan, Di Chen, Die Zhang, Dongqing Pang, Guanglong Yang, Guoqiang Hu, Huangxi Zhu, Jianzheng Gao, Jinghua Liang, Jinmei Wan, Junjie Yuan, Kang An, Lei Lei, Limin Zhong, Lun Cai, Mengqiang Ren, Min Xu, Mingliang Li, Mingxiao Li, Na Wang, Qiang Tong, Qiaoling Huang, Qingfu Du, Rui Wang, Shengchen Zhou, Shi Qiu, Shihao Peng, Shiliang Yang, Siqi Tu, Tianjiao Deng, Ting Xu, Tong Wang, WeiMing Niu, Wuxun Xie, Xianwei Zhang, Xianyu Feng, Xiaojia Liu, Xing Chen, Xiongbin Wu, Yan Wu, Yang Li, Yi Liu, Yifan Zhang, Yile Liu, Yongshen Long, Yu Luo, Yuanhao Ding, Yuhao Wang, Yuhe Yin, Yunfang Xu, Yuxiang Yang, Zhiguo Huang, Zhiyue Wu, Zichao Li, Zichao Zhou, Daxin Jiang, Future Li, Gang Yu, Xiangyu Zhang, Yibo Zhu

专题命中 偏好对齐 ：RLHF(summary_cn,abstract)；alignment(abstract)

AI总结 提出统一音频语言基础模型StepAudio 2.5，通过任务定制的RLHF后训练和专用解码策略，在ASR、TTS和实时口语交互三项任务上达到或超越专用系统水平。

QwenSafe: Multimodal Content Rating Description Identification via Preference-Aligned VLMs

QwenSafe: 通过偏好对齐的视觉语言模型实现多模态内容评级描述识别

Dishanika Denipitiyage, Aruna Seneviratne, Suranga Seneviratne

机构 * University of Sydney（悉尼大学） ； University of New South Wales（新南威尔士大学） 纠错

专题命中 偏好对齐 ：DPO(summary_cn,abstract)；alignment(abstract)

AI总结 本文提出QwenSafe，一种通过联合推理应用元数据和截图来自动识别苹果定义的内容评级描述（CRDs）的视觉语言模型，通过引入metadata2CRD数据构建管道和直接偏好优化（DPO）提升模型预测准确性，实验结果显示QwenSafe在二元CRD分类中显著优于现有模型。

GRLO: Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zero

GRLO：从零开始在开放环境中的通用强化学习

Shangjian Yin, Yu Fu, Yue Dong, Zhouxing Shi

机构 * University of California, Riverside（加州大学河滨分校） 纠错

专题命中 偏好对齐 ：RLHF(summary_cn,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 GRLO研究从少量交互数据中训练的RLHF在开放环境中的泛化能力，探索其对话能力是否能迁移至数学推理和代码生成等下游任务，展示出高效且低成本的训练方法。