大模型对齐与安全

A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment

Jean-Philippe Corbeil, Amin Dada, Jean-Michel Attendu, Asma Ben Abacha, Alessandro Sordoni, Lucas Caccia, François Beaulieu, Thomas Lin, Jens Kleesiek, Paul Vozila

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Journal ref ACL 2025

KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHF

Jason R Brown, Lennie Wells, Edward James Young, Sergio Bacallado

机构 * Computational and Biological Learning Group, Department of Engineering, University of Cambridge, Cambridge, UK（计算生物学学习组，工程系，剑桥大学，剑桥，英国） ； Department of Computer Science and Technology, University of Cambridge, Cambridge, UK（计算机科学与技术系，剑桥大学，剑桥，英国） ； Statistics Laboratory, Department of Pure Mathematics and Mathematical Statistics, University of Cambridge, UK（统计实验室，纯粹数学与数学统计系，剑桥大学，英国） 纠错

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.LG

CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions

Tae Soo Kim, Yoonjoo Lee, Yoonah Park, Jiho Kim, Young-Ho Kim, Juho Kim

机构 * KAIST（韩国科学技术院） ； Seoul National University（首尔国立大学） ； Calvin University（凯尔文大学） ； NAVER AI LAB（NAVER AI实验室） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to COLM 2025. Project Website: https://cupid.kixlab.org/

P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis

Feifan Song, Bofei Gao, Yifan Song, Yi Liu, Weimin Xiong, Yuyang Song, Tianyu Liu, Guoyin Wang, Houfeng Wang

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback

Tom S. Juzek, Zina B. Ward

机构 * Florida State University（佛罗里达州立大学） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted for publication in the Proceedings of the 5th Workshop on Bias and Fairness in AI (BIAS 2025) at ECML PKDD

G-Core: A Simple, Scalable and Balanced RLHF Trainer

Junyu Wu, Weiming Chang, Xiaotao Liu, Guanyou He, Haoqiang Hong, Boqi Liu, Hongtao Tian, Tao Yang, Yunsheng Shi, Feng Lin, Ting Yao

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Comments I haven't received company approval yet, and I uploaded it by mistake

Cultivating Helpful, Personalized, and Creative AI Tutors: A Framework for Pedagogical Alignment using Reinforcement Learning

Siyu Song, Wentao Liu, Ye Lu, Ruohua Zhang, Tao Liu, Jinze Lv, Xinyun Wang, Aimin Zhou, Fei Tan, Bo Jiang, Hao Hao

机构 * Shanghai Innavation Institute（上海创新研究院） ； Shanghai Institute of AI for Education（上海人工智能教育研究院） ； School of Computer Science and Technology（计算机科学与技术学院） ； Department of Educational Information Technology（教育信息技术系） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance

Shehzeen Hussain, Paarth Neekhara, Xuesong Yang, Edresson Casanova, Subhankar Ghosh, Mikyas T. Desta, Roy Fejgin, Rafael Valle, Jason Li

机构 * NVIDIA Corporation（英伟达公司） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Journal ref ICML Workshop on Machine Learning for Audio, 2025

MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models

Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Tianlin Zhang, Sophia Ananiadou

机构 * The University of Manchester（曼彻斯特大学） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by NeurIPS 2024 main track

Journal ref https://proceedings.neurips.cc/paper_files/paper/2024/hash/3d03800841fa1bb2f43ef1750aafcce4-Abstract-Conference.html

DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models

Sunghee Jung, Donghun Lee, Shinbok Lee, Gaeun Seo, Daniel Lee, Byeongil Ko, Junrae Cho, Kihyun Kim, Eunggyun Kim, Myeongcheol Shin

机构 * Kakao Corp.（韩国 Kakao 公司） 纠错

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.CL、cs.LG

Comments Accepted to SIGDIAL 2025

RSPO: Regularized Self-Play Alignment of Large Language Models

Xiaohang Tang, Sangwoong Yoon, Seongho Son, Huizhuo Yuan, Quanquan Gu, Ilija Bogunovic

机构 * University College London（伦敦大学学院） ； University of California, Los Angeles（加州大学洛杉矶分校） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Comments Preprint

Accelerating RLHF Training with Reward Variance Increase

Zonglin Yang, Zhexuan Gu, Houduo Qi, Yancheng Yuan

机构 * The Hong Kong Polytechnic University（香港理工大学） 纠错

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment

Xiaotian Zhang, Ruizhe Chen, Yang Feng, Zuozhu Liu

机构 * Zhejiang University（浙江大学） ； Angelalign Technology Inc.（Angelalign技术公司） ； Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments ACL Finding

Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding

Feifan Song, Shaohang Wei, Wen Luo, Yuxuan Fan, Tianyu Liu, Guoyin Wang, Houfeng Wang

机构 * State Key Laboratory of Multimedia Information Processing, School of Computer Science（多媒体信息处理国家重点实验室、计算机科学学院） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2025 Findings

SudoLM: Learning Access Control of Parametric Knowledge with Authorization Alignment

Qin Liu, Fei Wang, Chaowei Xiao, Muhao Chen

机构 * UC Davis（加州大学戴维斯分校） ； USC（南加州大学） ； UW-Madison（威斯康星大学麦迪逊分校） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments ACL 2025

Unlocking Recursive Thinking of LLMs: Alignment via Refinement

Haoke Zhang, Xiaobo Liang, Cunxiang Wang, Juntao Li, Min Zhang

机构 * Soochow University（苏州大学） ； Zhipu AI（智谱AI） ； Tsinghua University（清华大学） ； Key Laboratory of Data Intelligence and Advanced Computing, Soochow University（苏州大学数据智能与先进计算重点实验室） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to the Findings of ACL 2025

Multi-objective Large Language Model Alignment with Hierarchical Experts

Zhuo Li, Guodong Du, Weiyang Guo, Yigeng Zhou, Xiucheng Li, Wenya Wang, Fangming Liu, Yequan Wang, Deheng Ye, Min Zhang, Jing Li

机构 * Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳）） ； Nanyang Technological University（南洋理工大学） ； Peng Cheng Laboratory（鹏城实验室） ； Beijing Academy of Artificial Intelligence（北京人工智能研究院） ； Tencent（腾讯） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment

Jia-Nan Li, Jian Guan, Songhao Wu, Wei Wu, Rui Yan

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment

Wenqiao Zhu, Ji Liu, Lulu Wang, Jun Wu, Yulun Zhang

专题命中 偏好对齐 ：alignment(title)；DPO(abstract)；分类 cs.AI、cs.LG

Comments 18 pages, to appear in ACL'25

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model

Baijiong Lin, Weisen Jiang, Yuancheng Xu, Hao Chen, Ying-Cong Chen

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted by ICML 2025

Optimizing RLHF Training for Large Language Models with Stage Fusion

Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, Xin Jin

机构 * School of Computer Science, Peking University（北京大学计算机学院） 纠错

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.LG

VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models

Dahun Kim, AJ Piergiovanni, Ganesh Mallya, Anelia Angelova

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments CVPR 2025, project page at https://github.com/google-deepmind/video_comp

Preference-Based Alignment of Discrete Diffusion Models

Umberto Borso, Davide Paglieri, Jude Wells, Tim Rocktäschel

专题命中 偏好对齐 ：alignment(title)；DPO(abstract)；分类 cs.AI、cs.LG

Towards Federated RLHF with Aggregated Client Preference for LLMs

Feijie Wu, Xiaoze Liu, Haoyu Wang, Xingchen Wang, Lu Su, Jing Gao

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.LG

Comments ICLR'25

Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment

Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, Dylan J. Foster

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

A Shared Low-Rank Adaptation Approach to Personalized RLHF

Renpu Liu, Peng Wang, Donghao Li, Cong Shen, Jing Yang

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Comments Published as a conference paper at AISTATS 2025

Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization

Juntao Dai, Taiye Chen, Yaodong Yang, Qian Zheng, Gang Pan

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Comments Published as a conference paper at ICLR 2025

Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts

Beining Xu, Arkaitz Zubiaga

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.AI

Comments 14 pages, 3 figures

InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization

Yunan Wang, Jijie Li, Bo-Wen Zhang, Liangdong Wang, Guang Liu

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.CL、cs.LG

PAD: Personalized Alignment of LLMs at Decoding-Time

Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, Zuozhu Liu

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments ICLR 2025