大模型对齐与安全

Convex Optimization for Alignment and Preference Learning on a Single GPU

单GPU上的对齐与偏好学习的凸优化

Miria Feng, Mert Pilanci

机构 * Department of Electrical Engineering, Stanford University, California, United States（斯坦福大学电气工程系，加州，美国） 纠错

专题命中 偏好对齐 ：DPO(summary_cn,abstract)；alignment(title,abstract)；RLHF(abstract,abstract_cn)；分类 cs.LG

AI总结 提出COALA算法，利用凸优化重表述消除参考模型，在单GPU上高效训练，性能与DPO相当但计算量仅为其17.6%。

TPMM-DPO: Trajectory-aware Preference-guided Model Merging for Iterative Direct Preference Optimization

TPMM-DPO：轨迹感知偏好引导的模型合并用于迭代直接偏好优化

Lingling Fu, Yongfu Xu

专题命中 偏好对齐 ：DPO(title,title_cn)；alignment(abstract)

AI总结 针对迭代DPO中噪声累积导致的过优化和性能退化问题，提出TPMM-DPO方法，通过将迭代策略模型序列视为优化轨迹并自适应融合，构建更平滑鲁棒的参考模型，从而提升训练稳定性和生成质量。

Comments 11 pages,6 figures

StepAudio 2.5 Technical Report

StepAudio 2.5 技术报告

Bin Lin, Bo Zhao, Boyong Wu, Chao Yan, Chen Wu, Cheng Yi, Chengyuan Yao, Daijiao Liu, Fei Tian, Feng Tian, Haiyang Sun, Haoyang Zhang, Jiangjie Zhen, Jinglan Gong, Jun Chen, Li Xie, Peilin Li, Peng Yang, Pengfei Tan, Qingjian Lin, Runze Li, Shenghua Hu, Siyi Zhou, Wenwen Qu, Xiangyu Li, Xiangyu Tony Zhang, Xuerui Yang, Yang Yang, Yechang Huang, Yu Fu, Yuchu Luo, Yuxin Li, Yuxin Zhang, Zhengyan Sheng, Brian Li, Chang Zeng, Changlin Zhang, Chen Geng, Chenghao Dong, Chengli Feng, Dan Zhou, Danni Wan, Di Chen, Die Zhang, Dongqing Pang, Guanglong Yang, Guoqiang Hu, Huangxi Zhu, Jianzheng Gao, Jinghua Liang, Jinmei Wan, Junjie Yuan, Kang An, Lei Lei, Limin Zhong, Lun Cai, Mengqiang Ren, Min Xu, Mingliang Li, Mingxiao Li, Na Wang, Qiang Tong, Qiaoling Huang, Qingfu Du, Rui Wang, Shengchen Zhou, Shi Qiu, Shihao Peng, Shiliang Yang, Siqi Tu, Tianjiao Deng, Ting Xu, Tong Wang, WeiMing Niu, Wuxun Xie, Xianwei Zhang, Xianyu Feng, Xiaojia Liu, Xing Chen, Xiongbin Wu, Yan Wu, Yang Li, Yi Liu, Yifan Zhang, Yile Liu, Yongshen Long, Yu Luo, Yuanhao Ding, Yuhao Wang, Yuhe Yin, Yunfang Xu, Yuxiang Yang, Zhiguo Huang, Zhiyue Wu, Zichao Li, Zichao Zhou, Daxin Jiang, Future Li, Gang Yu, Xiangyu Zhang, Yibo Zhu

专题命中 偏好对齐 ：RLHF(summary_cn,abstract)；alignment(abstract)

AI总结 提出统一音频语言基础模型StepAudio 2.5，通过任务定制的RLHF后训练和专用解码策略，在ASR、TTS和实时口语交互三项任务上达到或超越专用系统水平。

RAG-Pull: Turning Retrieval into a Code-Injection Channel via Invisible Unicode Perturbations

RAG-Pull：通过不可见Unicode扰动将检索转化为代码注入通道

Aritra Dhar, Vasilije Stambolic, Lukas Cavigelli

机构 * Computing System Labs, Huawei Technologies Switzerland AG（华为瑞士技术有限公司计算系统实验室） ； BKW Energie AG（BKW能源集团） 纠错

专题命中 偏好对齐 ：alignment(abstract)；safety(abstract)；分类 cs.AI

AI总结 提出RAG-Pull攻击，通过向查询或外部代码库插入不可见UTF字符，使检索系统偏向恶意代码，从而破坏LLM的安全对齐，实现远程代码执行和SQL注入。

Mitigating Object Hallucinations via Sentence-Level Early Intervention

通过句子级早期干预缓解对象幻觉

Shangpin Peng, Senqiao Yang, Li Jiang, Zhuotao Tian

机构 * Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）） ； The Chinese University of Hong Kong（香港中文大学） ； The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）） 纠错

专题命中 偏好对齐 ：DPO(abstract,abstract_cn)

AI总结 提出SENTINEL框架，通过句子级早期干预和领域内偏好学习，无需人工标注即可显著减少多模态大语言模型的对象幻觉，并在幻觉和通用能力基准上超越现有方法。

Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modeling

超越基于VLM的奖励：扩散原生潜在奖励建模

Gongye Liu, Bo Yang, Yida Zhi, Zhizhou Zhong, Lei Ke, Didan Deng, Han Gao, Yongxiang Huang, Kaihao Zhang, Hongbo Fu, Wenhan Luo

机构 * The Hong Kong University of Science ； Huawei Hong Kong AI Framework \& Data Technologies Lab ； Tsinghua University ； The Australian National University 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

AI总结 提出扩散原生潜在奖励模型DiNa-LRM，直接在噪声扩散状态上进行偏好学习，通过噪声校准Thurstone似然和推理时噪声集成，实现高效且鲁棒的奖励建模。

Comments Accepted by ICML 2026. Code: https://github.com/HKUST-C4G/diffusion-rm

Human-in-the-Loop Multi-Agent Ventilator Decision Support with Contextual Bandit Preference Learning

人在回路的多智能体呼吸机决策支持与上下文赌博机偏好学习

Sijia Li, Xiaoyu Tan, Qixing Wang, Weiyi Zhao, Chen Zhan, Teqi Hao, Xuemin Wang, Lei Gu, Roland Eils, Xihe Qiu

机构 * Shanghai University of Engineering Science, Shanghai, China Tencent Youtu Lab, Tencent, China Department of Critical Care Medicine, Shanghai Tenth People's Hospital, Tongji University School of Medicine, Shanghai, China Department of Emergency ； Critical Disease, Songjiang Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, China Max Planck Institute for Heart ； Lung Research, Bad Nauheim, Germany Fudan University, Shanghai, China BIH at Charit\'e -- Universit\"atsmedizin Berlin, Berlin, Germany 纠错

专题命中 偏好对齐 ：safety(abstract)；分类 cs.AI

AI总结 提出VDSS框架，通过人在回路的多智能体协作和上下文赌博机在线偏好适应，实现呼吸机决策支持的个性化与可审计性。

Comments miccai 2026