大模型对齐与安全 - arXivDaily 专题

2504.02949 2025-04-07 cs.CV cs.AI 57%

VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning

Xianwei Zhuang, Yuxin Xie, Yufan Deng, Dongchao Yang, Liming Liang, Jinghan Ru, Yuguo Yin, Yuexian Zou

专题命中偏好对齐：DPO(abstract)；分类 cs.AI

Comments Code is available at: https://github.com/VARGPT-family/VARGPT-v1.1. arXiv admin note: text overlap with arXiv:2501.12327

2504.02463 2025-04-04 cs.IR cs.AI 57%

Evaluating AI Recruitment Sourcing Tools by Human Preference

Vladimir Slaykovskiy, Maksim Zvegintsev, Yury Sakhonchyk, Hrachik Ajamian

专题命中偏好对齐：alignment(abstract)；分类 cs.AI

2503.22230 2025-04-03 cs.LG 57%

Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback

Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue, Lin Yan

专题命中偏好对齐：RLHF(abstract)；分类 cs.LG

2503.18089 2025-03-25 cs.CL 57%

$D^2LoRA$: Data-Driven LoRA Initialization for Low Resource Tasks

Javad SeraJ, Mohammad Mahdi Mohajeri, Mohammad Javad Dousti

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

2503.17489 2025-03-25 cs.CL cs.CV 57%

Judge Anything: MLLM as a Judge Across Any Modality

Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang, Qi Qin, Zhongyi Zhang, Zhiyuan Zhang, Zetong Zhou, Shuang Gong, Yi Gui, Yao Wan, Philip S. Yu

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

2502.18008 2025-03-24 cs.SD cs.AI eess.AS 57%

NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigms

Yashan Wang, Shangda Wu, Jianhuai Hu, Xingjian Du, Yueqi Peng, Yongxin Huang, Shuai Fan, Xiaobing Li, Feng Yu, Maosong Sun

专题命中偏好对齐：DPO(abstract)；分类 cs.AI

2503.12167 2025-03-20 cs.CL 57%

PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing

Cheng Deng, Luoyang Sun, Jiwen Jiang, Yongcheng Zeng, Xinjian Wu, Wenxin Zhao, Qingfa Xiao, Jiachuan Wang, Haoyang Li, Lei Chen, Lionel M. Ni, Haifeng Zhang, Jun Wang

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL

2503.01224 2025-03-18 cs.LG 57%

CE-U: Cross Entropy Unlearning

Bo Yang

专题命中偏好对齐：DPO(abstract)；分类 cs.LG

2405.16236 2025-03-17 stat.ML cs.LG 57%

A transfer learning framework for weak-to-strong generalization

Seamus Somerstep, Felipe Maia Polo, Moulinath Banerjee, Ya'acov Ritov, Mikhail Yurochkin, Yuekai Sun

专题命中偏好对齐：alignment(abstract)；分类 cs.LG

Comments v2: Major changes to set up, theory, and experiments v3: Camera ready

2503.09223 2025-03-13 cs.IR cs.AI 57%

LREF: A Novel LLM-based Relevance Framework for E-commerce

Tian Tang, Zhixing Tian, Zhenyu Zhu, Chenyang Wang, Haiqing Hu, Guoyu Tang, Lin Liu, Sulong Xu

专题命中偏好对齐：DPO(abstract)；分类 cs.AI

2503.09029 2025-03-13 cs.CL 57%

DAST: Difficulty-Aware Self-Training on Large Language Models

Boyang Xue, Qi Zhu, Hongru Wang, Rui Wang, Sheng Wang, Hongling Xu, Fei Mi, Yasheng Wang, Lifeng Shang, Qun Liu, Kam-Fai Wong

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

2503.06358 2025-03-11 cs.LG 57%

Language Model Personalization via Reward Factorization

Idan Shenfeld, Felix Faltings, Pulkit Agrawal, Aldo Pacchiano

专题命中偏好对齐：RLHF(abstract)；分类 cs.LG

2503.04783 2025-03-10 cs.CL cs.CR 57%

Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospects

Anichur Rahman, Shahariar Hossain Mahir, Md Tanjum An Tashrif, Airin Afroj Aishi, Md Ahsan Karim, Dipanjali Kundu, Tanoy Debnath, Md. Abul Ala Moududi, MD. Zunead Abedin Eidmum

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL

2503.04222 2025-03-07 cs.CL 57%

FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion

Ziyi Yang, Fanqi Wan, Longguang Zhong, Canbin Huang, Guosheng Liang, Xiaojun Quan

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

Comments Technical report

2410.18640 2025-03-07 cs.CL 57%

Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model

Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, Rui Wang

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments ICLR 2025(Spotlight)

2405.11226 2025-03-06 cs.LG 57%

The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback

Ruitao Chen, Liwei Wang

专题命中偏好对齐：RLHF(abstract)；分类 cs.LG

2410.02712 2025-03-05 cs.CV cs.CL 57%

LLaVA-Critic: Learning to Evaluate Multimodal Models

Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, Chunyuan Li

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments Accepted by CVPR 2025; Project Page: https://llava-vl.github.io/blog/2024-10-03-llava-critic

2407.10944 2025-03-04 cs.CL 57%

Naturally Occurring Feedback is Common, Extractable and Useful

Shachar Don-Yehiya, Leshem Choshen, Omri Abend

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

2407.03856 2025-03-04 cs.LG 57%

Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation

Yi-Chen Li, Fuxiang Zhang, Wenjie Qiu, Lei Yuan, Chengxing Jia, Zongzhang Zhang, Yang Yu, Bo An

专题命中偏好对齐：RLHF(abstract)；分类 cs.LG

Comments Camera ready version of ICLR 2025

2403.08694 2025-03-04 cs.CL 57%

TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning

Shangding Gu, Alois Knoll, Ming Jin

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL

2410.17146 2025-03-03 cs.LG cs.CV 57%

LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging

Ke Wang, Nikolaos Dimitriadis, Alessandro Favero, Guillermo Ortiz-Jimenez, Francois Fleuret, Pascal Frossard

专题命中偏好对齐：RLHF(abstract)；分类 cs.LG

Comments The first two authors contributed equally to this work. Accepted at ICLR 2025. Project website: https://lines-merging.github.io

2409.02392 2025-03-03 cs.LG stat.ML 57%

Building Math Agents with Multi-Turn Iterative Preference Learning

Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, Chi Jin, Tong Zhang, Tianqi Liu

专题命中偏好对齐：DPO(abstract)；分类 cs.LG

Comments A multi-turn direct preference learning framework for tool-integrated reasoning tasks

2409.13156 2025-02-28 cs.CL 57%

RRM: Robust Reward Model Training Mitigates Reward Hacking

Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasiia Makarova, Jeremiah Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, Mohammad Saleh

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

Comments Accepted in ICLR 2025

2412.03187 2025-02-27 cs.CL 57%

Weighted-Reward Preference Optimization for Implicit Model Fusion

Ziyi Yang, Fanqi Wan, Longguang Zhong, Tianyuan Shi, Xiaojun Quan

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments Accepted to ICLR 2025

2410.09008 2025-02-27 cs.CL 57%

SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction

Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E. Gonzalez, Bin Cui, Shuicheng Yan

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

Comments ICLR 2025. Project: https://github.com/YangLing0818/SuperCorrect-llm

2502.14044 2025-02-26 cs.CV cs.LG 57%

Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data

Yucheng Shi, Quanzheng Li, Jin Sun, Xiang Li, Ninghao Liu

专题命中偏好对齐：alignment(abstract)；分类 cs.LG

Comments Accepted by ICLR 2025. Code: https://github.com/sycny/SelfSynthX

2502.10248 2025-02-25 cs.CV cs.CL 57%

Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model

Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, Yu Zhou, Deshan Sun, Deyu Zhou, Jian Zhou, Kaijun Tan, Kang An, Mei Chen, Wei Ji, Qiling Wu, Wen Sun, Xin Han, Yanan Wei, Zheng Ge, Aojie Li, Bin Wang, Bizhu Huang, Bo Wang, Brian Li, Changxing Miao, Chen Xu, Chenfei Wu, Chenguang Yu, Dapeng Shi, Dingyuan Hu, Enle Liu, Gang Yu, Ge Yang, Guanzhe Huang, Gulin Yan, Haiyang Feng, Hao Nie, Haonan Jia, Hanpeng Hu, Hanqi Chen, Haolong Yan, Heng Wang, Hongcheng Guo, Huilin Xiong, Huixin Xiong, Jiahao Gong, Jianchang Wu, Jiaoren Wu, Jie Wu, Jie Yang, Jiashuai Liu, Jiashuo Li, Jingyang Zhang, Junjing Guo, Junzhe Lin, Kaixiang Li, Lei Liu, Lei Xia, Liang Zhao, Liguo Tan, Liwen Huang, Liying Shi, Ming Li, Mingliang Li, Muhua Cheng, Na Wang, Qiaohui Chen, Qinglin He, Qiuyan Liang, Quan Sun, Ran Sun, Rui Wang, Shaoliang Pang, Shiliang Yang, Sitong Liu, Siqi Liu, Shuli Gao, Tiancheng Cao, Tianyu Wang, Weipeng Ming, Wenqing He, Xu Zhao, Xuelin Zhang, Xianfang Zeng, Xiaojia Liu, Xuan Yang, Yaqi Dai, Yanbo Yu, Yang Li, Yineng Deng, Yingming Wang, Yilei Wang, Yuanwei Lu, Yu Chen, Yu Luo, Yuchu Luo, Yuhe Yin, Yuheng Feng, Yuxiang Yang, Zecheng Tang, Zekai Zhang, Zidong Yang, Binxing Jiao, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu, Heung-Yeung Shum, Daxin Jiang

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

Comments 36 pages, 14 figures

2408.15240 2025-02-25 cs.LG 57%

Generative Verifiers: Reward Modeling as Next-Token Prediction

Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, Rishabh Agarwal

专题命中偏好对齐：DPO(abstract)；分类 cs.LG

Comments ICLR 2025

2502.14191 2025-02-21 cs.CV cs.AI 57%

Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models

Michihiro Yasunaga, Luke Zettlemoyer, Marjan Ghazvininejad

专题命中偏好对齐：safety(abstract)；分类 cs.AI

Comments Dataset available at https://github.com/facebookresearch/multimodal_rewardbench

2411.11761 2025-02-21 cs.LG cs.HC 57%

Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework

Yannick Metz, David Lindner, Raphaël Baur, Mennatallah El-Assady

专题命中偏好对齐：RLHF(abstract)；分类 cs.LG

1. 偏好对齐 3243 篇

VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning

Evaluating AI Recruitment Sourcing Tools by Human Preference

Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback

$D^2LoRA$: Data-Driven LoRA Initialization for Low Resource Tasks

Judge Anything: MLLM as a Judge Across Any Modality

NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigms

PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing

CE-U: Cross Entropy Unlearning

A transfer learning framework for weak-to-strong generalization

LREF: A Novel LLM-based Relevance Framework for E-commerce

DAST: Difficulty-Aware Self-Training on Large Language Models

Language Model Personalization via Reward Factorization

Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospects

FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion

Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model

The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback

LLaVA-Critic: Learning to Evaluate Multimodal Models

Naturally Occurring Feedback is Common, Extractable and Useful

Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation

TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning

LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging

Building Math Agents with Multi-Turn Iterative Preference Learning

RRM: Robust Reward Model Training Mitigates Reward Hacking

Weighted-Reward Preference Optimization for Implicit Model Fusion

SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction

Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data

Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model

Generative Verifiers: Reward Modeling as Next-Token Prediction

Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models

Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework