大模型对齐与安全

Improving Socratic Question Generation using Data Augmentation and Preference Optimization

Nischal Ashok Kumar, Andrew Lan

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.CY、cs.LG

Comments Published at the 19th BEA Workshop co-located with NAACL-2024

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, Tengyang Xie

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models

Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, Muhao Chen

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NAACL 2024

Advancing LLM Reasoning Generalists with Preference Trees

Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, Maosong Sun

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Models and data are available at https://github.com/OpenBMB/Eurus

Fine-Tuning Language Models with Reward Learning on Policy

Hao Lang, Fei Huang, Yongbin Li

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NAACL2024 Main Track Long Paper

Supervised Fine-Tuning as Inverse Reinforcement Learning

Hao Sun

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

$\mathbf{(N,K)}$-Puzzle: A Cost-Efficient Testbed for Benchmarking Reinforcement Learning Algorithms in Generative Language Model

Yufeng Zhang, Liyu Chen, Boyi Liu, Yingxiang Yang, Qiwen Cui, Yunzhe Tao, Hongxia Yang

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 8 pages

MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback

Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, Heng Ji

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2024. Code is available on our project website: https://xingyaoww.github.io/mint-bench

SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data

Jialu Li, Jaemin Cho, Yi-Lin Sung, Jaehong Yoon, Mohit Bansal

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments First two authors contributed equally; Project website: https://selma-t2i.github.io/

CogBench: a large language model walks into a psychology lab

Julian Coda-Forno, Marcel Binz, Jane X. Wang, Eric Schulz

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

RLVF: Learning from Verbal Feedback without Overgeneralization

Moritz Stephan, Alexander Khazatsky, Eric Mitchell, Annie S Chen, Sheryl Hsu, Archit Sharma, Chelsea Finn

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 9 pages, 9 figures

DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, Yuheng Zou

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs

Yann Hicke, Anmol Agarwal, Qianou Ma, Paul Denny

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Updates for camera-ready submission

Journal ref NeurIPS Workshop on Generative AI for Education (GAIED), 2023

Language Models can Solve Computer Tasks

Geunwoo Kim, Pierre Baldi, Stephen McAleer

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

Joey Hong, Sergey Levine, Anca Dragan

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 25 pages, 6 figures

Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery

Katie Z Luo, Zhenzhen Liu, Xiangyu Chen, Yurong You, Sagie Benaim, Cheng Perng Phoo, Mark Campbell, Wen Sun, Bharath Hariharan, Kilian Q. Weinberger

专题命中 偏好对齐 ：alignment(abstract)；RLHF(abstract)

Aligning Large Language Models through Synthetic Feedback

Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, Minjoon Seo

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to EMNLP 2023 main conference

KwaiYiiMath: Technical Report

Jiayi Fu, Lei Lin, Xiaoyang Gao, Pengli Liu, Zhengzong Chen, Zhirui Yang, Shengnan Zhang, Xue Zheng, Yan Li, Yuliang Liu, Xucheng Ye, Yiqiao Liao, Chao Liao, Bin Chen, Chengru Song, Junchen Wan, Zijia Lin, Fuzheng Zhang, Zhongyuan Wang, Di Zhang, Kun Gai

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments technical report. arXiv admin note: text overlap with arXiv:2306.16636 by other authors

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, Dylan Hadfield-Menell

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations

Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, Kathleen McKeown

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

ILLUME: Rationalizing Vision-Language Models through Human Interactions

Manuel Brack, Patrick Schramowski, Björn Deiseroth, Kristian Kersting

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Proceedings of the 40th International Conference on Machine Learning (ICML), 2023

Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, Tushar Khot

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint. Code at https://github.com/FranxYao/chain-of-thought-hub

DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents

Varun Nair, Elliot Schumacher, Geoffrey Tso, Anitha Kannan

专题命中 偏好对齐 ：safety(abstract)；分类 cs.CL、cs.AI、cs.LG

Discovering Language Model Behaviors with Model-Written Evaluations

Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments for associated data visualizations, see https://www.evals.anthropic.com/model-written/ for full datasets, see https://github.com/anthropics/evals

Learning Norms from Stories: A Prior for Value Aligned Agents

Spencer Frazier, Md Sultan Al Nahian, Mark Riedl, Brent Harrison

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments AIES 2020

Steerable Cultural Preference Optimization of Reward Models

可引导的文化偏好优化奖励模型

Minsik Oh, Advit Deepak, Sophie Wu, Douwe Kiela, Ekaterina Shutova

机构 * Stanford University（斯坦福大学） ； University of Amsterdam（阿姆斯特丹大学） 纠错

专题命中 偏好对齐 ：alignment(abstract,comments)；分类 cs.CL、cs.AI

AI总结 提出SCPO算法，通过平衡多种文化偏好训练奖励模型，在PRISM和GlobalOpinionQA数据集上提升少数群体偏好预测准确率最多7点，训练效率提高280%。

Comments Accepted to Pluralistic Alignment @ ICML 2026

Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedback

Vijay Keswani, Cyrus Cousins, Breanna Nguyen, Vincent Conitzer, Hoda Heidari, Jana Schaich Borg, Walter Sinnott-Armstrong

机构 * Duke University（杜克大学） ； Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 偏好对齐 ：alignment(abstract,comments)；分类 cs.AI、cs.CY

Comments To appear in the AAAI 2026 Alignment Track

Understanding Impact of Human Feedback via Influence Functions

Taywon Min, Haeone Lee, Yongchan Kwon, Kimin Lee

机构 * KAIST（韩国科学技术院） ； Columbia University（哥伦比亚大学） 纠错

专题命中 偏好对齐 ：RLHF(abstract,comments)；分类 cs.AI、cs.LG

Comments Accepted at ACL 2025, Source code: https://github.com/mintaywon/IF_RLHF

Journal ref Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics 63 (2025) 27471-27500

Multi-Task Reward Learning from Human Ratings

Mingkang Wu, Devin White, Evelyn Rose, Vernon Lawhern, Nicholas R Waytowich, Yongcan Cao

机构 * Department of Electrical and Computer Engineering, University of Texas at San Antonio, Texas, USA（电子与计算机工程系，德克萨斯州立大学圣安东尼奥分校） ； DEVCOM Army Research Lab, USA（陆军研究实验室） 纠错

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.AI、cs.LG；alignment(comments)

Comments Accepted to the workshop on Models of Human Feedback for AI Alignment at the 42nd International Conference on Machine Learning

Reliability Check: An Analysis of GPT-3's Response to Sensitive Topics and Prompt Wording

Aisha Khatun, Daniel G. Brown

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.LG；trustworthy(comments)

Comments Accepted in TrustNLP: Third Workshop on Trustworthy Natural Language Processing, co-located with ACL 2023