语言大模型 / LLM

Clone-Robust AI Alignment

Ariel D. Procaccia, Benjamin Schiffer, Shirley Zhang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information

Rasul Tutnov, Antoine Grosnit, Haitham Bou-Ammar

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

COPR: Continual Human Preference Learning via Optimal Policy Regularization

Han Zhang, Lin Gui, Yu Lei, Yuanzhao Zhai, Yehong Zhang, Yulan He, Hui Wang, Yue Yu, Kam-Fai Wong, Bin Liang, Ruifeng Xu

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments This is a duplicate submission to arXiv:2310.15694, and we believe that this submission has affected the citation of our original paper arXiv:2310.15694

SafeWorld: Geo-Diverse Safety Alignment

Da Yin, Haoyi Qiu, Kung-Hsiang Huang, Kai-Wei Chang, Nanyun Peng

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

Comments Accepted by NeurIPS 2024

Unsupervised Human Preference Learning

Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Dilek Hakkani-Tür

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；foundation model(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024 Main Conference

Journal ref EMNLP 2024

Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignment

Ran Tian, Yilin Wu, Chenfeng Xu, Masayoshi Tomizuka, Jitendra Malik, Andrea Bajcsy

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments Submitted to IJRR, this paper is an extended journal version of the conference paper arXiv:2310.07932 with new results and discussion. arXiv admin note: substantial text overlap with arXiv:2310.07932

Pistis-RAG: Enhancing Retrieval-Augmented Generation with Human Feedback

Yu Bai, Yukai Miao, Li Chen, Dawei Wang, Dan Li, Yanyu Ren, Hongtao Xie, Ce Yang, Xuhui Cai

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs

Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, Tong Zhang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments NeurIPS 2024

Personalized Adaptation via In-Context Preference Learning

Allison Lau, Younwoo Choi, Vahid Balazadeh, Keertana Chidambaram, Vasilis Syrgkanis, Rahul G. Krishnan

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment

Zekun Moore Wang, Shawn Wang, Kang Zhu, Jiaheng Liu, Ke Xu, Jie Fu, Wangchunshu Zhou, Wenhao Huang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 28 pages

Anchored Alignment for Self-Explanations Enhancement

Luis Felipe Villa-Arenas, Ata Nizamoglu, Qianli Wang, Sebastian Möller, Vera Schmitt

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation

Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, Yang Liu

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Orchestrating LLMs with Different Personalizations

Jin Peng Zhou, Katie Z Luo, Jingwen Gu, Jason Yuan, Kilian Q. Weinberger, Wen Sun

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

WARP: On the Benefits of Weight Averaged Rewarded Policies

Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, Olivier Bachem

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments 11 main pages (34 pages with Appendix)

On the Transformations across Reward Model, Parameter Update, and In-Context Prompt

Deng Cai, Huayang Li, Tingchen Fu, Siheng Li, Weiwen Xu, Shuaiyi Li, Bowen Cao, Zhisong Zhang, Xinting Huang, Leyang Cui, Yan Wang, Lemao Liu, Taro Watanabe, Shuming Shi

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；prompting(abstract)；分类 cs.CL、cs.AI

Generalizing Reward Modeling for Out-of-Distribution Preference Learning

Chen Jia

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments 31 pages

LIRE: listwise reward enhancement for preference alignment

Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, Zhendong Mao

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments Accepted by ACL 2024 Findings

Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering

Hongyu Yang, Liyang He, Min Hou, Shuanghong Shen, Rui Li, Jiahui Hou, Jianhui Ma, Junda Zhao

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Prompt Optimization with Human Feedback

Xiaoqiang Lin, Zhongxiang Dai, Arun Verma, See-Kiong Ng, Patrick Jaillet, Bryan Kian Hsiang Low

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Preprint, 18 pages

Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents

San Kim, Gary Geunbae Lee

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 7 figures, accepted to NAACL findings 2024

Understanding the performance gap between online and offline alignment algorithms

Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, Will Dabney

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Contrastive Preference Learning: Learning from Human Feedback without RL

Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, Dorsa Sadigh

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments ICLR 2024. Code released at https://github.com/jhejna/cpl

Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles

Zhiwei Tang, Dmitry Rybin, Tsung-Hui Chang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments ICLR 2024

Evidence-Driven Retrieval Augmented Response Generation for Online Misinformation

Zhenrui Yue, Huimin Zeng, Yimeng Lu, Lanyu Shang, Yang Zhang, Dong Wang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments Accepted to NAACL 2024

Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections

Lihan Zha, Yuchen Cui, Li-Heng Lin, Minae Kwon, Montserrat Gonzalez Arenas, Andy Zeng, Fei Xia, Dorsa Sadigh

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments 8 pages, 4 figures, videos and code links on website https://sites.google.com/stanford.edu/droc

Investigating Bias Representations in Llama 2 Chat via Activation Steering

Dawn Lu, Nina Rimsky

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Reinforcement learning for question answering in programming domain using public community scoring as a human feedback

Alexey Gorbatovski, Sergey Kovalchuk

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Learning Optimal Advantage from Preferences and Mistaking it for Reward

W. Bradley Knox, Stephane Hatgis-Kessell, Sigurdur Orn Adalgeirsson, Serena Booth, Anca Dragan, Peter Stone, Scott Niekum

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments 8 pages (16 pages with references and appendix), 11 figures

A General Language Assistant as a Laboratory for Alignment

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Jared Kaplan

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments 26+19 pages; v2 typos fixed, refs added, figure scale / colors fixed; v3 correct very non-standard TruthfulQA formatting and metric, alignment implications slightly improved

COPA: Continual Preference Optimization for Adaptive Prompt Injection Defense

COPA：用于自适应提示注入防御的持续偏好优化

Roshan Sood, Onat Gungor, Tajana Rosing

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)

AI总结 该研究提出COPA框架，将提示注入防御视为终身学习问题，通过GRPO优化和边际加权经验回放实现持续防御，使攻击成功率最高降低6.3倍、平均降低4.4倍。