大模型对齐与安全

Learning to be Safe: Deep RL with a Safety Critic

Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, Chelsea Finn

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments In submission, 16 pages (including appendix)

L2B: Learning to Balance the Safety-Efficiency Trade-off in Interactive Crowd-aware Robot Navigation

Mai Nishimura, Ryo Yonetani

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments Accepted at IROS2020. Project site: https://denkiwakame.github.io/l2b/

Assessing Safety-Critical Systems from Operational Testing: A Study on Autonomous Vehicles

Xingyu Zhao, Kizito Salako, Lorenzo Strigini, Valentin Robu, David Flynn

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Comments Accepted by Information and Software Technology. arXiv admin note: substantial text overlap with arXiv:1908.06540

SMT-based Safety Verification of Parameterised Multi-Agent Systems

Paolo Felli, Alessandro Gianola, Marco Montali

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Intelligent Reputation System for Safety Messages in VANET

Ghassan Samara

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Comments 9 pages

Journal ref IAES International Journal of Artificial Intelligence (IJ-AI) Vol. 9, No. 3, September 2020, pp. 439~447, ISSN: 2252-8938, DOI: 10.11591/ijai.v9.i3.pp439-447

Learning for Dose Allocation in Adaptive Clinical Trials with Safety Constraints

Cong Shen, Zhiyang Wang, Sofia S. Villar, Mihaela van der Schaar

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments Accepted to the 37th International Conference on Machine Learning (ICML 2020)

Using Computer Vision to enhance Safety of Workforce in Manufacturing in a Post COVID World

Prateek Khandelwal, Anuj Khandelwal, Snigdha Agarwal, Deep Thomas, Naveen Xavier, Arun Raghuraman

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments 6 pages, 7 figure, 1 table

Learning for Safety-Critical Control with Control Barrier Functions

Andrew Taylor, Andrew Singletary, Yisong Yue, Aaron Ames

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments Extended version (12 Pages), Short version submitted to Learning for Dynamics & Control (L4DC) 2020 Conference

Practical Solutions for Machine Learning Safety in Autonomous Vehicles

Sina Mohseni, Mandar Pitale, Vasu Singh, Zhangyang Wang

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments Accepted at Safe AI workshop at AAAI 2020

Improving Safety in Reinforcement Learning Using Model-Based Architectures and Human Intervention

Bharat Prakash, Mohit Khatwani, Nicholas Waytowich, Tinoosh Mohsenin

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Constructing Trustworthy and Safe Communities on a Blockchain-Enabled Social Credits System

Ronghua Xu, Xuheng Lin, Qi Dong, Yu Chen

专题命中 安全训练 ：trustworthy(title,abstract)；分类 cs.CY

Comments Position paper, accepted by The 1st Workshop on Distributed Ledger of Things (DLoT 2018), co-located with EAI Mobiquitous 2018. Nov. 5, 2018, New York, NY, USA

Safe Exploration in Markov Decision Processes with Time-Variant Safety using Spatio-Temporal Gaussian Process

Akifumi Wachi, Hiroshi Kajino, Asim Munawar

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Comments 12 pages, 7 figures

Safety-Aware Apprenticeship Learning

Weichao Zhou, Wenchao Li

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Comments Accepted by International Conference on Computer Aided Verification (CAV) 2018

On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products

Kush R. Varshney, Homa Alemzadeh

专题命中 安全训练 ：safety(title,abstract)；分类 cs.CY

Comments Big Data, 2017

Policy Prototyping for LLMs: Pluralistic Alignment via Interactive and Collaborative Policymaking

K. J. Kevin Feng, Inyoung Cheong, Quan Ze Chen, Amy X. Zhang

专题命中 安全训练 ：alignment(title,abstract)

Comments Bidirectional Human-AI Alignment (Bi-Align) Workshop @ ICLR 2025

Design and Development of a Roaming Wireless Safety Emergency Stop

Henry Beuster, Thomas Doebbert, Christoph Cammin, Dmytro Krush, Gerd Scholl

专题命中 安全训练 ：safety(title,abstract)

Comments 7 Pages, accepted for Safety of Industrial Automated Systems, SIAS 2024

Low-Agreeableness Persona Conditioning for Safe LLM Fine-Tuning

低宜人性人格条件化用于安全的大语言模型微调

Austin MY Cheung, Yi Yang

机构 * Hong Kong University of Science and Technology（香港科技大学） 纠错

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI

AI总结 针对温暖微调降低大语言模型安全性的问题，提出基于低宜人性人格的改写流水线，在保持对话温暖的同时降低越狱成功率与有害输出率。

Comments 9 pages, 8 tables, 5 figures

NP-Hard Lower Bound Complexity for Semantic Self-Verification

语义自验证的NP难下界复杂度

Robin Young

机构 * University of Cambridge（剑桥大学） 纠错

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；AI safety(abstract)；分类 cs.CL、cs.AI

AI总结 本文证明语义自验证问题在特定框架下为NP完全，通过将3SAT问题归约到SSV，揭示了即使简化形式的语义自验证也面临计算障碍，对AI安全和公平性方法有重要影响。

Comments EACL 2026

Persona Features Control Emergent Misalignment

Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A. Chi, Samuel Miserendino, Jeffrey Wang, Achyuta Rajaram, Johannes Heidecke, Tejal Patwardhan, Dan Mossing

机构 * OpenAI 纠错

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；AI safety(abstract)；分类 cs.AI、cs.LG

A Call to Action for a Secure-by-Design Generative AI Paradigm

Dalal Alharthi, Ivan Roberto Kawaminami Garcia

机构 * University of Arizona（亚利桑那大学） 纠错

专题命中 安全训练 ：safety(abstract)；prompt injection(abstract)；AI safety(abstract)；分类 cs.AI、cs.LG

A Framework for Inherently Safer AGI through Language-Mediated Active Inference

Bo Wen

机构 * IBM T.J. Watson Research Center（IBM T.J. Watson研究院） 纠错

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；AI safety(abstract)；分类 cs.AI、cs.LG

SEALGuard: Safeguarding the Multilingual Conversations in Southeast Asian Languages for LLM Software Systems

Wenliang Shan, Michael Fu, Rui Yang, Chakkrit Tantithamthavorn

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI

Report on NSF Workshop on Science of Safe AI

Rajeev Alur, Greg Durrett, Hadas Kress-Gazit, Corina Păsăreanu, René Vidal

机构 * University of Pennsylvania（宾夕法尼亚大学） ； NASA Ames and CMU（NASA阿姆斯特朗研究中心和卡内基梅隆大学） ； The University of Texas at Austin（德克萨斯大学奥斯汀分校） ； Cornell University（康奈尔大学） ； Penn（宾夕法尼亚大学） ； UT Austin（德克萨斯大学奥斯汀分校） ； CMU（卡内基梅隆大学） 纠错

专题命中 安全训练 ：safety(abstract)；trustworthy(abstract)；AI safety(abstract)；分类 cs.AI、cs.CY

Learning Natural Language Constraints for Safe Reinforcement Learning of Language Agents

Jaymari Chua, Chen Wang, Lina Yao

专题命中 安全训练 ：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?

Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt MacDermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oliver Richardson, Marc-Antoine Rondeau, Pierre-Luc St-Charles, David Williams-King

专题命中 安全训练 ：safety(abstract)；trustworthy(abstract)；AI safety(abstract)；分类 cs.AI、cs.LG

Comments v2 with fixed formatting for URLs and hyperlinks

PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routing

Blazej Manczak, Eliott Zemour, Eric Lin, Vaikkunth Mugunthan

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI

Comments ICML 2024 NextGenAISafety workshop version with links to implementation and dataset

Learn to Disguise: Avoid Refusal Responses in LLM's Defense via a Multi-agent Attacker-Disguiser Game

Qianqiao Xu, Zhiliang Tian, Hongyan Wu, Zhen Huang, Yiping Song, Feng Liu, Dongsheng Li

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI

Comments 13 pages, 2 figures

OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research

Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, Yaodong Yang

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；AI safety(abstract)；分类 cs.AI、cs.LG

An overview of 11 proposals for building safe advanced AI

Evan Hubinger

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；AI safety(abstract)；分类 cs.AI、cs.LG

Reliability-Safety Trade-off in AI Distillation: A Renormalization-Group Approach

AI知识蒸馏中的可靠性-安全性权衡：一种重整化群方法

Y. M. Du, Miao-Miao Yi, Tan-Ji Zhou, C. P. Sun

专题命中 安全训练 ：safety(title,abstract)

AI总结 该研究基于统计力学构建知识蒸馏的粗粒化模型，以参数K表征危险辨别能力，揭示了AI蒸馏中可靠性与安全性的权衡关系，为多代蒸馏提供了可检验的标度预测。