多模态大模型

Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques

Jisu An, Junseok Lee, Jeoungeun Lee, Yongseok Son

机构 * Seoul National University（首尔国立大学） ； University of California San Diego（加州大学圣地亚哥分校） ； Chung-Ang University（Chung-Ang 大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 18 pages, 3 figures, 3 tables

DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models

Jianyu Liu, Hangyu Guo, Ranjie Duan, Xingyuan Bu, Yancheng He, Shilong Li, Hui Huang, Jiaheng Liu, Yucheng Wang, Chenchen Jing, Xingwei Qu, Xiao Zhang, Yingshui Tan, Yanan Wu, Jihao Gu, Yangguang Li, Jianke Zhu

机构 * Alibaba Group（阿里巴巴集团） ； Zhejiang University（浙江大学） ； M-A-P ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments [NAACL 2025] The first four authors contribute equally, 23 pages, repo at https://github.com/Kizna1ver/DREAM

SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representation

Thomas Pickard, Aline Villavicencio, Maggie Mi, Wei He, Dylan Phelps, Marco Idiart

机构 * University of Sheffield, UK（谢菲尔德大学） ； University of Exeter, UK（埃克塞特大学） ； Federal University of Rio Grande do Sul, Brazil（里约格朗德杜斯尔大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Author accepted version; SemEval-2025 proceedings to appear at ACL 2025. This version corrects a typo in the results table

MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale

Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, Xiang Yue

机构 * Carnegie Mellon University（卡内基梅隆大学） ； M-A-P ； Nanyang Technological University（南洋理工大学） ； University of Waterloo（滑铁卢大学） ； The University of Manchester（曼彻斯特大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments ACL 2025 Main

Brain-Adapter: Enhancing Neurological Disorder Analysis with Adapter-Tuning Multimodal Large Language Models

Jing Zhang, Xiaowei Yu, Yanjun Lyu, Lu Zhang, Tong Chen, Chao Cao, Yan Zhuang, Minheng Chen, Tianming Liu, Dajiang Zhu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Test-Time Immunization: A Universal Defense Framework Against Jailbreaks for (Multimodal) Large Language Models

Yongcan Yu, Yanbo Wang, Ran He, Jian Liang

机构 * NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences（国家工程实验室与人工智能研究所，中国科学院） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Under Review

Generative RLHF-V: Learning Principles from Multi-modal Human Preference

Jiayi Zhou, Jiaming Ji, Boyuan Chen, Jiapeng Sun, Wenqi Chen, Donghai Hong, Sirui Han, Yike Guo, Yaodong Yang

机构 * Peking University（北京大学） ； Hong Kong University of Science and Technology（香港科技大学） ； University College London（伦敦大学学院） 纠错

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 8 figures

Enhanced Multimodal Aspect-Based Sentiment Analysis by LLM-Generated Rationales

Jun Cao, Jiyi Li, Ziwei Yang, Renjie Zhou

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 2 figures, 6 tables. Accepted by ICONIP2024

Discovering Pathology Rationale and Token Allocation for Efficient Multimodal Pathology Reasoning

Zhe Xu, Cheng Jin, Yihui Wang, Ziyi Liu, Hao Chen

机构 * Department of Computer Science Engineering（计算机科学与工程系） ； HKUST（香港科技大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Picturized and Recited with Dialects: A Multimodal Chinese Representation Framework for Sentiment Analysis of Classical Chinese Poetry

Xiaocong Du, Haoyu Pei, Haipeng Zhang

机构 * ShanghaiTech University（上海科技大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

A Multi-modal Fusion Network for Terrain Perception Based on Illumination Aware

Rui Wang, Shichun Yang, Yuyi Chen, Zhuoyang Li, Zexiang Tong, Jianyi Xu, Jiayi Lu, Xinjie Feng, Yaoguang Cao

机构 * Department of Transportation Science and Engineering, Beihang University（北京航空航天大学交通运输科学与工程学院） ； Innovation Center of New Energy Vehicle Digital Supervision Technology and Application for State Market Regulation（国家新能源汽车数字监管技术及应用创新中心） ； Hangzhou International Innovation Institute, Beihang University（杭州国际创新研究院） ； State Key Lab of Intelligent Transportation System, Beihang University（智能交通系统国家重点实验室） 纠错

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.AI、cs.MM

Mitigating Modality Bias in Multi-modal Entity Alignment from a Causal Perspective

Taoyu Su, Jiawei Sheng, Duohe Ma, Xiaodong Li, Juwei Yue, Mengxiao Song, Yingkai Tang, Tingwen Liu

机构 * Institute of Information Engineering, Chinese Academy of Sciences（信息工程研究所，中国科学院） ； School of Cyber Security, University of Chinese Academy of Sciences（网络安全学院，中国科学院大学） 纠错

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.MM

Comments Accepted by SIGIR 2025, 11 pages, 10 figures, 4 tables,

TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment Analysis

Weize Quan, Yunfei Feng, Ming Zhou, Yunzhen Zhao, Tong Wang, Dong-Ming Yan

机构 * State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） ； School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院） ； Tencent（腾讯） ； School of Information Science and Technology, Donghua University（东华大学信息科学与技术学院） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

Multimodal and Multiview Deep Fusion for Autonomous Marine Navigation

Dimitrios Dagdilelis, Panagiotis Grigoriadis, Roberto Galeazzi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

UoR-NCL at SemEval-2025 Task 1: Using Generative LLMs and CLIP Models for Multilingual Multimodal Idiomaticity Representation

Thanet Markchom, Tong Wu, Liting Huang, Huizhi Liang

机构 * Department of Computer Science, University of Reading（阅读大学计算机科学系） ； School of Computing, Newcastle University（新castle大学计算机学院） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Learning Modality-Aware Representations: Adaptive Group-wise Interaction Network for Multimodal MRI Synthesis

Tao Song, Yicheng Wu, Minhao Hu, Xiangde Luo, Linda Wei, Guotai Wang, Yi Guo, Feng Xu, Shaoting Zhang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

A multi-scale vision transformer-based multimodal GeoAI model for mapping Arctic permafrost thaw

Wenwen Li, Chia-Yu Hsu, Sizhe Wang, Zhining Gu, Yili Yang, Brendan M. Rogers, Anna Liljedahl

机构 * School of Geographical Sciences and Urban Planning Arizona State University（地理科学与城市规划学院亚利桑那州立大学） ； Woodwell Climate Research Center（伍德沃德气候研究中心） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

4D Multimodal Co-attention Fusion Network with Latent Contrastive Alignment for Alzheimer's Diagnosis

Yuxiang Wei, Yanteng Zhang, Xi Xiao, Tianyang Wang, Xiao Wang, Vince D. Calhoun

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.MM

SDIGLM: Leveraging Large Language Models and Multi-Modal Chain of Thought for Structural Damage Identification

Yunkai Zhang, Shiyin Wei, Yong Huang, Yawu Su, Shanshan Lu, Hui Li

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

From Conceptual Data Models to Multimodal Representation

Peter Stockinger

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments in French language

DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis

Efthymios Georgiou, Vassilis Katsouros, Yannis Avrithis, Alexandros Potamianos

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Preprint

Leveraging Label Potential for Enhanced Multimodal Emotion Recognition

Xuechun Shao, Yinfeng Yu, Liejun Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI、eess.AS

Comments Main paper (8 pages). Accepted for publication by IJCNN 2025

Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots

Erfan Shayegani, G M Shahariar, Sara Abdali, Lei Yu, Nael Abu-Ghazaleh, Yue Dong

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models

Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, Yong Zhang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPR 2025

A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model

Yingxue Xu, Yihui Wang, Fengtao Zhou, Jiabo Ma, Cheng Jin, Shu Yang, Jinbang Li, Zhengyu Zhang, Chenglong Zhao, Huajun Zhou, Zhenhui Li, Huangjing Lin, Xin Wang, Jiguang Wang, Anjia Han, Ronald Cheong Kin Chan, Li Liang, Xiuming Zhang, Hao Chen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 62 pages

Distilled Prompt Learning for Incomplete Multimodal Survival Prediction

Yingxue Xu, Fengtao Zhou, Chenyu Zhao, Yihui Wang, Can Yang, Hao Chen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR2025

Babel: A Scalable Pre-trained Model for Multi-Modal Sensing via Expandable Modality Alignment

Shenghong Dai, Shiqi Jiang, Yifan Yang, Ting Cao, Mo Li, Suman Banerjee, Lili Qiu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted by SenSys'25

A Multimodal Fusion Model Leveraging MLP Mixer and Handcrafted Features-based Deep Learning Networks for Facial Palsy Detection

Heng Yim Nicole Oo, Min Hun Lee, Jeong Hoon Lim

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments PAKDD 2025. arXiv admin note: text overlap with arXiv:2405.16496

Exploring a Multimodal Fusion-based Deep Learning Network for Detecting Facial Palsy

Heng Yim Nicole Oo, Min Hun Lee, Jeong Hoon Lim

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments IJCAI 2024 4th AI for Ageless Aging Workshop (AIAA)

Enhancing Sentiment Analysis through Multimodal Fusion: A BERT-DINOv2 Approach

Taoxu Zhao, Meisi Li, Kehao Chen, Liye Wang, Xucheng Zhou, Kunal Chaturvedi, Mukesh Prasad, Ali Anaissi, Ali Braytee

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 12 pages