多模态大模型

Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reports

Changde Du, Yizhuo Lu, Zhongyu Huang, Yi Sun, Zisen Zhou, Shaozheng Qin, Huiguang He

机构 * State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences（脑认知与脑启发智能技术重点实验室，自动化研究所，中国科学院） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学） ； School of Future Technology, University of Chinese Academy of Sciences（未来技术学院，中国科学院大学） ； State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University（认知神经科学与学习国家重点实验室，北京师范大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI、cs.MM

ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model

Weitai Kang, Weiming Zhuang, Zhizhong Li, Yan Yan, Lingjuan Lyu

机构 * University of Illinois Chicago（伊利诺伊大学香槟分校） ； Sony AI（索尼人工智能） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 8 pages for the main paper

Few-shot Vision-based Human Activity Recognition with MLLM-based Visual Reinforcement Learning

Wenqi Zheng, Yutaka Arakawa

机构 * Graduate School（研究生院） ； Faculty of Information Science（信息科学系） ； Electrical Engineering（电气工程） ； Kyushu University（九州大学） ； JAPAN（日本） 纠错

专题命中 多模态训练与对齐 ：MLLM(title,abstract)；multimodal(abstract)；multi-modal(abstract)

Analyzing Finetuning Representation Shift for Multimodal LLMs Steering

Pegah Khayatan, Mustafa Shukor, Jayneel Parekh, Arnaud Dapogny, Matthieu Cord

机构 * ISIR, Sorbonne Université（ISIR，索邦大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICCV 2025. The first three authors contributed equally. Project page and code: https://pegah- kh.github.io/projects/lmm-finetuning-analysis-and-steering/

On Pre-training of Multimodal Language Models Customized for Chart Understanding

Wan-Cyuan Fan, Yen-Chun Chen, Mengchen Liu, Lu Yuan, Leonid Sigal

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2024 Workshop on Adaptive Foundation Models

Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation

Yupu Liang, Yaping Zhang, Zhiyang Zhang, Yang Zhao, Lu Xiang, Chengqing Zong, Yu Zhou

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by ACL 2025 Main

Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach

Timo Fudala, Vasileios Tsouvalas, Nirvana Meratnia

机构 * Eindhoven University of Technology（埃因霍温理工大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；image-text(abstract)；audio-visual(abstract)

Comments 10 pages, 5 figures, accepted to FedGenAI-IJCAI 2025

Gated Recursive Fusion: A Stateful Approach to Scalable Multimodal Transformers

Yusuf Shihata

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 13 pages, 2 figures

MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension

Ting Liu, Zunnan Xu, Yue Hu, Liangtao Shi, Zhiqiang Wang, Quanjun Yin

机构 * College of Systems Engineering, National University of Defense Technology（系统工程学院，国防科技大学） ； Tsinghua University（清华大学） ； Hefei University of Technology（合肥工业大学） ； iFLYTEK Research（iFLYTEK研究院） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2024 main

VITA: Towards Open-Source Interactive Omni Multimodal LLM

Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, Haoyu Cao, Di Yin, Long Ma, Xiawu Zheng, Rongrong Ji, Yunsheng Wu, Ran He, Caifeng Shan, Xing Sun

机构 * NJU（南京大学） ； Tencent Youtu Lab（腾讯计算机系统有限公司视觉实验室） ； XMU（厦门大学） ； CASIA（中国科学院自动化研究所） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://vita-home.github.io

Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models

Jiachen Jiang, Jinxin Zhou, Bo Peng, Xia Ning, Zhihui Zhu

机构 * Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学） ； Translational Data Analytics Institute, The Ohio State University（转化数据分析研究所，俄亥俄州立大学） ； Department of Biomedical Informatics, The Ohio State University（生物医学信息学系，俄亥俄州立大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Enhancing Cross-Modal Contextual Congruence for Crowdfunding Success using Knowledge-infused Learning

Trilok Padhi, Ugur Kursuncu, Yaman Kumar, Valerie L. Shalin, Lane Peterson Fronczek

机构 * Georgia State University（佐治亚州立大学） ； Adobe MDSR ； Wright State University（怀特州立大学） ； California Polytechnic State University（加州州立大学帕克校区） 纠错

专题命中 多模态训练与对齐 ：cross-modal(title)；multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at IEEE International Conference on Big Data 2024 (IEEE BigData 2024)

Journal ref IEEE International Conference on Big Data 2024 (IEEE BigData 2024)

MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning

Ke Wang, Junting Pan, Linda Wei, Aojun Zhou, Weikang Shi, Zimu Lu, Han Xiao, Yunqiao Yang, Houxing Ren, Mingjie Zhan, Hongsheng Li

机构 * Multimedia Laboratory (MMLab)（多媒体实验室） ； The Chinese University of Hong Kong（香港中文大学） ； CPII under InnoHK（创新香港科技署CPII） 纠错

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ACL 2025 Findings

Token Sequence Compression for Efficient Multimodal Computing

Yasmine Omri, Parth Shroff, Thierry Tambe

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

CleanMAP: Distilling Multimodal LLMs for Confidence-Driven Crowdsourced HD Map Updates

Ankit Kumar Shaw, Kun Jiang, Tuopu Wen, Chandan Kumar Sah, Yining Shi, Mengmeng Yang, Diange Yang, Xiaoli Lian

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Kun Jiang, Mengmeng Yang and Diange Yang are Corresponding Author. The main paper and supplementary material are both included here, total 23 pages (main paper is 10 pages and supplementary material is 13 pages), total 17 figures (6 figures in main paper and 11 figures in supplementary material), this paper is Accepted to CVPR WDFM-AD Workshop 2025, The code will be available at https://Ankit-Zefan.github.io/CleanMap/

Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?

Yanbo Wang, Jiyang Guan, Jian Liang, Ran He

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to CVPR 2025, codes in process

List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs

An Yan, Zhengyuan Yang, Junda Wu, Wanrong Zhu, Jianwei Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Julian McAuley, Jianfeng Gao, Lijuan Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments published at COLM-2024

Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models

Weihao Ye, Qiong Wu, Wenhao Lin, Yiyi Zhou

专题命中 多模态训练与对齐 ：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.MM

A Concept-Based Explainability Framework for Large Multimodal Models

Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, Matthieu Cord

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2024

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

Qifei Li, Yingming Gao, Yuhua Wen, Cong Wang, Ya Li

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments The paper has been accepted by INTERSPEECH 2024

Enhancing Emotion Recognition in Incomplete Data: A Novel Cross-Modal Alignment, Reconstruction, and Refinement Framework

Haoqin Sun, Shiwan Zhao, Shaokai Li, Xiangyu Kong, Xuechen Wang, Aobo Kong, Jiaming Zhou, Yong Chen, Wenjia Zeng, Yong Qin

专题命中 多模态训练与对齐 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Ovis: Structural Embedding Alignment for Multimodal Large Language Model

Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Han-Jia Ye

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Explore the Limits of Omni-modal Pretraining at Scale

Yiyuan Zhang, Handong Li, Jing Liu, Xiangyu Yue

专题命中 多模态训练与对齐 ：omni-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Project Website: https://invictus717.github.io/MiCo/

Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?

Xiujun Li, Yujie Lu, Zhe Gan, Jianfeng Gao, William Yang Wang, Yejin Choi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Github: https://github.com/VIM-Bench/VIM_TOOL, Model and Data: https://huggingface.co/VIM-Bench

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

Ziyan Yao, Fei Lin, Sheng Chai, Weijie He, Lu Dai, Xinghui Fei

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Honeybee: Locality-enhanced Projector for Multimodal LLM

Junbum Cha, Wooyoung Kang, Jonghwan Mun, Byungseok Roh

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2024 camera-ready

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, Zongqing Lu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 14 pages, 2 figures, 11 tables

More than Vanilla Fusion: a Simple, Decoupling-free, Attention Module for Multimodal Fusion Based on Signal Theory

Peiwen Sun, Yifan Zhang, Zishan Liu, Donghao Chen, Honggang Zhang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.AI、cs.MM、eess.AS

Multimodal Document Analytics for Banking Process Automation

Christopher Gerling, Stefan Lessmann

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments A Preprint

MIMIC-IT: Multi-Modal In-Context Instruction Tuning

Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, Ziwei Liu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project page: https://otter-ntu.github.io/ Dataset & code: https://github.com/Luodian/otter Initial release, work in progress