多模态大模型 - arXivDaily 专题

2501.10834 2025-01-22 cs.CV cs.AI cs.LG 84%

Visual RAG: Expanding MLLM visual knowledge without fine-tuning

Mirco Bonomo, Simone Bianco

专题命中多模态训练与对齐：MLLM(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

2401.02797 2025-01-17 cs.CL cs.AI 84%

PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging

Jinlong He, Pengfei Li, Gang Liu, Genrong He, Zhaolin Chen, Shenjun Zhong

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CL、cs.AI

Comments 12 pages, 8 figures, 12 tables

2501.02778 2025-01-07 eess.IV cs.AI cs.CV 84%

ICFNet: Integrated Cross-modal Fusion Network for Survival Prediction

Binyu Zhang, Zhu Meng, Junhao Dong, Fei Su, Zhicheng Zhao

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

2501.02268 2025-01-07 cs.CV cs.AI 84%

What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph

Yutao Jiang, Qiong Wu, Wenhao Lin, Wei Yu, Yiyi Zhou

专题命中多模态训练与对齐：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 6 figures

2501.01535 2025-01-06 cs.HC cs.AI cs.CL cs.CY 84%

A Metasemantic-Metapragmatic Framework for Taxonomizing Multimodal Communicative Alignment

Eugene Yu Ji

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments 34 pages, 1 figure, 3 tables. Draft presented at 2023 ZJU Logic and AI Summit EAI Workshop

2412.05243 2024-12-09 cs.CV cs.AI cs.LG 84%

CompCap: Improving Multimodal Large Language Models with Composite Captions

Xiaohui Chen, Satya Narayan Shukla, Mahmoud Azab, Aashu Singh, Qifan Wang, David Yang, ShengYun Peng, Hanchao Yu, Shen Yan, Xuewen Zhang, Baosheng He

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

2408.14471 2024-12-09 cs.CV cs.CL cs.LG 84%

A Practitioner's Guide to Continual Multimodal Pretraining

Karsten Roth, Vishaal Udandarao, Sebastian Dziadzio, Ameya Prabhu, Mehdi Cherti, Oriol Vinyals, Olivier Hénaff, Samuel Albanie, Matthias Bethge, Zeynep Akata

专题命中多模态训练与对齐：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.CL

Comments Technical Report. 52 pages. Shorter version published at the NeurIPS 2024 Dataset & Benchmarks track

2412.01289 2024-12-05 cs.CV cs.AI 84%

Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion

Zhuokun Chen, Jinwu Hu, Zeshuai Deng, Yufeng Wang, Bohan Zhuang, Mingkui Tan

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

2410.07278 2024-12-03 cs.CV cs.AI 84%

PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Models

Yingen Liu, Fan Wu, Ruihui Li, Zhuo Tang, Kenli Li

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 5 figures,3 tables

2411.15099 2024-11-25 cs.CV cs.CL cs.LG 84%

Context-Aware Multimodal Pretraining

Karsten Roth, Zeynep Akata, Dima Damen, Ivana Balažević, Olivier J. Hénaff

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

2411.10937 2024-11-19 cs.CV cs.CL 84%

Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquiry

Wenjun Hou, Yi Cheng, Kaishuai Xu, Yan Hu, Wenjie Li, Jiang Liu

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

2411.10928 2024-11-19 cs.CL cs.AI 84%

Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning

Wenke Huang, Jian Liang, Zekun Shi, Didi Zhu, Guancheng Wan, He Li, Bo Du, Dacheng Tao, Mang Ye

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

2306.17799 2024-11-18 cs.CV cs.SD eess.AS 84%

A Low-rank Matching Attention based Cross-modal Feature Fusion Method for Conversational Emotion Recognition

Yuntao Shou, Huan Liu, Xiangyong Cao, Deyu Meng, Bo Dong

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments 13 pages, 5 figures

2407.12315 2024-10-29 cs.CV cs.AI cs.HC cs.IR 84%

ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map

Yilin Ye, Shishi Xiao, Xingchen Zeng, Wei Zeng

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments VIS 2024

2407.00087 2024-10-07 cs.AI cs.CL cs.LG 84%

ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback

Ju-Seung Byun, Jiyun Chun, Jihyung Kil, Andrew Perrault

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2024

2409.20566 2024-10-01 cs.CV cs.CL cs.LG 84%

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yang, Aleksei Timofeev, Mingze Xu, Hong-You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, Zirui Wang, Afshin Dehghan, Peter Grasch, Yinfei Yang

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

2402.06264 2024-09-19 cs.AI cs.CL cs.SI 84%

LLaVA-Docent: Instruction Tuning with Multimodal Large Language Model to Support Art Appreciation Education

Unggi Lee, Minji Jeon, Yunseo Lee, Gyuri Byun, Yoorim Son, Jaeyoon Shin, Hongkyu Ko, Hyeoncheol Kim

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

Comments 37 pages, 4 figures, 10 tables

2407.09050 2024-09-09 cs.CR cs.AI cs.CV cs.LG 84%

Refusing Safe Prompts for Multi-modal Large Language Models

Zedian Shao, Hongbin Liu, Yuepeng Hu, Neil Zhenqiang Gong

专题命中多模态训练与对齐：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

2408.13661 2024-08-27 cs.CV cs.AI cs.LG 84%

Hierarchical Network Fusion for Multi-Modal Electron Micrograph Representation Learning with Foundational Large Language Models

Sakhinana Sagar Srinivas, Geethan Sannidhi, Venkataramana Runkana

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Our paper is published at the workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models at NeurIPS 2023

2408.11039 2024-08-21 cs.AI cs.CV 84%

Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, Omer Levy

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 23 pages

2405.18523 2024-08-20 cs.CV cs.AI 84%

MM-Mixing: Multi-Modal Mixing Alignment for 3D Understanding

Jiaze Wang, Yi Wang, Ziyu Guo, Renrui Zhang, Donghao Zhou, Guangyong Chen, Anfeng Liu, Pheng-Ann Heng

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

2408.08527 2024-08-19 cs.CV cs.AI 84%

Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Grading

Li Pan, Yupei Zhang, Qiushi Yang, Tan Li, Xiaohan Xing, Maximus C. F. Yeung, Zhen Chen

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

2408.07080 2024-08-15 cs.LG cs.AI cs.CV 84%

DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learning

Dino Ienco, Cassio Fraga Dantas

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Journal ref British Machine Vision Conference, Nov 2024, Glasgow, United Kingdom

2407.20174 2024-08-13 cs.CV cs.AI 84%

Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning

Xingchen Zeng, Haichuan Lin, Yilin Ye, Wei Zeng

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 11 pages, 7 figures

2408.00290 2024-08-02 cs.CV cs.AI 84%

Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

Bin Cheng, Jiaxuan Lu

专题命中多模态训练与对齐：multi-modal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

2312.11816 2024-08-02 cs.AI cs.CV 84%

A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking

Shezheng Song, Shan Zhao, Chengyu Wang, Tianwei Yan, Shasha Li, Xiaoguang Mao, Meng Wang

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments AAAI23 Accept

2407.12825 2024-07-19 cs.CL cs.AI cs.LG 84%

A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention

Shengjie Li, Yinhao Xiao

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CL、cs.AI

2407.12580 2024-07-18 cs.CL cs.CV cs.IR 84%

E5-V: Universal Embeddings with Multimodal Large Language Models

Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Code and models are available at https://github.com/kongds/E5-V

2406.18546 2024-06-28 cs.CV cs.AI 84%

Application of Multimodal Fusion Deep Learning Model in Disease Recognition

Xiaoyi Liu, Hongjie Qiu, Muqing Li, Zhou Yu, Yutian Yang, Yafeng Yan

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

2402.13561 2024-06-27 cs.CL cs.CV 84%

Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment

Yunxin Li, Xinyu Chen, Baotian Hu, Haoyuan Shi, Min Zhang

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 12 pages,4 figures; Accepted by ACL 2024 Main Conference

1. 多模态训练与对齐 6856 篇

Visual RAG: Expanding MLLM visual knowledge without fine-tuning

PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging

ICFNet: Integrated Cross-modal Fusion Network for Survival Prediction

What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph

A Metasemantic-Metapragmatic Framework for Taxonomizing Multimodal Communicative Alignment

CompCap: Improving Multimodal Large Language Models with Composite Captions

A Practitioner's Guide to Continual Multimodal Pretraining

Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion

PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Models

Context-Aware Multimodal Pretraining

Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquiry

Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning

A Low-rank Matching Attention based Cross-modal Feature Fusion Method for Conversational Emotion Recognition

ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map

ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

LLaVA-Docent: Instruction Tuning with Multimodal Large Language Model to Support Art Appreciation Education

Refusing Safe Prompts for Multi-modal Large Language Models

Hierarchical Network Fusion for Multi-Modal Electron Micrograph Representation Learning with Foundational Large Language Models

Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

MM-Mixing: Multi-Modal Mixing Alignment for 3D Understanding

Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Grading

DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learning

Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning

Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking

A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention

E5-V: Universal Embeddings with Multimodal Large Language Models

Application of Multimodal Fusion Deep Learning Model in Disease Recognition

Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment