多模态大模型 - arXivDaily 专题

2409.16723 2024-09-27 cs.CV 83%

EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models

Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, Yu-Gang Jiang

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

2409.15936 2024-09-25 cs.CY cs.CV cs.HC 83%

DepMamba: Progressive Fusion Mamba for Multimodal Depression Detection

Jiaxin Ye, Junping Zhang, Hongming Shan

专题命中多模态训练与对齐：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2407.02392 2024-08-29 cs.CV 83%

TokenPacker: Efficient Visual Projector for Multimodal LLM

Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, Lei Zhang

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments 16 pages, Codes:https://github.com/CircleRadon/TokenPacker

2408.12867 2024-08-26 cs.CV 83%

Semantic Alignment for Multimodal Large Language Models

Tao Wu, Mengze Li, Jingyuan Chen, Wei Ji, Wang Lin, Jinyang Gao, Kun Kuang, Zhou Zhao, Fei Wu

专题命中多模态训练与对齐：multimodal(title)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by MM 2024

2408.10703 2024-08-21 cs.CV 83%

Large Language Models for Multimodal Deformable Image Registration

Mingrui Ma, Weijie Wang, Jie Ning, Jianfeng He, Nicu Sebe, Bruno Lepri

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2408.10086 2024-08-20 cs.AI 83%

ARMADA: Attribute-Based Multimodal Data Augmentation

Xiaomeng Jin, Jeonghwan Kim, Yu Zhou, Kuan-Hao Huang, Te-Lin Wu, Nanyun Peng, Heng Ji

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；分类 cs.AI

2408.05019 2024-08-12 cs.CV 83%

Instruction Tuning-free Visual Token Complement for Multimodal LLMs

Dongsheng Wang, Jiequan Cui, Miaoge Li, Wang Lin, Bo Chen, Hanwang Zhang

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted by ECCV2024 (20pages)

2408.01800 2024-08-06 cs.CV 83%

MiniCPM-V: A GPT-4V Level MLLM on Your Phone

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun

专题命中多模态训练与对齐：MLLM(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments preprint

2408.01003 2024-08-05 cs.AI 83%

Piculet: Specialized Models-Guided Hallucination Decrease for MultiModal Large Language Models

Kohou Wang, Xiang Liu, Zhaoxiang Liu, Kai Wang, Shiguo Lian

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.AI

Comments 14 pages, 5 figures

2407.19414 2024-07-30 cs.AI 83%

Appformer: A Novel Framework for Mobile App Usage Prediction Leveraging Progressive Multi-Modal Data Fusion and Feature Extraction

Chuike Sun, Junzhou Chen, Yue Zhao, Hao Han, Ruihai Jing, Guang Tan, Di Wu

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI

2407.18999 2024-07-30 cs.CV cs.LG 83%

Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models

Baao Xie, Qiuyu Chen, Yunnan Wang, Zequn Zhang, Xin Jin, Wenjun Zeng

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments 9 pages, 7 figures

2406.04727 2024-07-29 cs.LG cond-mat.soft cs.AI 83%

MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Prediction

Fanmeng Wang, Wentao Guo, Minjie Cheng, Shen Yuan, Hongteng Xu, Zhifeng Gao

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments Accepted by the 33rd ACM International Conference on Information and Knowledge Management (CIKM 2024)

2407.16714 2024-07-25 cs.LG cs.AI 83%

Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation

Tao Meng, Fuchen Zhang, Yuntao Shou, Hongen Shao, Wei Ai, Keqin Li

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments 15 pages, 9 figures

2407.16168 2024-07-24 cs.CL 83%

Progressively Modality Freezing for Multi-Modal Entity Alignment

Yani Huang, Xuefeng Zhang, Richong Zhang, Junfan Chen, Jaein Kim

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments 13pages, 8 figures, Accepted by ACL2024

2407.15237 2024-07-23 cs.CL 83%

Two eyes, Two views, and finally, One summary! Towards Multi-modal Multi-tasking Knowledge-Infused Medical Dialogue Summarization

Anisha Saha, Abhisek Tiwari, Sai Ruthvik, Sriparna Saha

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CL

2404.02059 2024-07-23 cs.IR cs.CV 83%

IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT

Junchen Fu, Xuri Ge, Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, Jie Wang, Joemon M. Jose

专题命中多模态训练与对齐：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV

Comments Accepted by SIGIR2024

2407.11895 2024-07-17 cs.CV 83%

OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces

Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Hengshuang Zhao, Zhou Zhao

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Homepage is http://omnibind.github.io

2405.08786 2024-07-11 cs.CV 83%

Incorporating Clinical Guidelines through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring

Tiantian Zhang, Manxi Lin, Hongda Guo, Xiaofan Zhang, Ka Fung Peter Chiu, Aasa Feragen, Qi Dou

专题命中多模态训练与对齐：multi-modal(title,abstract)；MLLM(abstract)；分类 cs.CV

2407.06309 2024-07-10 cs.CY cs.AI 83%

Multimodal Chain-of-Thought Reasoning via ChatGPT to Protect Children from Age-Inappropriate Apps

Chuanbo Hu, Bin Liu, Minglei Yin, Yilu Zhou, Xin Li

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.AI

2407.05540 2024-07-09 cs.CV 83%

GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation

Chenxin Li, Xinyu Liu, Cheng Wang, Yifan Liu, Weihao Yu, Jing Shao, Yixuan Yuan

专题命中多模态训练与对齐：omni-modal(title)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ECCV2024

2312.00589 2024-07-04 cs.CV 83%

Merlin:Empowering Multimodal LLMs with Foresight Minds

En Yu, Liang Zhao, Yana Wei, Jinrong Yang, Dongming Wu, Lingyu Kong, Haoran Wei, Tiancai Wang, Zheng Ge, Xiangyu Zhang, Wenbing Tao

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted by ECCV2024. Project page: https://ahnsun.github.io/merlin

2406.17679 2024-06-26 cs.CV 83%

Local-to-Global Cross-Modal Attention-Aware Fusion for HSI-X Semantic Segmentation

Xuming Zhang, Naoto Yokoya, Xingfa Gu, Qingjiu Tian, Lorenzo Bruzzone

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

2310.13876 2024-06-19 cs.CV 83%

Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Images

Bissmella Bahaduri, Zuheng Ming, Fangchen Feng, Anissa Mokraou

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by ICIP2024

2406.09696 2024-06-17 eess.IV cs.CV 83%

MoME: Mixture of Multimodal Experts for Cancer Survival Prediction

Conghao Xiong, Hao Chen, Hao Zheng, Dong Wei, Yefeng Zheng, Joseph J. Y. Sung, Irwin King

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments 8 + 1/2 pages, early accepted to MICCAI2024

2406.05130 2024-06-10 cs.CL 83%

An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models

Xiongtao Zhou, Jie He, Yuhua Ke, Guangyao Zhu, Víctor Gutiérrez-Basulto, Jeff Z. Pan

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL

Comments ACL finding 2024

2406.02263 2024-06-05 cs.CV 83%

M3DM-NR: RGB-3D Noisy-Resistant Industrial Anomaly Detection via Multimodal Denoising

Chengjie Wang, Haokun Zhu, Jinlong Peng, Yue Wang, Ran Yi, Yunsheng Wu, Lizhuang Ma, Jiangning Zhang

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2406.01210 2024-06-05 cs.CV 83%

GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer

Ding Jia, Jianyuan Guo, Kai Han, Han Wu, Chao Zhang, Chang Xu, Xinghao Chen

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ICML 2024, code and models are available at https://github.com/JiaDingCN/GeminiFusion

2405.20985 2024-06-03 cs.CV 83%

DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models

Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, Lu Hou

专题命中多模态训练与对齐：multimodal(title)；MLLM(abstract)；cross-modal(abstract)；分类 cs.CV

2305.16556 2024-04-25 cs.LG cs.AI 83%

LANISTR: Multimodal Learning from Structured and Unstructured Data

Sayna Ebrahimi, Sercan O. Arik, Yihe Dong, Tomas Pfister

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

2309.04001 2024-04-22 cs.CV cs.LG 83%

MMSFormer: Multimodal Transformer for Material and Semantic Segmentation

Md Kaykobad Reza, Ashley Prater-Bennette, M. Salman Asif

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by IEEE Open Journal of Signal Processing. 15 pages, 3 figures, 9 tables

1. 多模态训练与对齐 6872 篇

EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models

DepMamba: Progressive Fusion Mamba for Multimodal Depression Detection

TokenPacker: Efficient Visual Projector for Multimodal LLM

Semantic Alignment for Multimodal Large Language Models

Large Language Models for Multimodal Deformable Image Registration

ARMADA: Attribute-Based Multimodal Data Augmentation

Instruction Tuning-free Visual Token Complement for Multimodal LLMs

MiniCPM-V: A GPT-4V Level MLLM on Your Phone

Piculet: Specialized Models-Guided Hallucination Decrease for MultiModal Large Language Models

Appformer: A Novel Framework for Mobile App Usage Prediction Leveraging Progressive Multi-Modal Data Fusion and Feature Extraction

Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models

MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Prediction

Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation

Progressively Modality Freezing for Multi-Modal Entity Alignment

Two eyes, Two views, and finally, One summary! Towards Multi-modal Multi-tasking Knowledge-Infused Medical Dialogue Summarization

IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT

OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces

Incorporating Clinical Guidelines through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring

Multimodal Chain-of-Thought Reasoning via ChatGPT to Protect Children from Age-Inappropriate Apps

GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation

Merlin:Empowering Multimodal LLMs with Foresight Minds

Local-to-Global Cross-Modal Attention-Aware Fusion for HSI-X Semantic Segmentation

Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Images

MoME: Mixture of Multimodal Experts for Cancer Survival Prediction

An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models

M3DM-NR: RGB-3D Noisy-Resistant Industrial Anomaly Detection via Multimodal Denoising

GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer

DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models

LANISTR: Multimodal Learning from Structured and Unstructured Data

MMSFormer: Multimodal Transformer for Material and Semantic Segmentation