多模态大模型 - arXivDaily 专题

2501.16400 2025-01-29 eess.IV physics.med-ph 82%

CSF-Net: Cross-Modal Spatiotemporal Fusion Network for Pulmonary Nodule Malignancy Predicting

Yin Shen, Zhaojie Fang, Ke Zhuang, Guanyu Zhou, Xiao Yu, Yucheng Zhao, Yuan Tian, Ruiquan Ge, Changmiao Wang, Xiaopeng Fan, Ahmed Elazab

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)

Comments This paper has been accepted by the 2025 IEEE International Symposium on Biomedical Imaging

2407.21033 2025-01-28 cs.IR cs.AI cs.CL cs.CV 82%

Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition

Jielong Tang, Zhenxing Wang, Ziyang Gong, Jianxing Yu, Xiangwei Zhu, Jian Yin

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 13 pages, 7 figures

2501.11538 2025-01-22 cs.LG 82%

DenoMAE: A Multimodal Autoencoder for Denoising Modulation Signals

Atik Faysal, Taha Boushine, Mohammad Rostami, Reihaneh Gh. Roshan, Huaxia Wang, Nikhil Muralidhar, Avimanyu Sahoo, Yu-Dong Yao

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

2412.16243 2024-12-24 cs.LG 82%

Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data

Zhiqiang Tang, Zihan Zhong, Tong He, Gerald Friedland

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

2406.15334 2024-12-23 cs.CV cs.AI cs.CL cs.LG 82%

Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, Roei Herzig

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Published in NeurIPS 2024

2412.14536 2024-12-20 q-bio.NC 82%

Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis

Yu Zhu, Bo Lei, Chunfeng Song, Wanli Ouyang, Shan Yu, Tiejun Huang

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

Comments AAAI 2025

2412.04903 2024-12-17 cs.CV cs.AI cs.CL cs.LG 82%

EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation

Yongxin Wang, Meng Cao, Haokun Lin, Mingfei Han, Liang Ma, Jin Jiang, Yuhao Cheng, Xiaodan Liang

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 19 pages

2410.16081 2024-11-19 astro-ph.SR astro-ph.GA astro-ph.IM physics.comp-ph physics.data-an 82%

Deep Multimodal Representation Learning for Stellar Spectra

Tobias Buck, Christian Schwarz

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

Comments accepted to the Machine Learning and the Physical Sciences Workshop, NeurIPS 2024

2411.00029 2024-11-04 cs.CL cs.AI cs.CV cs.LG 82%

Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models

Donghoon Kim, Gusang Lee, Kyuhong Shim, Byonghyo Shim

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Findings of EMNLP 2024

2404.01595 2024-10-30 cs.LG stat.ME stat.ML 82%

Propensity Score Alignment of Unpaired Multimodal Data

Johnny Xi, Jana Osea, Zuheng Xu, Jason Hartford

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)

Comments NeurIPS 2024

2103.01296 2024-10-28 cs.RO 82%

Learning Multimodal Contact-Rich Skills from Demonstrations Without Reward Engineering

Mythra V. Balakuntala, Upinder Kaur, Xin Ma, Juan Wachs, Richard M. Voyles

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)

Comments Accepted at IEEE ICRA 2021. This work has been submitted to the IEEE for possible publication

2407.02623 2024-10-15 cs.CY cs.AI cs.CL cs.CV 82%

Uplifting Lower-Income Data: Strategies for Socioeconomic Perspective Shifts in Large Multi-modal Models

Joan Nwatu, Oana Ignat, Rada Mihalcea

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2410.04491 2024-10-08 cs.CL cs.AI cs.MM 82%

Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis

Xinyu Feng, Yuming Lin, Lihua He, You Li, Liang Chang, Ya Zhou

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted to EMNLP Findings 2024

2406.11839 2024-10-08 cs.CV cs.AI cs.CL cs.LG 82%

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, Muhao Chen

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to EMNLP 2024 Main Conference. Project website: https://feiwang96.github.io/mDPO

2008.06682 2024-10-08 eess.AS cs.AI cs.CL cs.SD 82%

Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

Shamane Siriwardhana, Andrew Reis, Rivindu Weerasekera, Suranga Nanayakkara

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to INTERSPEECH 2020

2407.10385 2024-10-01 cs.CL cs.AI cs.CV cs.LG 82%

By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting

Hyungjun Yoon, Biniyam Aschalew Tolera, Taesik Gong, Kimin Lee, Sung-Ju Lee

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024) Main

2406.05766 2024-09-24 cs.LG cs.AI cs.CL cs.CV 82%

Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution View

Zijia Song, Zelin Zang, Yelin Wang, Guozheng Yang, Kaicheng yu, Wanyu Chen, Miaoyu Wang, Stan Z. Li

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2408.12321 2024-08-27 cs.CL cs.CV cs.MM 82%

MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model

Chaoya Jiang, Jia Hongrui, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

2408.10500 2024-08-23 cs.MM cs.CV cs.SD eess.AS 82%

SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition

Zebang Cheng, Shuyuan Tu, Dawei Huang, Minghan Li, Xiaojiang Peng, Zhi-Qi Cheng, Alexander G. Hauptmann

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Ranked 1st in MER24@IJCAI and MRAC24@ACM MM (MER-NOISE & MER-OV (self-evaluated))

2408.09798 2024-08-20 cs.LG 82%

Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting

Yun-Da Tsai, Ting-Yu Yen, Keng-Te Liao, Shou-De Lin

专题命中多模态训练与对齐：multimodal(title,abstract)；multimodal foundation model(abstract)

Comments arXiv admin note: text overlap with arXiv:2407.05036

2407.21391 2024-08-01 cs.SD cs.CV cs.MM eess.AS 82%

Design and Development of Laughter Recognition System Based on Multimodal Fusion and Deep Learning

Fuzheng Zhao, Yu Bai

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 7 pages,2 figures

2405.17430 2024-07-30 cs.CV cs.AI cs.CL cs.LG 82%

Matryoshka Multimodal Models

Mu Cai, Jianwei Yang, Jianfeng Gao, Yong Jae Lee

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://matryoshka-mm.github.io/

2407.12438 2024-07-18 cs.LG 82%

Semantic-Aware Representation of Multi-Modal Data for Data Ingress: A Literature Review

Pierre Lamart, Yinan Yu, Christian Berger

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

Comments Accepted at the 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) 2024 as a short paper

2304.08658 2024-06-25 physics.app-ph cs.LG 82%

In-situ surface porosity prediction in DED (directed energy deposition) printed SS316L parts using multimodal sensor fusion

Adithyaa Karthikeyan, Himanshu Balhara, Abhishek Hanchate, Andreas K Lianos, Satish TS Bukkapatnam

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)

2406.06048 2024-06-17 cs.CV cs.AI cs.MM 82%

Robust Latent Representation Tuning for Image-text Classification

Hao Sun, Yu Song

专题命中多模态训练与对齐：image-text(title)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

2406.00017 2024-06-14 cs.CL cs.AI cs.MM 82%

PTA: Enhancing Multimodal Sentiment Analysis through Pipelined Prediction and Translation-based Alignment

Shezheng Song, Shasha Li, Shan Zhao, Chengyu Wang, Xiaopeng Li, Jie Yu, Qian Wan, Jun Ma, Tianwei Yan, Wentao Ma, Xiaoguang Mao

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Code will be released upon publication

2304.05600 2024-06-11 cs.SD cs.CV cs.LG cs.MM eess.AS 82%

Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning

Nikhil Singh, Chih-Wei Wu, Iroro Orife, Mahdi Kalayeh

专题命中多模态训练与对齐：cross-modal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to CVPR 2024

2404.04818 2024-04-11 cs.AI cs.CL cs.CV 82%

DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking

Shezheng Song, Shasha Li, Shan Zhao, Xiaopeng Li, Chengyu Wang, Jie Yu, Jun Ma, Tianwei Yan, Bin Ji, Xiaoguang Mao

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments under review on TOIS. arXiv admin note: substantial text overlap with arXiv:2312.11816

2404.03171 2024-04-05 cs.SE cs.LG cs.PL 82%

Multi-modal Learning for WebAssembly Reverse Engineering

Hanxian Huang, Jishen Zhao

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

Comments Accepted by ISSTA '24

2311.17076 2024-04-02 cs.CV cs.AI cs.CL cs.LG 82%

Compositional Chain-of-Thought Prompting for Large Multimodal Models

Chancharik Mitra, Brandon Huang, Trevor Darrell, Roei Herzig

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

1. 多模态训练与对齐 6872 篇

CSF-Net: Cross-Modal Spatiotemporal Fusion Network for Pulmonary Nodule Malignancy Predicting

Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition

DenoMAE: A Multimodal Autoencoder for Denoising Modulation Signals

Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data

Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis

EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation

Deep Multimodal Representation Learning for Stellar Spectra

Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models

Propensity Score Alignment of Unpaired Multimodal Data

Learning Multimodal Contact-Rich Skills from Demonstrations Without Reward Engineering

Uplifting Lower-Income Data: Strategies for Socioeconomic Perspective Shifts in Large Multi-modal Models

Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting

Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution View

MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model

SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition

Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting

Design and Development of Laughter Recognition System Based on Multimodal Fusion and Deep Learning

Matryoshka Multimodal Models

Semantic-Aware Representation of Multi-Modal Data for Data Ingress: A Literature Review

In-situ surface porosity prediction in DED (directed energy deposition) printed SS316L parts using multimodal sensor fusion

Robust Latent Representation Tuning for Image-text Classification

PTA: Enhancing Multimodal Sentiment Analysis through Pipelined Prediction and Translation-based Alignment

Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning

DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking

Multi-modal Learning for WebAssembly Reverse Engineering

Compositional Chain-of-Thought Prompting for Large Multimodal Models