多模态大模型

MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation

Jinpeng Wang, Ziyun Zeng, Yunxiao Wang, Yuting Wang, Xingyu Lu, Tianxiang Li, Jun Yuan, Rui Zhang, Hai-Tao Zheng, Shu-Tao Xia

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.AI、cs.MM

Comments Accepted to ACM MM 2023. Data and code are available

ConceptFusion: Open-set Multimodal 3D Mapping

Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, Antonio Torralba

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments RSS 2023. Project page: https://concept-fusion.github.io Explainer video: https://www.youtube.com/watch?v=rkXgws8fiDs Code: https://github.com/concept-fusion/concept-fusion

Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining

Emanuele Bugliarello, Aida Nematzadeh, Lisa Anne Hendricks

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2023

Universal Multi-modal Entity Alignment via Iteratively Fusing Modality Similarity Paths

Bolin Zhu, Xiaoze Liu, Xin Mao, Zhuo Chen, Lingbing Guo, Tao Gui, Qi Zhang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

DRIN: Dynamic Relation Interactive Network for Multimodal Entity Linking

Shangyu Xing, Fei Zhao, Zhen Wu, Chunhui Li, Jianbing Zhang, Xinyu Dai

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

Comments Accepted by ACM MM 2023

SeMAnD: Self-Supervised Anomaly Detection in Multimodal Geospatial Datasets

Daria Reshetova, Swetava Ganguli, C. V. Krishnakumar Iyer, Vipul Pandey

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Extended version of the accepted research track paper at the 31st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL 2023), Hamburg, Germany. 11 pages, 8 figures, 6 tables

Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection

Yichao Cao, Qingfei Tang, Feng Yang, Xiu Su, Shan You, Xiaobo Lu, Chang Xu

专题命中 多模态训练与对齐 ：cross-modal(title)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments ICCV2023

SDVRF: Sparse-to-Dense Voxel Region Fusion for Multi-modal 3D Object Detection

Binglu Ren, Jianqin Yin

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning

Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Fazel-Zarandi, Asli Celikyilmaz, Luke Zettlemoyer, Armen Aghajanyan

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Exchanging-based Multimodal Fusion with Transformer

Renyu Zhu, Chengcheng Han, Yong Qian, Qiushi Sun, Xiang Li, Ming Gao, Xuezhi Cao, Yunsen Xian

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding

Hao Feng, Zijian Wang, Jingqun Tang, Jinghui Lu, Wengang Zhou, Houqiang Li, Can Huang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Reinforcement Learning Based Multi-modal Feature Fusion Network for Novel Class Discovery

Qiang Li, Qiuyang Ma, Weizhi Nie, Anan Liu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.AI、cs.MM

Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Prompts

Xiaocui Yang, Shi Feng, Daling Wang, Pengfei Hong, Soujanya Poria

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

Comments 9 pages, 2 figures, 7 tables. It has been accepted ACM MM 2023

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

Zhuo Chen, Jiaoyan Chen, Wen Zhang, Lingbing Guo, Yin Fang, Yufeng Huang, Yichi Zhang, Yuxia Geng, Jeff Z. Pan, Wenting Song, Huajun Chen

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments ACM Multimedia 2023 Accpeted, Repo: https://github.com/zjukg/MEAformer

Journal ref ACM MM 2023

ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning

Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, Xiangyu Zhang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 15 pages, 8 figures

TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Prediction

Linfeng Liu, Junyan Lyu, Siyu Liu, Xiaoying Tang, Shekhar S. Chandra, Fatima A. Nasrallah

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Revisiting Modality Imbalance In Multimodal Pedestrian Detection

Arindam Das, Sudip Das, Ganesh Sistu, Jonathan Horgan, Ujjwal Bhattacharya, Edward Jones, Martin Glavin, Ciarán Eising

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 5 pages, 3 figure, 4 tables

Journal ref In proceedings of the IEEE 2023 International Conference on Image Processing

STrajNet: Multi-modal Hierarchical Transformer for Occupancy Flow Field Prediction in Autonomous Driving

Haochen Liu, Zhiyu Huang, Chen Lv

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

SFusion: Self-attention based N-to-One Multimodal Fusion Block

Zecheng Liu, Jia Wei, Rui Li, Jianlong Zhou

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments This paper has been accepted by MICCAI 2023

Deep Equilibrium Multimodal Fusion

Jinhong Ni, Yalong Bai, Wei Zhang, Ting Yao, Tao Mei

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.MM

Multi-modal Representation Learning for Social Post Location Inference

Ruiting Dai, Jiayi Luo, Xucheng Luo, Lisi Mo, Wanlun Ma, Fan Zhou

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments 6 pages, 2023 International Conference on Communications

MultiModal-GPT: A Vision and Language Model for Dialogue with Humans

Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, Kai Chen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 10 pages, 8 figures

M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning

Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, Qi Liu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Fix dataset url: https://huggingface.co/datasets/MMInstruction/M3IT Project: https://m3-it.github.io/

A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics

Hong-Yu Zhou, Yizhou Yu, Chengdi Wang, Shu Zhang, Yuanxu Gao, Jia Pan, Jun Shao, Guangming Lu, Kang Zhang, Weimin Li

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by Nature Biomedical Engineering

Brain encoding models based on multimodal transformers can transfer across language and vision

Jerry Tang, Meng Du, Vy A. Vo, Vasudev Lal, Alexander G. Huth

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Modeling Paragraph-Level Vision-Language Semantic Alignment for Multi-Modal Summarization

Chenhao Cui, Xinnian Liang, Shuangzhi Wu, Zhoujun Li

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram

Ming-Liang Zhang, Fei Yin, Cheng-Lin Liu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted to IJCAI 2023

Prediction of brain tumor recurrence location based on multi-modal fusion and nonlinear correlation learning

Tongxue Zhou, Alexandra Noeuveglise, Romain Modzelewski, Fethi Ghazouani, Sébastien Thureau, Maxime Fontanilles, Su Ruan

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 23 pages, 4 figures

Journal ref Computerized Medical Imaging and Graphics, 2023

Multi-modal Fake News Detection on Social Media via Multi-grained Information Fusion

Yangming Zhou, Yuzhou Yang, Qichao Ying, Zhenxing Qian, Xinpeng Zhang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.MM

Comments Accepted by ICMR 2023

Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition

Yujin Wu, Mohamed Daoudi, Ali Amad

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI、eess.AS

Comments Accepted IEEE Transactions On Affective Computing