多模态大模型 - arXivDaily 专题

2406.07686 2024-06-13 cs.CV 83%

AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation

Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, Yapeng Tian

专题命中多模态生成：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

2311.07470 2024-06-12 cs.CL 83%

Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers

Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, Meng Wang

专题命中多模态生成：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CL

2402.12451 2024-06-07 cs.CV cs.AI cs.CL cs.MM 83%

The Revolution of Multimodal Large Language Models: A Survey

Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara

专题命中多模态生成：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2024 (Findings)

2402.14859 2024-06-04 cs.CR cs.AI cs.CY cs.LG 83%

The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative

Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Yu Kong, Tianlong Chen, Huan Liu

专题命中多模态生成：MLLM(title,abstract)；multimodal(abstract)；分类 cs.AI

Comments Accepted to workshop on ReGenAI@CVPR 2024

2405.01926 2024-05-06 cs.CV 83%

Auto-Encoding Morph-Tokens for Multimodal LLM

Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, Hanwang Zhang

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted by ICML 2024

2404.05674 2024-04-09 cs.CV 83%

MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, Xiao Yang

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

2312.06988 2023-12-19 cs.CV 83%

MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving

Guangfeng Jiang, Jun Liu, Yuzhi Wu, Wenlong Liao, Tao He, Pai Peng

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments AAAI2024

2312.06739 2023-12-13 cs.CV 83%

SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models

Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, Ying Shan

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Project page: https://yuzhou914.github.io/SmartEdit/

2312.06685 2023-12-13 cs.AI 83%

Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models

Shitian Zhao, Zhuowan Li, Yadong Lu, Alan Yuille, Yan Wang

专题命中多模态生成：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.AI

2312.01771 2023-12-05 cs.CV 83%

IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks

Jiarui Xu, Yossi Gandelsman, Amir Bar, Jianwei Yang, Jianfeng Gao, Trevor Darrell, Xiaolong Wang

专题命中多模态生成：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Project page: https://jerryxu.net/IMProv

2312.01745 2023-12-05 cs.CV 83%

Cross-Modal Adaptive Dual Association for Text-to-Image Person Retrieval

Dixuan Lin, Yixing Peng, Jingke Meng, Wei-Shi Zheng

专题命中多模态生成：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

2311.17618 2023-12-04 cs.CV 83%

ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model

Fukun Yin, Xin Chen, Chi Zhang, Biao Jiang, Zibo Zhao, Jiayuan Fan, Gang Yu, Taihao Li, Tao Chen

专题命中多模态生成：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

2311.18835 2023-12-01 cs.CV 83%

InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation

Rongyao Fang, Shilin Yan, Zhaoyang Huang, Jingqiu Zhou, Hao Tian, Jifeng Dai, Hongsheng Li

专题命中多模态生成：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 10 pages

2311.00618 2023-11-02 cs.CV 83%

De-Diffusion Makes Text a Strong Cross-Modal Interface

Chen Wei, Chenxi Liu, Siyuan Qiao, Zhishuai Zhang, Alan Yuille, Jiahui Yu

专题命中多模态生成：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Technical report. Project page: https://dediffusion.github.io

2310.07222 2023-10-12 cs.CV 83%

Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model

Shiyuan Yang, Xiaodong Chen, Jing Liao

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ACMMM'23

2308.09306 2023-08-21 cs.CV 83%

DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability

Runhui Huang, Jianhua Han, Guansong Lu, Xiaodan Liang, Yihan Zeng, Wei Zhang, Hang Xu

专题命中多模态生成：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments ICCV2023

2306.17115 2023-07-04 cs.CV 83%

Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, Shenghua Gao

专题命中多模态生成：image-text(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Project Website: https://neuralcarver.github.io/michelangelo

2306.06583 2023-06-13 cs.CV 83%

REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge

Siyang Song, Micol Spitale, Cheng Luo, German Barquero, Cristina Palmero, Sergio Escalera, Michel Valstar, Tobias Baur, Fabien Ringeval, Elisabeth Andre, Hatice Gunes

专题命中多模态生成：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2305.18433 2023-05-31 cs.LG cs.CV 83%

Cognitively Inspired Cross-Modal Data Generation Using Diffusion Models

Zizhao Hu, Mohammad Rostami

专题命中多模态生成：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2303.06555 2023-05-31 cs.LG cs.CV 83%

One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale

Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, Jun Zhu

专题命中多模态生成：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted to ICML2023

2212.09478 2023-03-27 cs.CV 83%

MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, Baining Guo

专题命中多模态生成：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by CVPR 2023

2209.03430 2023-02-21 cs.LG cs.AI cs.CL cs.CV cs.MM 83%

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency

专题命中多模态生成：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2206.07771 2023-02-17 cs.CV 83%

Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation

Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, Yan Yan

专题命中多模态生成：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments ICLR 2023. Project at https://github.com/L-YeZhu/CDCD

2212.09918 2022-12-21 cs.AI 83%

Generalizing Multimodal Variational Methods to Sets

Jinzhao Zhou, Yiqun Duan, Zhihong Chen, Yu-Cheng Chang, Chin-Teng Lin

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments First Submission

2206.13393 2022-07-15 eess.IV cs.CV cs.LG q-bio.NC 83%

Cross-Modal Transformer GAN: A Brain Structure-Function Deep Fusing Framework for Alzheimer's Disease

Junren Pan, Shuqiang Wang

专题命中多模态生成：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2207.03430 2022-07-08 eess.IV cs.CV 83%

A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completion

Xiangxi Meng, Yuning Gu, Yongsheng Pan, Nizhuan Wang, Peng Xue, Mengkang Lu, Xuming He, Yiqiang Zhan, Dinggang Shen

专题命中多模态生成：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2101.04702 2022-04-15 cs.CV 83%

Cross-Modal Contrastive Learning for Text-to-Image Generation

Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, Yinfei Yang

专题命中多模态生成：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments CVPR 2021

2110.08515 2022-03-30 cs.CL cs.AI cs.CV cs.LG cs.MM 83%

Multimodal Dialogue Response Generation

Qingfeng Sun, Yujing Wang, Can Xu, Kai Zheng, Yaming Yang, Huang Hu, Fei Xu, Jessica Zhang, Xiubo Geng, Daxin Jiang

专题命中多模态生成：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ACL 2022 Main Conference

2108.10509 2021-08-25 cs.MM 83%

Improving Fake News Detection by Using an Entity-enhanced Framework to Fuse Diverse Multimodal Clues

Peng Qi, Juan Cao, Xirong Li, Huan Liu, Qiang Sheng, Xiaoyue Mi, Qin He, Yongbiao Lv, Chenyang Guo, Yingchao Yu

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.MM

Comments To appear in MM 2021 industrial track (long paper)

2103.00823 2021-06-01 cs.CL 83%

M6: A Chinese Multimodal Pretrainer

Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, Jie Zhang, Jianwei Zhang, Xu Zou, Zhikang Li, Xiaodong Deng, Jie Liu, Jinbao Xue, Huiling Zhou, Jianxin Ma, Jin Yu, Yong Li, Wei Lin, Jingren Zhou, Jie Tang, Hongxia Yang

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments 12 pages, technical report. Extension of paper "M6" accepted to KDD 2021

1. 多模态生成 4932 篇

AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation

Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers

The Revolution of Multimodal Large Language Models: A Survey

The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative

Auto-Encoding Morph-Tokens for Multimodal LLM

MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving

SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models

Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models

IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks

Cross-Modal Adaptive Dual Association for Text-to-Image Person Retrieval

ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model

InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation

De-Diffusion Makes Text a Strong Cross-Modal Interface

Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model

DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability

Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge

Cognitively Inspired Cross-Modal Data Generation Using Diffusion Models

One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale

MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation

Generalizing Multimodal Variational Methods to Sets

Cross-Modal Transformer GAN: A Brain Structure-Function Deep Fusing Framework for Alzheimer's Disease

A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completion

Cross-Modal Contrastive Learning for Text-to-Image Generation

Multimodal Dialogue Response Generation

Improving Fake News Detection by Using an Entity-enhanced Framework to Fuse Diverse Multimodal Clues

M6: A Chinese Multimodal Pretrainer