多模态大模型 - arXivDaily 专题

2401.01044 2024-01-03 cs.SD cs.AI cs.CL eess.AS 67%

Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation

Jinlong Xue, Yayue Deng, Yingming Gao, Ya Li

专题命中多模态生成：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Demo and implementation at https://auffusion.github.io

2305.19228 2023-12-25 cs.CL cs.AI cs.SD eess.AS 67%

Unsupervised Melody-to-Lyric Generation

Yufei Tian, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone, Gunnar Sigurdsson, Chenyang Tao, Wenbo Zhao, Yiwen Chen, Tagyoung Chung, Jing Huang, Nanyun Peng

专题命中多模态生成：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments ACL 2023. arXiv admin note: substantial text overlap with arXiv:2305.07760

2307.04749 2023-12-07 cs.CV cs.AI cs.LG cs.MM stat.ML 67%

Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback

Jaskirat Singh, Liang Zheng

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Journal ref Published at NeurIPS 2023

2311.11097 2023-11-21 cs.CV cs.AI cs.CL cs.LG 67%

Radiology Report Generation Using Transformers Conditioned with Non-imaging Data

Nurbanu Aksoy, Nishant Ravikumar, Alejandro F Frangi

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

2305.16397 2023-11-06 cs.CV cs.AI cs.CL 67%

Are Diffusion Models Vision-And-Language Reasoners?

Benno Krojer, Elinor Poole-Dayan, Vikram Voleti, Christopher Pal, Siva Reddy

专题命中多模态生成：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to NeurIPS 2023

2301.12503 2023-09-12 cs.SD cs.AI cs.MM eess.AS eess.SP 67%

AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, Mark D. Plumbley

专题命中多模态生成：cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted by ICML 2023. Demo and implementation at https://audioldm.github.io. Evaluation toolbox at https://github.com/haoheliu/audioldm_eval

2309.03549 2023-09-08 cs.CV cs.AI cs.MM 67%

Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation

Jiaxi Gu, Shicong Wang, Haoyu Zhao, Tianyi Lu, Xing Zhang, Zuxuan Wu, Songcen Xu, Wei Zhang, Yu-Gang Jiang, Hang Xu

专题命中多模态生成：image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

2308.07605 2023-08-16 cs.CV cs.AI cs.MM 67%

SGDiff: A Style Guided Diffusion Model for Fashion Synthesis

Zhengwentai Sun, Yanghong Zhou, Honghong He, P. Y. Mok

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by ACM MM'23

2307.15199 2023-08-16 cs.CV cs.AI cs.CL cs.LG 67%

PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization

Junhyeong Cho, Gilhyun Nam, Sungyeon Kim, Hunmin Yang, Suha Kwak

专题命中多模态生成：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ICCV 2023, Project Page: https://promptstyler.github.io/

2308.01546 2023-08-04 cs.SD cs.AI cs.LG cs.MM eess.AS 67%

MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies

Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg-Kirkpatrick, Shlomo Dubnov

专题命中多模态生成：cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 16 pages, 3 figures, 2 tables, demo page: https://musicldm.github.io/

2307.02971 2023-07-07 cs.CV cs.AI cs.CL 67%

On the Cultural Gap in Text-to-Image Generation

Bingshuai Liu, Longyue Wang, Chenyang Lyu, Yong Zhang, Jinsong Su, Shuming Shi, Zhaopeng Tu

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Equal contribution: Bingshuai Liu and Longyue Wang. Work done while Bingshuai Liu and Chengyang Lyu were interning at Tencent AI Lab. Zhaopeng Tu is the corresponding author

2305.07760 2023-05-29 cs.AI cs.CL cs.MM 67%

Unsupervised Melody-Guided Lyrics Generation

Yufei Tian, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone, Gunnar Sigurdsson, Chenyang Tao, Wenbo Zhao, Tagyoung Chung, Jing Huang, Nanyun Peng

专题命中多模态生成：cross-modal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Presented at AAAI23 CreativeAI workshop (Non-Archival). A later version is accepted to ACL23

2211.04604 2023-04-26 cs.RO cs.AI cs.CL cs.CV cs.LG 67%

StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects

Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, Chris Paxton

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to Robotics: Science and Systems (RSS) 2023. The previous version appeared in CoRL Workshop on Language and Robot Learning 2022

2211.14794 2022-12-12 cs.CV cs.AI cs.LG cs.MM stat.ML 67%

Traditional Classification Neural Networks are Good Generators: They are Competitive with DDPMs and GANs

Guangrun Wang, Philip H. S. Torr

专题命中多模态生成：image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments This paper has 29 pages with 22 figures, including rich supplementary information. Project page is at \url{https://classifier-as-generator.github.io/}

2210.13729 2022-10-26 cs.AI cs.CL cs.CV 67%

Hybrid Reinforced Medical Report Generation with M-Linear Attention and Repetition Penalty

Wenting Xu, Zhenghua Xu, Junyang Chen, Chang Qi, Thomas Lukasiewicz

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments This paper is current under peer-review in IEEE TNNLS

2208.11253 2022-08-25 cs.CV cs.AI cs.CL cs.LG 67%

FashionVQA: A Domain-Specific Visual Question Answering System

Min Wang, Ata Mahjoubfar, Anupama Joshi

专题命中多模态生成：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2203.09052 2022-03-18 cs.CV cs.AI cs.CL 67%

DU-VLG: Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-training

Luyang Huang, Guocheng Niu, Jiachen Liu, Xinyan Xiao, Hua Wu

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments To appear at Findings of ACL 2022

2110.09756 2021-10-20 cs.CV cs.CL cs.MM 67%

A Picture is Worth a Thousand Words: A Unified System for Diverse Captions and Rich Images Generation

Yupan Huang, Bei Liu, Jianlong Fu, Yutong Lu

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments ACM MM 2021 (Video and Demo Track). Code: https://github.com/researchmm/generate-it

1905.11240 2021-09-01 cs.CL cs.CV cs.MM 67%

HUMBO: Bridging Response Generation and Facial Expression Synthesis

Shang-Yu Su, Po-Wei Lin, Yun-Nung Chen

专题命中多模态生成：multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments The first two authors contributed to this work equally

2106.10876 2021-06-22 cs.CV cs.AI cs.MM 67%

Total Generate: Cycle in Cycle Generative Adversarial Networks for Generating Human Faces, Hands, Bodies, and Natural Scenes

Hao Tang, Nicu Sebe

专题命中多模态生成：cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted to TMM, an extended version of a paper published in ACM MM 2019. arXiv admin note: substantial text overlap with arXiv:1908.00999

2102.02779 2021-05-25 cs.CL cs.AI cs.CV cs.LG 67%

Unifying Vision-and-Language Tasks via Text Generation

Jaemin Cho, Jie Lei, Hao Tan, Mohit Bansal

专题命中多模态生成：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICML 2021 (15 pages, 4 figures, 14 tables)

1906.09676 2020-07-03 cs.CV cs.AI cs.CL 67%

CORAL8: Concurrent Object Regression for Area Localization in Medical Image Panels

Sam Maksoud, Arnold Wiliem, Kun Zhao, Teng Zhang, Lin Wu, Brian C. Lovell

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted for MICCAI 2019

1811.08763 2019-11-06 q-bio.NC 67%

Comparison of Brain Networks based on Predictive Models of Connectivity

Fani Deligianni, Jonathan D. Clayden, Guang-Zhong Yang

专题命中多模态生成：multimodal(abstract)；multi-modal(abstract)

Comments 7 pages, 4 figures

Journal ref 19th IEEE International Conference on Bioinformatics and Bioengineering (IEEE BIBE, 2019)

1909.04800 2019-10-18 cs.CV cs.CL cs.LG cs.MM 67%

Probabilistic framework for solving Visual Dialog

Badri N. Patro, Anupriy, Vinay P. Namboodiri

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

1910.09134 2022-04-20 cs.CV cs.CL 66%

Good, Better, Best: Textual Distractors Generation for Multiple-Choice Visual Question Answering via Reinforcement Learning

Jiaying Lu, Xin Ye, Yi Ren, Yezhou Yang

专题命中多模态生成：multi-modal(abstract,journal_ref)；分类 cs.CV、cs.CL

Journal ref CVPR'2022 Workshop on Open-Domain Retrieval Under a Multi-Modal Setting

2608.18076 2026-08-19 cs.CV cs.AI 新提交 62%

From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation

从语料到协同进化的能力：面向通用图像生成的以能力为中心的数据设计

Xingjian Wang, Zhao Wang, Taihang Hu, Jun Zheng, Qing Jin, Qinye Zhou, Zhengtao Wu, Yongchao Du, Zuan Gao, Chao Lin, Yefeng Shen, Xiaoli Xu, Zhengze Xu, Hao Yan, Yuhang Yu, Mingzhou Zhang, Mengting Chen

机构 * Alibaba Group（阿里巴巴集团）

专题命中多模态生成：multimodal(abstract)；分类 cs.CV、cs.AI

AI总结该研究提出能力驱动的数据基础设施，构建三类监督引擎，整理大规模图像语料，训练多模态扩散模型，在CPI-Bench等评估中展现良好生成与迁移能力。

Comments 19 pages, 10 figures

2608.07565 2026-08-19 cs.CV cs.AI 版本更新 62%

What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systems

下一步编辑什么：对话系统中视觉对齐的图像编辑后续建议

Zhijing Zhang, Jinpeng Yu, Xin Song, Bingnan Li, Chuyue Li, Changhui Du, Xiaolin Fang, Jiaming Liu, Ruihua Huang

机构 * Qwen Business Unit of Alibaba（阿里巴巴通义千问业务单元）； Southeast University（东南大学）

专题命中多模态生成：multimodal(abstract)；分类 cs.CV、cs.AI

AI总结该研究针对对话系统的图像创作场景，提出三阶段多模态推荐框架，经测试可降低视觉不一致率并显著提升推荐相关指标。

2608.16765 2026-08-18 cs.CV cs.AI 新提交 62%

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

TRACE-Bench：多参考图像生成的分解与诊断

Haoran Wang, Chaofan Ma, Ran Yi, Lizhuang Ma

机构 * Shanghai Jiao Tong University（上海交通大学）

专题命中多模态生成：multimodal(abstract)；分类 cs.CV、cs.AI

AI总结研究针对多参考图像生成基准的缺陷，构建TRACE-Bench基准，采用算子分解方法评估9个主流模型，发现解耦与属性绑定是核心瓶颈，最优模型属性保真度仅0.74

Comments Accepted to ACM Multimedia 2026 (ACM MM 2026)

2608.13602 2026-08-18 cs.MM cs.CV cs.SD 版本更新 62%

Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Video Avatar Generation

Omni-LiveAvatar：分钟级实时流式视听联动数字人生成

Lunjie Zhu, Xingtong Ge, Fangyu Lin, Yi Zhang, Zhening Liu, Mengfei Li, Yumeng Zhang, Guanglu Song, Yu Liu, Jun Zhang

专题命中多模态生成：cross-modal(abstract)；分类 cs.CV、cs.MM

AI总结本研究提出Omni-LiveAvatar框架，通过渐进式自回归蒸馏等技术，实现分钟级实时流式视听联动数字人生成，速度较LTX-2提升33倍且性能优于基线模型。

2607.19942 2026-08-18 cs.CV cs.AI 版本更新 62%

G-MAD: A Game-Based Data Generation Framework for Multi-View RGB-T Aerial Object Detection

G-MAD：用于多视图RGB-T航空目标检测的基于游戏的数据生成框架

Yechan Kim, JongHyun Park, Dongho Yoon, Namhoon Jung, Moongu Jeon

机构 * LIG Defense&Aerospace（LIG国防与航空航天公司）； GIST（韩国科学技术院）

专题命中多模态生成：multi-modal(abstract)；分类 cs.CV、cs.AI

AI总结研究针对航空目标检测中数据集构建的问题，提出基于游戏的G-MAD数据生成框架，可解决视点控制、数据对齐及标注成本等问题，支持多视图相机放置等功能，还构建并发布了AMOD基准。

Comments ACM Multimedia 2026 (Supplementary Material Included)

1. 多模态生成 4959 篇

Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation

Unsupervised Melody-to-Lyric Generation

Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback

Radiology Report Generation Using Transformers Conditioned with Non-imaging Data

Are Diffusion Models Vision-And-Language Reasoners?

AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation

SGDiff: A Style Guided Diffusion Model for Fashion Synthesis

PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization

MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies

On the Cultural Gap in Text-to-Image Generation

Unsupervised Melody-Guided Lyrics Generation

StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects

Traditional Classification Neural Networks are Good Generators: They are Competitive with DDPMs and GANs

Hybrid Reinforced Medical Report Generation with M-Linear Attention and Repetition Penalty

FashionVQA: A Domain-Specific Visual Question Answering System

DU-VLG: Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-training

A Picture is Worth a Thousand Words: A Unified System for Diverse Captions and Rich Images Generation

HUMBO: Bridging Response Generation and Facial Expression Synthesis

Total Generate: Cycle in Cycle Generative Adversarial Networks for Generating Human Faces, Hands, Bodies, and Natural Scenes

Unifying Vision-and-Language Tasks via Text Generation

CORAL8: Concurrent Object Regression for Area Localization in Medical Image Panels

Comparison of Brain Networks based on Predictive Models of Connectivity

Probabilistic framework for solving Visual Dialog

Good, Better, Best: Textual Distractors Generation for Multiple-Choice Visual Question Answering via Reinforcement Learning

From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation

What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systems

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Video Avatar Generation

G-MAD: A Game-Based Data Generation Framework for Multi-View RGB-T Aerial Object Detection