多模态大模型

UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding

Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, Yu-Gang Jiang

专题命中 多模态生成 ：multimodal(title)；分类 cs.CV、cs.AI

Comments Accpeted to CVPR 2025 workshop

Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control

NVIDIA, :, Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, Dieter Fox, Yunhao Ge, Jinwei Gu, Ali Hassani, Michael Isaev, Pooya Jannaty, Shiyi Lan, Tobias Lasser, Huan Ling, Ming-Yu Liu, Xian Liu, Yifan Lu, Alice Luo, Qianli Ma, Hanzi Mao, Fabio Ramos, Xuanchi Ren, Tianchang Shen, Xinglong Sun, Shitao Tang, Ting-Chun Wang, Jay Wu, Jiashu Xu, Stella Xu, Kevin Xie, Yuchong Ye, Xiaodong Yang, Xiaohui Zeng, Yu Zeng

专题命中 多模态生成 ：multimodal(title)；分类 cs.CV、cs.AI

UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing

Yiheng Li, Ruibing Hou, Hong Chang, Shiguang Shan, Xilin Chen

专题命中 多模态生成 ：multimodal(title)；分类 cs.CV、cs.AI

Fundus2Video: Cross-Modal Angiography Video Generation from Static Fundus Photography with Clinical Knowledge Guidance

Weiyi Zhang, Siyu Huang, Jiancheng Yang, Ruoyu Chen, Zongyuan Ge, Yingfeng Zheng, Danli Shi, Mingguang He

专题命中 多模态生成 ：cross-modal(title)；分类 cs.CV、cs.AI

Comments The paper has been accepted by Medical Image Computing and Computer Assisted Intervention Society (MICCAI) 2024

Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation

Kohei Uehara, Nabarun Goswami, Hanqin Wang, Toshiaki Baba, Kohtaro Tanaka, Tomohiro Hashimoto, Kai Wang, Rei Ito, Takagi Naoya, Ryo Umagami, Yingyi Wen, Tanachai Anakewat, Tatsuya Harada

专题命中 多模态生成 ：multi-modal(title)；分类 cs.CV、cs.CL

Leveraging Multimodal CycleGAN for the Generation of Anatomically Accurate Synthetic CT Scans from MRIs

Leonardo Crespi, Samuele Camnasio, Damiano Dei, Nicola Lambri, Pietro Mancosu, Marta Scorsetti, Daniele Loiacono

专题命中 多模态生成 ：multimodal(title)；分类 cs.CV、cs.AI

Comments Currently submitted to: Scientific Reports

Network Bending of Diffusion Models for Audio-Visual Generation

Luke Dzwonczyk, Carmine Emanuele Cella, David Ban

专题命中 多模态生成 ：audio-visual(title)；分类 cs.MM、eess.AS

Comments 8 pages, 5 figures, to be published in the proceedings of the 27th International Conference on Digital Audio Effects (DAFx24), for additional image and video examples see https://dzluke.github.io/DAFX2024/

nach0: Multimodal Natural and Chemical Languages Foundation Model

Micha Livne, Zulfat Miftahutdinov, Elena Tutubalina, Maksim Kuznetsov, Daniil Polykovskiy, Annika Brundyn, Aastha Jhunjhunwala, Anthony Costa, Alex Aliper, Alán Aspuru-Guzik, Alex Zhavoronkov

专题命中 多模态生成 ：multimodal(title)；分类 cs.CL、cs.AI

Comments Accepted to Chemical Science Journal. Models are publicly available via https://huggingface.co/insilicomedicine/nach0_base and https://huggingface.co/insilicomedicine/nach0_large

Journal ref Chemical Science, 15(22), 8380-8389, 2024

NeRF-Insert: 3D Local Editing with Multimodal Control Signals

Benet Oriol Sabat, Alessandro Achille, Matthew Trager, Stefano Soatto

专题命中 多模态生成 ：multimodal(title)；分类 cs.CV、cs.AI

Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lacking

David Bayani

专题命中 多模态生成 ：cross-modal(title)；分类 cs.CL、cs.AI

Comments Accepted in EACL 2024 as a long paper. See accepted/#long-papers" target="_blank" rel="noopener">https://2024.eacl.org/program/findings-accepted/#long-papers . Note: this paper's ArXiv version includes additional discussion, analysis, and types of experiments compared to the EACL version. Changes introduced in V2 of ArXiv paper: only this comment metadata. V1 was initially submission on July 26th, 2023 - release was delayed by ArXiv for a few days

Optimisation-Based Multi-Modal Semantic Image Editing

Bowen Li, Yongxin Yang, Steven McDonagh, Shifeng Zhang, Petru-Daniel Tudosiu, Sarah Parisot

专题命中 多模态生成 ：multi-modal(title)；分类 cs.CV、cs.CL

Cross-modal Generative Model for Visual-Guided Binaural Stereo Generation

Zhaojian Li, Bin Zhao, Yuan Yuan

专题命中 多模态生成 ：cross-modal(title)；分类 cs.CV、eess.AS

Clue: Cross-modal Coherence Modeling for Caption Generation

Malihe Alikhani, Piyush Sharma, Shengjie Li, Radu Soricut, Matthew Stone

专题命中 多模态生成 ：cross-modal(title)；分类 cs.CV、cs.CL

Comments Accepted as a long paper to ACL 2020

Warping of Radar Data into Camera Image for Cross-Modal Supervision in Automotive Applications

Christopher Grimm, Tai Fei, Ernst Warsitz, Ridha Farhoud, Tobias Breddermann, Reinhold Haeb-Umbach

专题命中 多模态生成 ：cross-modal(title)；分类 cs.CV、cs.AI

Multi-modal application: Image Memes Generation

Zhiyuan Liu, Chuanzheng Sun, Yuxin Jiang, Shiqi Jiang, Mei Ming

专题命中 多模态生成 ：multi-modal(title)；分类 cs.CV、cs.AI

Multimodal Story Generation on Plural Images

Jing Jiang

专题命中 多模态生成 ：multimodal(title)；分类 cs.CV、cs.CL

Comments This is an undergraduate project report. Completed Dec. 2019 at the Cooper Union

Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders

Masha Itkina, Boris Ivanovic, Ransalu Senanayake, Mykel J. Kochenderfer, Marco Pavone

专题命中 多模态生成 ：multimodal(title)；分类 cs.CV、cs.AI

Comments 21 pages, 15 figures, 34th Conference on Neural Information Processing Systems (NeurIPS 2020)

Multimodal Research in Vision and Language: A Review of Current and Emerging Trends

Shagun Uppal, Sarthak Bhagat, Devamanyu Hazarika, Navonil Majumdar, Soujanya Poria, Roger Zimmermann, Amir Zadeh

专题命中 多模态生成 ：multimodal(title)；分类 cs.CV、cs.CL

A Formal Analysis of Multimodal Referring Strategies Under Common Ground

Nikhil Krishnaswamy, James Pustejovsky

专题命中 多模态生成 ：multimodal(title)；分类 cs.CL、cs.AI

Comments 9 pages (incl refs), 7 figures, 3 tables, proceedings of LREC 2020 (postponed due to COVID-19)

DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder

Xiaodong Gu, Kyunghyun Cho, Jung-Woo Ha, Sunghun Kim

专题命中 多模态生成 ：multimodal(title)；分类 cs.CL、cs.AI

Comments Published as a conference paper at ICLR 2019

Physics-Informed Latent Diffusion for Multimodal Brain MRI Synthesis

Sven Lüpke, Yousef Yeganeh, Ehsan Adeli, Nassir Navab, Azade Farshad

专题命中 多模态生成 ：multimodal(title,comments)；分类 cs.CV

Comments 5th International Workshop on Multiscale Multimodal Medical Imaging (MICCAI 2024), Project page: https://sven-luepke.github.io/phy-ldm-mri/

CustomDance: Customized 3D Dance Generation with Coarse-to-Fine Human-Centered Interactive Control

CustomDance：基于以人为中心的粗到细交互式控制的定制化3D舞蹈生成

Xulong Tang, Kaixing Yang, Xiaohu Guo, Prabhakaran Balakrishnan, Rawan Alghofaili

专题命中 多模态生成 ：MLLM(abstract,abstract_cn)；multimodal(abstract)

AI总结 CustomDance是基于粗到细交互式控制的定制化3D舞蹈生成系统，通过三阶段流程实现AI辅助编舞，在定量和定性比较中优于基线，为用户提供全面控制。

AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation

AVTok: 用于整体音视频生成的1D统一分词化

Kien T. Pham, I Chieh Chen, Qifeng Chen, Long Chen

机构 * The Hong Kong University of Science and Technology（香港科技大学） 纠错

专题命中 多模态生成 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

AI总结 提出AVTok统一分词器，采用双流Transformer架构和分层训练策略，将音视频对编码为紧凑的一维潜在表示，在音视频重建及下游生成任务中表现优异。

Comments ECCV 2026

UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

UniversalRAG: 在多样模态和粒度的语料库上实现检索增强生成

Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Baek, Sung Ju Hwang

机构 * KAIST（韩国科学技术院） 纠错

专题命中 多模态生成 ：any-to-any(abstract,abstract_cn)；分类 cs.CV、cs.CL、cs.AI

AI总结 本文提出UniversalRAG，一种能够处理多种模态和粒度的检索增强生成框架，通过动态路由机制和多粒度组织，提升跨模态知识检索的有效性，实验表明其在多个模态基准上的优越性。

Comments ACL 2026. Project page : https://universalrag.github.io

Inference-Time Scaling for Joint Audio-Video Generation

联合音视频生成的推理时缩放

Jaemin Jung, Kyeongha Rho, Inkyu Shin, Joon Son Chung

机构 * Korea Advanced Institute of Science and Technology（韩国科学技术院） ； Luma AI 纠错

专题命中 多模态生成 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

AI总结 针对联合音视频生成中多目标优化的挑战，提出多验证器框架与自适应奖励加权算法，在无需额外训练的情况下显著提升语义对齐、感知质量和音视频同步。

Comments Accepted by Transactions on Machine Learning Research (TMLR). Project page: https://jung-jaemin.github.io/ITS-AVGen-Proj/

ETCHR: Editing To Clarify and Harness Reasoning

ETCHR: 通过编辑来澄清和利用推理

Beichen Zhang, Yuhong Liu, Jinsong Li, Yuhang Zang, Jiaqi Wang, Dahua Lin

机构 * The Chinese University of Hong Kong Shanghai AI Laboratory（香港中文大学上海人工智能实验室） ； Shanghai AI Laboratory（上海人工智能实验室） ； Shanghai Jiao Tong University（上海交通大学） ； Shanghai Innovation Institute（上海创新研究院） 纠错

专题命中 多模态生成 ：multimodal(abstract)；MLLM(abstract_cn)；分类 cs.CV、cs.CL、cs.AI

AI总结 针对多模态大语言模型在需要细粒度关注或视角变换的问题上纯文本思维链的瓶颈，提出了一种解耦的图像编辑模型ETCHR，通过两阶段训练（推理模仿和推理增强）弥合语言侧和生成侧差距，无需训练即可插入不同MLLM，在五个任务族上平均Pass@1提升4.61-5.47个百分点。

Comments Code, model and data are open-sourced at https://github.com/InternLM/ETCHR

OSCBench: Benchmarking Object State Change in Text-to-Video Generation

OSCBench：文本到视频生成中对象状态变化的基准测试

Xianjing Han, Bin Zhu, Shiqi Hu, Franklin Mingzhe Li, Patrick Carrington, Roger Zimmermann, Jingjing Chen

机构 * National University of Singapore（新加坡国立大学） ； Singapore Management University（新加坡管理大学） ； Carnegie Mellon University（卡内基梅隆大学） ； Fudan University（复旦大学） 纠错

专题命中 多模态生成 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结 本文提出OSCBench基准，用于评估文本到视频模型在对象状态变化上的性能，揭示当前模型在处理新场景时的不足。

Comments ACL 2026 Main Conference, Project page: https://hanxjing.github.io/OSCBench

Tiny Inference-Time Scaling with Latent Verifiers

微小推理时间缩放与潜在验证器

Davide Bucciarelli, Evelyn Turri, Lorenzo Baraldi, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

机构 * University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学） ； University of Pisa（比萨大学） 纠错

专题命中 多模态生成 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结 本文提出VHS验证器，直接在扩散变换器单步生成器的中间隐藏表示上操作，减少验证成本并提升性能，实现更高效的推理时间缩放。

Comments Findings of CVPR 2026 - Code at: https://aimagelab.github.io/VHS/

One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generation

一个尺寸，多种适配：在大规模广告图像生成中对多样化群体点击偏好进行对齐

Shuo Lu, Haohan Wang, Wei Feng, Weizhen Wang, Shen Zhang, Yaoyu Li, Ao Ma, Zheng Zhang, Jingjing Lv, Junjie Shen, Ching Law, Bing Zhan, Yuan Xu, Huizai Yao, Yongcan Yu, Chenyang Si, Jian Liang

机构 * NLPR & MAIS, CASIA（中国科学院长春光学精密机械与物理研究所 & 中国科学院自动化所） ； School of AI, UCAS（中国科学院大学人工智能学院） ； HKUST(gz)（香港科技大学） ； PRLab, NJU（南京大学PRLab） 纠错

专题命中 多模态生成 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结 本文提出OSMF框架，通过自适应分组和群组感知多模态模型，解决广告图像生成中用户群体点击偏好多样性的优化问题。

TalkVerse: Democratizing Minute-Long Audio-Driven Video Generation

TalkVerse：民主化分钟级音频驱动视频生成

Zhenzhi Wang, Jian Wang, Ke Ma, Dahua Lin, Bing Zhou

机构 * The Chinese University of Hong Kong（香港中文大学） ； Snap Inc.（Snap公司） 纠错

专题命中 多模态生成 ：MLLM(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结 TalkVerse通过大规模开放数据和高效模型，实现了分钟级音频驱动视频生成，降低研究门槛，提升生成质量与效率。

Comments open-sourced single-person full-body talking video generation dataset, training code and checkpoints