多模态大模型 - arXivDaily 专题

2410.13861 2024-10-22 cs.CV 85%

PUMA: Empowering Unified MLLM with Multi-granular Visual Generation

Rongyao Fang, Chengqi Duan, Kun Wang, Hao Li, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, Hongsheng Li, Xihui Liu

专题命中多模态生成：MLLM(title,abstract)；multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.CV

Comments Project page: https://rongyaofang.github.io/puma/

2407.08683 2024-10-14 cs.CV 85%

SEED-Story: Multimodal Long Story Generation with Large Language Model

Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, Yingcong Chen

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；image-text(abstract)；分类 cs.CV

Comments Our models, codes and datasets are released in https://github.com/TencentARC/SEED-Story

2403.03194 2024-10-04 cs.CL 85%

MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets

Hossein Aboutalebi, Hwanjun Song, Yusheng Xie, Arshit Gupta, Justin Sun, Hang Su, Igor Shalyminov, Nikolaos Pappas, Siffi Singh, Saab Mansour

专题命中多模态生成：multi-modal(title,abstract)；multimodal(abstract)；image-text(abstract)；分类 cs.CL

2401.13388 2024-06-07 cs.CV 85%

UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion

Wei Li, Xue Xu, Jiachen Liu, Xinyan Xiao

专题命中多模态生成：multimodal(title,abstract)；multi-modal(abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted by ACL 2024, Main Conference, Long Paper

2405.17927 2024-05-29 cs.AI cs.CL cs.CV cs.LG eess.AS 85%

The Evolution of Multimodal Model Architectures

Shakti N. Wadekar, Abhishek Chaurasia, Aman Chadha, Eugenio Culurciello

专题命中多模态生成：multimodal(title,abstract)；any-to-any(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 30 pages, 6 tables, 7 figures

2211.04861 2022-11-10 cs.CV 85%

ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation

Bin Shan, Yaqian Han, Weichong Yin, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

专题命中多模态生成：cross-modal(title,abstract)；multimodal(abstract)；image-text(abstract)；分类 cs.CV

Comments 13 pages, 2 figures

2107.00249 2021-07-07 cs.CV 85%

OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation

Jing Liu, Xinxin Zhu, Fei Liu, Longteng Guo, Zijia Zhao, Mingzhen Sun, Weining Wang, Hanqing Lu, Shiyu Zhou, Jiajun Zhang, Jinqiao Wang

专题命中多模态生成：cross-modal(title,abstract)；multi-modal(abstract)；image-text(abstract)；分类 cs.CV

2607.27699 2026-07-31 cs.CV cs.AI 新提交 85%

RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation

RefineSVG：视觉反馈驱动的图像转SVG生成强化学习

Shaobo Liu, Feiqiao Mao, Shuaishuai Zhou, Yan Zhan, Weiqi Tan, Zhiqiong Lu, Zhengping Liang

机构 * Shenzhen University（深圳大学）； Peking University（北京大学）

专题命中多模态生成：MLLM(summary_cn,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

AI总结 RefineSVG是一种视觉反馈驱动的闭环框架，通过引入SVG语义词汇表和渐进式训练，使MLLM实现高保真图像转SVG生成，性能优于现有基线。

Comments 17 pages, 5 main-paper figures. Accepted at the 34th ACM International Conference on Multimedia (ACM MM 2026). Includes the complete supplementary material

2603.12252 2026-06-19 cs.CV cs.CL 版本更新 85%

EndoCoT: Scaling Endogenous Chain-of-Thought Reasoning in Diffusion Models

EndoCoT：扩散模型中的内生思维链推理扩展

Xuanlang Dai, Yujie Zhou, Long Xing, Jiazi Bu, Xilin Wei, Yuhong Liu, Beichen Zhang, Kai Chen, Yuhang Zang

机构 * Shanghai AI Laboratory（上海人工智能实验室）； Xi’an Jiaotong University（西安交通大学）； University of Science and Technology of China（中国科学技术大学）； Shanghai Jiaotong University（上海交通大学）； Fudan University（复旦大学）； The Chinese University of Hong Kong（香港中文大学）

专题命中多模态生成：MLLM(summary_cn,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

AI总结提出EndoCoT框架，通过迭代思维引导模块激活MLLM的推理潜力，并利用终端思维接地模块确保推理轨迹与文本监督对齐，使DiT逐步执行复杂任务，在多个基准上平均准确率达92.1%。

Comments 23 pages, 18 figures, The code and dataset are publicly available at https://internlm.github.io/EndoCoT/

2406.11403 2025-02-06 cs.CV cs.CL 85%

Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report

Franz Louis Cesista

专题命中多模态生成：multimodal(title,abstract)；multimodal foundation model(abstract,comments)；分类 cs.CV、cs.CL

Comments Conference on Computer Vision and Pattern Recognition's 2nd Multimodal Foundation Models Challenge

2407.12009 2024-07-18 cs.CV cs.AI 85%

Using Multimodal Foundation Models and Clustering for Improved Style Ambiguity Loss

James Baker

专题命中多模态生成：multimodal(title)；multimodal foundation model(title)；分类 cs.CV、cs.AI

2607.20057 2026-07-23 q-bio.BM cs.LG 新提交 85%

Antigen-specific Antibody Multi-modal Foundation Model for Functional Antibody Design

用于功能性抗体设计的抗原特异性抗体多模态基础模型

Xiaoliang Shi, Zichen Wang, Runze Ma, Zhongyue Zhang, Shuangjia Zheng

机构 * Shanghai Jiao Tong University（上海交通大学）

专题命中多模态生成：multi-modal(title)；multimodal(abstract)；cross-modal(abstract)；multimodal foundation model(abstract)

AI总结该研究针对抗原特异性抗体设计难题，引入AAMFM多模态基础模型，通过跨模态适配器整合抗原信息，用校准直接偏好优化微调，实现抗体 - 抗原相互作用联合建模，实验证明其在功能性抗体设计中性能最优，有抗原特异性抗体工程潜力。

2604.25420 2026-07-02 cs.SE cs.HC 85%

Recommending Usability Improvements with Multimodal Large Language Models

利用多模态大语言模型推荐可用性改进

Sebastian Lubos, Alexander Felfernig, Damian Garber, Viet-Man Le, Manuel Henrich

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract,abstract_cn)

AI总结本文提出一种利用多模态大语言模型自动识别和推荐用户界面可用性改进的方法，通过分析有限的应用上下文和用户交互屏幕记录，基于尼尔森可用性启发式方法生成优先级排序的改进建议，验证了其在软件工程中的实用性。

Comments Accepted for publication at the ACM International Conference on the Foundations of Software Engineering (FSE 2026)

2602.22284 2026-06-26 cs.LG 版本更新 85%

BrepCoder: A Unified Multimodal Large Language Model for Multi-task B-rep Reasoning

BrepCoder: 用于多任务B-rep推理的统一多模态大语言模型

Mingi Kim, Yongjun Kim, Jungwoo Kang, Hyungki Kim

机构 * Chungnam National University（全南国立大学）

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract,abstract_cn)

AI总结提出BrepCoder，一种统一的多模态大语言模型，通过将CAD建模序列转换为类Python代码并与B-rep对齐，采用两阶段训练策略，实现从B-rep输入执行多种CAD任务，包括补全、纠错和问答。

2606.15210 2026-06-16 cs.DC 新提交 85%

Generation Quality-Latency Tradeoff-Aware Inference Offloading for Multimodal LLMs in Cloud-Edge Continuum

云边连续体中多模态大模型的生成质量-延迟权衡感知推理卸载

Zhongxiao Wang, Yueshen Xu, Xinkui Zhao, Wei Shao, Rui Li

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract,abstract_cn)

AI总结针对云边协同多模态大模型推理，提出QLMIO框架以优化生成质量与响应延迟的权衡，并构建MIOBench基准，实验表明延迟降低58.14%且任务完成率不变。

Comments This manuscript was submitted to IEEE Transactions on Mobile Computing on June 3, 2026

2606.09169 2026-06-09 cs.AI cs.CV cs.MM 新提交 85%

IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation

IMUG-Bench：交错理解与生成的统一多模态模型基准

Lingyi Meng, Zecong Tang, Haoran Li, Tengju Ru, Zhejun Cui, Weitong Lian, Qi Kang, Hangshuo Cao, Yichen Zhu, Yechi Liu, Kaixuan Wang, Yu-Jie Yuan, Chunwei Wang, Yu Zhang, Bo Dai

机构 * Zhejiang University（浙江大学）； The University of Hong Kong（香港大学）； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）； Huawei（华为）

专题命中多模态生成：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结提出IMUG-Bench基准，用于评估统一多模态模型在多轮交错图文对话中的理解与生成能力，包含3113样本和12034交互轮次，揭示了生成侧暴露偏差，并探索了测试时扩展策略。

2505.12658 2025-11-11 cs.DC 85%

HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving

Xianzhe Dong, Tongxuan Liu, Yuting Zeng, Liangyu Liu, Yang Liu, Siyu Wu, Yu Wu, Hailong Yang, Ke Zhang, Jing Li

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；cross-modal(abstract)

2505.22453 2025-10-28 cs.CL cs.AI cs.CV cs.LG 85%

First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training

Lai Wei, Yuting Li, Chen Wang, Yue Wang, Linghe Kong, Weiran Huang, Lichao Sun

机构 * School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）； Zhongguancun Academy（中关村学院）； Shanghai Innovation Institute（上海创新研究院）； Lehigh University（莱特大学）

专题命中多模态生成：multi-modal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by NeurIPS 2025

2510.14866 2025-10-17 cs.CV cs.AI cs.CL 85%

Benchmarking Multimodal Large Language Models for Face Recognition

Hatef Otroshi Shahreza, Sébastien Marcel

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

2510.01284 2025-10-03 cs.MM cs.CV cs.SD eess.AS 85%

Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

Chetwin Low, Weimin Wang, Calder Katyal

机构 * Character AI ； Yale University（耶鲁大学）

专题命中多模态生成：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

2508.05954 2025-08-11 cs.CV cs.AI cs.CL 85%

Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents

Han Lin, Jaemin Cho, Amir Zadeh, Chuan Li, Mohit Bansal

机构 * UNC Chapel Hill（北卡罗来纳大学教堂山分校）； Lambda

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://bifrost-1.github.io

2507.21391 2025-07-31 cs.CV cs.AI cs.CL 85%

Multimodal LLMs as Customized Reward Models for Text-to-Image Generation

Shijie Zhou, Ruiyi Zhang, Huaisheng Zhu, Branislav Kveton, Yufan Zhou, Jiuxiang Gu, Jian Chen, Changyou Chen

机构 * University at Buffalo（布法罗大学）； Adobe Research（Adobe研究）； Pennsylvania State University（宾夕法尼亚州立大学）

专题命中多模态生成：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at ICCV 2025. Code available at https://github.com/sjz5202/LLaVA-Reward

2505.00759 2025-05-14 cs.CV cs.AI cs.CL 85%

Multi-Modal Language Models as Text-to-Image Model Evaluators

Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat, Koustuv Sinha, Melissa Hall, Michal Drozdzal, Adriana Romero-Soriano

机构 * FAIR at Meta - Montreal, New York（Meta 蒙特利尔 FAIR）； University of Texas at Austin（德克萨斯大学奥斯汀分校）； Mila ； McGill University（麦吉尔大学）； Canada CIFAR AI chair（加拿大 CIFAR 人工智能主席）

专题命中多模态生成：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2405.16919 2025-03-11 cs.CV cs.AI cs.CL 85%

VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models

Zejun Li, Ruipu Luo, Jiwen Zhang, Minghui Qiu, Xuanjing Huang, Zhongyu Wei

专题命中多模态生成：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by NAACL 2025 main conference

2407.06304 2024-07-10 cs.CV cs.AI cs.CL 85%

VIMI: Grounding Video Generation through Multi-modal Instruction

Yuwei Fang, Willi Menapace, Aliaksandr Siarohin, Tsai-Shien Chen, Kuan-Chien Wang, Ivan Skorokhodov, Graham Neubig, Sergey Tulyakov

专题命中多模态生成：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2401.08567 2024-01-17 cs.LG cs.AI cs.CL cs.CV 85%

Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data

Yuhui Zhang, Elaine Sui, Serena Yeung-Levy

专题命中多模态生成：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Published at ICLR 2024

2401.01952 2024-01-05 cs.CV cs.AI cs.CL 85%

Instruct-Imagen: Image Generation with Multi-modal Instruction

Hexiang Hu, Kelvin C. K. Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, Ming-Wei Chang, Xuhui Jia

专题命中多模态生成：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 20 pages, 18 figures

2312.03025 2023-12-07 cs.AI cs.CL cs.CV cs.LG 85%

Training on Synthetic Data Beats Real Data in Multimodal Relation Extraction

Zilin Du, Haoxin Li, Xu Guo, Boyang Li

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

2308.04126 2023-08-21 cs.CV cs.LG cs.MM cs.SD eess.AS 85%

OmniDataComposer: A Unified Data Structure for Multimodal Data Fusion and Infinite Data Generation

Dongyang Yu, Shihao Wang, Yuan Fang, Wangpeng An

专题命中多模态生成：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2605.02937 2026-08-12 cs.LG cs.AI cs.CE 版本更新 84%

Proteo-R1: Reasoning Foundation Models for De Novo Protein Design

Proteo-R1：用于从头蛋白质设计的推理基础模型

Fang Wu, Weihao Xuan, Heli Qi, Hanqun Cao, Heng-Jui Chang, Zeqi Zhou, Haokai Zhao, Ma Jian, Carl Ma, Yu-Chi Cheng, Kuan Pang, Xiangru Tang, Zehong Wang, Guanlue Li, Hanchen Wang, Kejun Ying, Pan Lu, Chiho Im, Seungju Han, Peng Xia, Tinson Xu, Yinxi Li, Deyao Zhu, Pheng-Ann Heng, Naoto Yokoya, Masashi Sugiyama, Li Erran Li, Jure Leskovec, Yejin Choi

专题命中多模态生成：MLLM(summary_cn,abstract)；multimodal(abstract)；分类 cs.AI

AI总结针对现有从头蛋白质设计模型缺乏推理能力的问题，本文提出 Proteo-R1 双专家架构框架，通过 MLLM 识别关键功能残基作为硬约束，结合扩散模型实现稳定可解释的蛋白质设计。

Journal ref ICML 2026

1. 多模态生成 4925 篇

PUMA: Empowering Unified MLLM with Multi-granular Visual Generation

SEED-Story: Multimodal Long Story Generation with Large Language Model

MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets

UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion

The Evolution of Multimodal Model Architectures

ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation

OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation

RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation

EndoCoT: Scaling Endogenous Chain-of-Thought Reasoning in Diffusion Models

Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report

Using Multimodal Foundation Models and Clustering for Improved Style Ambiguity Loss

Antigen-specific Antibody Multi-modal Foundation Model for Functional Antibody Design

Recommending Usability Improvements with Multimodal Large Language Models

BrepCoder: A Unified Multimodal Large Language Model for Multi-task B-rep Reasoning

Generation Quality-Latency Tradeoff-Aware Inference Offloading for Multimodal LLMs in Cloud-Edge Continuum

IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation

HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving

First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training

Benchmarking Multimodal Large Language Models for Face Recognition

Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents

Multimodal LLMs as Customized Reward Models for Text-to-Image Generation

Multi-Modal Language Models as Text-to-Image Model Evaluators

VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models

VIMI: Grounding Video Generation through Multi-modal Instruction

Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data

Instruct-Imagen: Image Generation with Multi-modal Instruction

Training on Synthetic Data Beats Real Data in Multimodal Relation Extraction

OmniDataComposer: A Unified Data Structure for Multimodal Data Fusion and Infinite Data Generation

Proteo-R1: Reasoning Foundation Models for De Novo Protein Design