多模态大模型

Visual Agentic Reinforcement Fine-Tuning

Ziyu Liu, Yuhang Zang, Yushan Zou, Zijian Liang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, Jiaqi Wang

机构 * Shanghai Jiaotong University（上海交通大学） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； The Chinese University of Hong Kong（香港中文大学） ； Wuhan University（武汉大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments project url: https://github.com/Liuziyu77/Visual-RFT/tree/main/Visual-ARFT

Automatic Synthetic Data and Fine-grained Adaptive Feature Alignment for Composed Person Retrieval

Delong Liu, Haiwen Li, Zhaohui Hou, Zhicheng Zhao, Fei Su, Yuan Dong

机构 * Beijing University of Posts and Telecommunications（北京邮电大学） ； SenseTime（商汤科技） ； Beijing Key Laboratory of Network System and Network Culture（北京网络系统与网络文化重点实验室） ； Key Laboratory of Intereactive Technology and Experience System, Ministry of Culture and Tourism（文化和旅游部交互技术与体验系统重点实验室） 纠错

专题命中 多模态训练与对齐 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Segment Any RGB-Thermal Model with Language-aided Distillation

Dong Xing, Xianxun Zhu, Wei Zhou, Qika Lin, Hang Yang, Yuqing Wang

机构 * Changchun Institute of Optics, Fine Mechanicsand Physics, Chinese Academy of Sciences University（长春光学精密机械与物理研究所，中国科学院大学） ； School of Computing, Macquarie University（麦考瑞大学计算机学院） ； School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院） ； Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学 Saw Swee Hock 公共卫生学院） 纠错

专题命中 多模态训练与对齐 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments arXiv admin note: text overlap with arXiv:2412.04220 by other authors

PREMISE: Matching-based Prediction for Accurate Review Recommendation

Wei Han, Hui Chen, Soujanya Poria

机构 * Singapore University of Technology and Design（新加坡科技设计大学） ； National University of Singapore（国立新加坡大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.MM

Comments 19 pages, 16 figures

RGB-Thermal Infrared Fusion for Robust Depth Estimation in Complex Environments

Zelin Meng, Takanori Fukao

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 7 pages, 2 figures

Astrea: A MOE-based Visual Understanding Model with Progressive Alignment

Xiaoda Yang, JunYu Lu, Hongshun Qiu, Sijing Li, Hao Li, Shengpeng Ji, Xudong Tang, Jiayang Xu, Jiaqi Duan, Ziyue Jiang, Cong Lin, Sihang Cai, Zejian Xie, Zhuoyang Song, Songxin Zhang

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Heterogeneous bimodal attention fusion for speech emotion recognition

Jiachen Luo, Huy Phan, Lin Wang, Joshua Reiss

专题命中 多模态训练与对齐 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compression

Dongchen Lu, Yuyao Sun, Zilu Zhang, Leping Huang, Jianliang Zeng, Mao Shu, Huo Cao

专题命中 多模态训练与对齐 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Technical Approach for the EMI Challenge in the 8th Affective Behavior Analysis in-the-Wild Competition

Jun Yu, Lingsi Zhu, Yanjun Chi, Yunxiang Zhang, Yang Zheng, Yongqi Wang, Xilong Lu

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

DELST: Dual Entailment Learning for Hyperbolic Image-Gene Pretraining in Spatial Transcriptomics

Xulin Chen, Junzhou Huang

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Modality Interactive Mixture-of-Experts for Fake News Detection

Yifan Liu, Yaokun Liu, Zelin Li, Ruichen Yao, Yang Zhang, Dong Wang

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepted by the Proceedings of the ACM Web Conference 2025

Efficient Reasoning with Hidden Thinking

Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, Jiuxiang Gu

专题命中 多模态训练与对齐 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

Comments Preprint version

Natural Language Supervision for Low-light Image Enhancement

Jiahui Tang, Kaihua Zhou, Zhijian Luo, Yueen Hou

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 12 pages, 10 figures

From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Information

Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, Ying Shen

专题命中 多模态训练与对齐 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 32 pages, 22 tables, 7 figures

LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Information

Ke Wang, Hong Xuan

专题命中 多模态训练与对齐 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity

Zitao Shuai, Chenwei Wu, Zhengxu Tang, Liyue Shen

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes

He Cao, Yanjun Shao, Zhiyuan Liu, Zijing Liu, Xiangru Tang, Yuan Yao, Yu Li

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

MISS: A Generative Pretraining and Finetuning Approach for Med-VQA

Jiawei Chen, Dingkang Yang, Yue Jiang, Yuxuan Lei, Lihua Zhang

专题命中 多模态训练与对齐 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments ICANN, 2024

MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization

Yu Zhang, Qi Zhang, Zixuan Gong, Yiwei Shi, Yepeng Liu, Duoqian Miao, Yang Liu, Ke Liu, Kun Yi, Wei Fan, Liang Hu, Changwei Wang

专题命中 多模态训练与对齐 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments ICML 2024

Listen Then See: Video Alignment with Speaker Attention

Aviral Agrawal, Carlos Mateo Samudio Lezcano, Iqui Balam Heredia-Marin, Prabhdeep Singh Sethi

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

LaPA: Latent Prompt Assist Model For Medical Visual Question Answering

Tiancheng Gu, Kaicheng Yang, Dongnan Liu, Weidong Cai

专题命中 多模态训练与对齐 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments 10 pages, 4 figures, Accepted by CVPRW2024

LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding

Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, Cong Yao

专题命中 多模态训练与对齐 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Comments CVPR 2024

An Intermediate Fusion ViT Enables Efficient Text-Image Alignment in Diffusion Models

Zizhao Hu, Shaochong Jia, Mohammad Rostami

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web

Yibo Yan, Haomin Wen, Siru Zhong, Wei Chen, Haodong Chen, Qingsong Wen, Roger Zimmermann, Yuxuan Liang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CL、cs.AI

Comments Accepted by The Web Conference 2024

Modality-Agnostic fMRI Decoding of Vision and Language

Mitja Nikolaus, Milad Mozafari, Nicholas Asher, Leila Reddy, Rufin VanRullen

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments To appear at ICLR 2024 workshop on Representational Alignment (Re-Align)

Synthetic Privileged Information Enhances Medical Image Representation Learning

Lucas Farndale, Chris Walsh, Robert Insall, Ke Yuan

专题命中 多模态训练与对齐 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback

Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, Tat-Seng Chua

专题命中 多模态训练与对齐 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Comments Accepted by CVPR 2024

Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation

Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tianjiao Zhang, Jiangchao Yao, Ya Zhang, Yanfeng Wang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments 14 pages, Accept in NeurIPS 2023

Ferret: Refer and Ground Anything Anywhere at Any Granularity

Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, Yinfei Yang

专题命中 多模态训练与对齐 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Comments 30 pages, 10 figures. Code/Project Website: https://github.com/apple/ml-ferret

ProtoHPE: Prototype-guided High-frequency Patch Enhancement for Visible-Infrared Person Re-identification

Guiwei Zhang, Yongfei Zhang, Zichang Tan

专题命中 多模态训练与对齐 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI