多模态大模型

Uni-SMART: Universal Science Multimodal Analysis and Research Transformer

Hengxing Cai, Xiaochen Cai, Shuwen Yang, Jiankun Wang, Lin Yao, Zhifeng Gao, Junhan Chang, Sihang Li, Mingjun Xu, Changxin Wang, Hongshuai Wang, Yongge Li, Mujie Lin, Yaqi Li, Yuqi Yin, Linfeng Zhang, Guolin Ke

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

Gagan Bhatia, El Moatez Billah Nagoudi, Hasan Cavusoglu, Muhammad Abdul-Mageed

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers

Anirudh Sundar, Jin Xu, William Gay, Christopher Richardson, Larry Heck

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 14 pages, 1 figure

FlexLoc: Conditional Neural Networks for Zero-Shot Sensor Perspective Invariance in Object Localization with Distributed Multimodal Sensors

Jason Wu, Ziqi Wang, Xiaomin Ouyang, Ho Lyun Jeong, Colin Samplawski, Lance Kaplan, Benjamin Marlin, Mani Srivastava

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

BloomVQA: Assessing Hierarchical Multi-modal Comprehension

Yunye Gong, Robik Shrestha, Jared Claypoole, Michael Cogswell, Arijit Ray, Christopher Kanan, Ajay Divakaran

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by ACL Findings (2024). Dataset available at https://huggingface.co/datasets/ygong/BloomVQA

Integrating Text and Image Pre-training for Multi-modal Algorithmic Reasoning

Zijian Zhang, Wei Liu

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Evaluation of data inconsistency for multi-modal sentiment analysis

Yufei Wang, Mengyue Wu

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models

Fuwen Luo, Chi Chen, Zihao Wan, Zhaolu Kang, Qidong Yan, Yingjie Li, Xiaolong Wang, Siyu Wang, Ziyue Wang, Xiaoyue Mi, Peng Li, Ning Ma, Maosong Sun, Yang Liu

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models

Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, Qi Liu

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Project page: https://mm-arxiv.github.io, Camera Ready Version of ACL 2024

Lumos : Empowering Multimodal LLMs with Scene Text Recognition

Ashish Shenoy, Yichao Lu, Srihari Jayakumar, Debojeet Chatterjee, Mohsen Moslehpour, Pierce Chuang, Abhay Harpale, Vikas Bhardwaj, Di Xu, Shicong Zhao, Longfang Zhao, Ankit Ramchandani, Xin Luna Dong, Anuj Kumar

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to KDD 2024 (ADS Track)

MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception

Yuhao Wang, Yusheng Liao, Heyang Liu, Hongcheng Liu, Yu Wang, Yanfeng Wang

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

NoiseBoost: Alleviating Hallucination with Noise Perturbation for Multimodal Large Language Models

Kai Wu, Boyuan Jiang, Zhengkai Jiang, Qingdong He, Donghao Luo, Shengzhi Wang, Qingwen Liu, Chengjie Wang

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 14 pages, 5 figures with supplementary material

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 51 pages, 21 figures

Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks

Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, Muhammad Abdul-Mageed

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving

Jiaxin Zhang, Zhongzhi Li, Mingliang Zhang, Fei Yin, Chenglin Liu, Yashar Moshfeghi

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted in ACL 2024 Findings

PolyGlotFake: A Novel Multilingual and Multimodal DeepFake Dataset

Yang Hou, Haitao Fu, Chuankai Chen, Zida Li, Haoyu Zhang, Jianjun Zhao

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.AI、eess.AS

Comments 13 page, 4 figures

Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots

Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, Ping Luo

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Exploring the Capabilities of Large Multimodal Models on Dense Text

Shuo Zhang, Biao Yang, Zhang Li, Zhiyin Ma, Yuliang Liu, Xiang Bai

专题命中 多模态评测 ：multimodal(title)；multi-modal(abstract)；分类 cs.CL、cs.AI

Multi-Prompt with Depth Partitioned Cross-Modal Learning

Yingjie Tian, Yiqi Wang, Xianda Guo, Zheng Zhu, Long Chen

专题命中 多模态评测 ：cross-modal(title)；multi-modal(abstract)；分类 cs.CV、cs.AI

3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset

Xinyu Ma, Xuebo Liu, Derek F. Wong, Jun Rao, Bei Li, Liang Ding, Lidia S. Chao, Dacheng Tao, Min Zhang

专题命中 多模态评测 ：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.AI

VideoXum: Cross-modal Visual and Textural Summarization of Videos

Jingyang Lin, Hang Hua, Ming Chen, Yikang Li, Jenhao Hsiao, Chiuman Ho, Jiebo Luo

专题命中 多模态评测 ：cross-modal(title,abstract)；分类 cs.CV、cs.CL

Comments 13 pages, 7 figures

Journal ref IEEE Transactions on Multimedia, VOL. 26 (2024) 5548-5560

Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images

Ali Naseh, Katherine Thai, Mohit Iyyer, Amir Houmansadr

专题命中 多模态评测 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL

Towards Efficient Resume Understanding: A Multi-Granularity Multi-Modal Pre-Training Approach

Feihu Jiang, Chuan Qin, Jingshuai Zhang, Kaichun Yao, Xi Chen, Dazhong Shen, Chen Zhu, Hengshu Zhu, Hui Xiong

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments ICME 2024 Accepted

MultiCorrupt: A Multi-Modal Robustness Dataset and Benchmark of LiDAR-Camera Fusion for 3D Object Detection

Till Beemelmanns, Quan Zhang, Christian Geller, Lutz Eckstein

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments Code: https://github.com/ika-rwth-aachen/MultiCorrupt

What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models

Letian Zhang, Xiaotong Zhai, Zhongkai Zhao, Yongshuo Zong, Xin Wen, Bingchen Zhao

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning

Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, Dong Yu

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to NAACL 2024

MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting

Avinash Anand, Janak Kapuriya, Apoorv Singh, Jay Saraf, Naman Lal, Astha Verma, Rushali Gupta, Rajiv Shah

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?

Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, Xiang Yue

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Interpretable Multimodal Learning for Cardiovascular Hemodynamics Assessment

Prasun C Tripathi, Sina Tabakhi, Mohammod N I Suvon, Lawrence Schöb, Samer Alabed, Andrew J Swift, Shuo Zhou, Haiping Lu

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

TinyVQA: Compact Multimodal Deep Neural Network for Visual Question Answering on Resource-Constrained Devices

Hasib-Al Rashid, Argho Sarkar, Aryya Gangopadhyay, Maryam Rahnemoonfar, Tinoosh Mohsenin

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted as a full paper by the tinyML Research Symposium 2024