多模态大模型 - arXivDaily 专题

2408.08632 2024-09-09 cs.CL cs.AI cs.CV 82%

A Survey on Benchmarks of Multimodal Large Language Models

Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, Ying Tai, Wankou Yang, Yabiao Wang, Chengjie Wang

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2401.08396 2024-09-04 cs.CV cs.AI cs.CL 82%

Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine

Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, Justin M. Cheung, Robert Chen, Ronald M. Summers, Justin F. Rousseau, Peiyun Ni, Marc J Landsman, Sally L. Baxter, Subhi J. Al'Aref, Yijia Li, Alex Chen, Josef A. Brejt, Michael F. Chiang, Yifan Peng, Zhiyong Lu

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Journal ref npj Digital Medicine, 2024

2408.13742 2024-08-29 cs.RO 82%

Multi-modal Integrated Prediction and Decision-making with Adaptive Interaction Modality Explorations

Tong Li, Lu Zhang, Sikang Liu, Shaojie Shen

专题命中多模态评测：multi-modal(title,abstract)；multimodal(abstract)

Comments 8 pages, 9 figures

2408.14964 2024-08-28 cs.LG 82%

Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learning

Sakhinana Sagar Srinivas, Venkataramana Runkana

专题命中多模态评测：cross-modal(title,abstract)；multi-modal(abstract)

Comments Paper Accepted at Workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models at NeurIPS 2023

2403.11289 2024-08-23 cs.RO 82%

ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models

Siyuan Huang, Iaroslav Ponomarenko, Zhengkai Jiang, Xiaoqi Li, Xiaobin Hu, Peng Gao, Hongsheng Li, Hao Dong

专题命中多模态评测：multi-modal(title,abstract)；image-text(abstract)

Comments Code and dataset are publicly available at https://github.com/SiyuanHuang95/ManipVQA. Accepted by IROS2024

2408.11237 2024-08-22 cs.AI cs.CL cs.CV cs.LG 82%

Out-of-Distribution Detection with Attention Head Masking for Multimodal Document Classification

Christos Constantinou, Georgios Ioannides, Aman Chadha, Aaron Elkins, Edwin Simpson

专题命中多模态评测：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

2407.19832 2024-08-22 cs.CV cs.AI cs.CL 82%

ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2

Wenjun Huang, Jiakai Pan, Jiahao Tang, Yanyu Ding, Yifei Xing, Yuhe Wang, Zhengzhuo Wang, Jianguo Hu

专题命中多模态评测：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2403.14624 2024-08-20 cs.CV cs.AI cs.CL cs.LG 82%

MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?

Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, Hongsheng Li

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by ECCV 2024, 46 Pages, Benchmark Project Page: https://mathverse-cuhk.github.io

2402.15745 2024-08-07 cs.CL cs.AI cs.CV 82%

GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation

Yi Zong, Xipeng Qiu

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2407.19192 2024-07-30 cs.CL cs.CV cs.MM 82%

Harmfully Manipulated Images Matter in Multimodal Misinformation Detection

Bing Wang, Shengsheng Wang, Changchun Li, Renchu Guan, Ximing Li

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted by ACM MM 2024. Code: https://github.com/wangbing1416/HAMI-M3D

2402.12750 2024-07-29 cs.CV cs.AI cs.CL 82%

Model Composition for Multimodal Large Language Models

Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, Yang Liu

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL2024 Main Conference; Code is available at https://github.com/THUNLP-MT/ModelCompose

2407.15680 2024-07-23 cs.CV cs.AI cs.LG cs.MM 82%

HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning

Zhecan Wang, Garrett Bingham, Adams Yu, Quoc Le, Thang Luong, Golnaz Ghiasi

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted as a main conference paper at ECCV 2024 (https://github.com/google/haloquest)

2404.15592 2024-07-23 cs.CV cs.AI cs.CL cs.IR cs.LG 82%

ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction

Henry Peng Zou, Vinay Samuel, Yue Zhou, Weizhi Zhang, Liancheng Fang, Zihe Song, Philip S. Yu, Cornelia Caragea

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by ACL 2024 (Findings) - Scores: Soundness - 4/4/4, Dataset - 4/4/4, Overall Assessment - 4/3.5/3.5, Meta - 4

2402.17553 2024-07-23 cs.AI cs.CL cs.CV cs.HC 82%

OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web

Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, Ruslan Salakhutdinov

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2407.09705 2024-07-16 cs.CV cs.AI cs.MM 82%

Diagnosing and Re-learning for Balanced Multimodal Learning

Yake Wei, Siwei Li, Ruoxuan Feng, Di Hu

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by ECCV 2024

2405.13049 2024-07-09 cs.CL cs.AI cs.MM 82%

SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations

Fanfan Wang, Heqing Ma, Jianfei Yu, Rui Xia, Erik Cambria

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted to the 18th International Workshop on Semantic Evaluation (SemEval-2024). 12 pages, 3 figures, 4 Tables

Journal ref https://aclanthology.org/2024.semeval-1.277/

2407.01284 2024-07-02 cs.AI cs.CL cs.CV cs.LG cs.SC 82%

We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, Honggang Zhang

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Work in progress

2407.00024 2024-07-02 cs.CV cs.AI cs.MM 82%

LMVD: A Large-Scale Multimodal Vlog Dataset for Depression Detection in the Wild

Lang He, Kai Chen, Junnan Zhao, Yimeng Wang, Ercheng Pei, Haifeng Chen, Jiewei Jiang, Shiqing Zhang, Jie Zhang, Zhongmin Wang, Tao He, Prayag Tiwari

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2401.15847 2024-06-28 cs.CV cs.AI cs.CL 82%

Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQA

Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Xinze Guan, Xin Eric Wang

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2024

2406.17806 2024-06-27 cs.CL cs.AI cs.CR cs.CV cs.LG 82%

MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?

Xirui Li, Hengguang Zhou, Ruochen Wang, Tianyi Zhou, Minhao Cheng, Cho-Jui Hsieh

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2406.11217 2024-06-25 cs.AI cs.CL cs.CV physics.ao-ph 82%

WeatherQA: Can Multimodal Language Models Reason about Severe Weather?

Chengqian Ma, Zhanxiang Hua, Alexandra Anderson-Frey, Vikram Iyer, Xin Liu, Lianhui Qin

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 26 pages, 9 figures

2403.05680 2024-06-19 cs.AI cs.CL cs.CV 82%

How Well Do Multi-modal LLMs Interpret CT Scans? An Auto-Evaluation Framework for Analyses

Qingqing Zhu, Benjamin Hou, Tejas S. Mathai, Pritam Mukherjee, Qiao Jin, Xiuying Chen, Zhizheng Wang, Ruida Cheng, Ronald M. Summers, Zhiyong Lu

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2406.12321 2024-06-19 cs.AI cs.CL cs.CV 82%

Automatic benchmarking of large multimodal models via iterative experiment programming

Alessandro Conti, Enrico Fini, Paolo Rota, Yiming Wang, Massimiliano Mancini, Elisa Ricci

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 31 pages, 6 figures, code is available at https://github.com/altndrr/apex

2403.05396 2024-06-19 cs.CV cs.AI cs.CL cs.LG 82%

HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction

Zhengrui Guo, Jiabo Ma, Yingxue Xu, Yihui Wang, Liansheng Wang, Hao Chen

专题命中多模态评测：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by MICCAI2024

2309.06255 2024-06-17 cs.CV cs.AI cs.LG cs.MM 82%

Enhancing multimodal cooperation via sample-level modality valuation

Yake Wei, Ruoxuan Feng, Zihe Wang, Di Hu

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by CVPR 2024

2406.02208 2024-06-05 cs.CV cs.AI cs.CL 82%

Why Only Text: Empowering Vision-and-Language Navigation with Multi-modal Prompts

Haodong Hong, Sen Wang, Zi Huang, Qi Wu, Jiajun Liu

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments IJCAI 2024

2402.03190 2024-05-28 cs.CL cs.AI cs.IR cs.LG cs.MM 82%

Unified Hallucination Detection for Multimodal Large Language Models

Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted by ACL 2024 (main conference)

2405.16473 2024-05-28 cs.CV cs.AI cs.CL 82%

M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought

Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, Wanxiang Che

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at ACL2024 Main Conference

2401.14011 2024-05-09 cs.CL cs.AI cs.MM 82%

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, Hua Huang

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

2405.02287 2024-05-06 cs.CL cs.AI cs.CV 82%

Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models

Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, Ethan Yeo, Eugenie Lamprecht, Qi Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng, Cyprien de Masson d'Autume, Dani Yogatama, Mikel Artetxe, Yi Tay

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

1. 多模态评测 9111 篇

A Survey on Benchmarks of Multimodal Large Language Models

Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine

Multi-modal Integrated Prediction and Decision-making with Adaptive Interaction Modality Explorations

Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learning

ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models

Out-of-Distribution Detection with Attention Head Masking for Multimodal Document Classification

ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2

MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?

GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation

Harmfully Manipulated Images Matter in Multimodal Misinformation Detection

Model Composition for Multimodal Large Language Models

HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning

ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction

OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web

Diagnosing and Re-learning for Balanced Multimodal Learning

SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations

We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

LMVD: A Large-Scale Multimodal Vlog Dataset for Depression Detection in the Wild

Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQA

MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?

WeatherQA: Can Multimodal Language Models Reason about Severe Weather?

How Well Do Multi-modal LLMs Interpret CT Scans? An Auto-Evaluation Framework for Analyses

Automatic benchmarking of large multimodal models via iterative experiment programming

HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction

Enhancing multimodal cooperation via sample-level modality valuation

Why Only Text: Empowering Vision-and-Language Navigation with Multi-modal Prompts

Unified Hallucination Detection for Multimodal Large Language Models

M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models