语言大模型 / LLM - arXivDaily 专题

2212.04267 2023-03-17 cs.CV cs.LG 83%

Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval

Mustafa Shukor, Nicolas Thome, Matthieu Cord

专题命中预训练与数据：pretraining(title,abstract)；foundation model(abstract)；分类 cs.LG

Comments Code: https://github.com/mshukor/VLPCook

2303.02151 2023-03-06 cs.CV cs.CL 83%

Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners

Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, Peng Gao

专题命中预训练与数据：foundation model(title,abstract)；prompting(abstract)；分类 cs.CL

Comments Accepted by CVPR 2023. Code is available at https://github.com/ZrrSkywalker/CaFo. arXiv admin note: substantial text overlap with arXiv:2209.12255

2203.06125 2023-01-31 cs.LG 83%

Protein Representation Learning by Geometric Structure Pretraining

Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, Jian Tang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.LG

2301.11796 2023-01-30 cs.CL 83%

Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus

Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, Chengxu Zhuang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

2301.10444 2023-01-26 cs.IR cs.CL 83%

An Experimental Study on Pretraining Transformers from Scratch for IR

Carlos Lassance, Hervé Déjean, Stéphane Clinchant

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments Accepted at ECIR 2023

2301.01162 2023-01-04 cs.SD cs.CL eess.AS 83%

Language Models are Drummers: Drum Composition with Natural Language Pre-Training

Li Zhang, Chris Callison-Burch

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL

Comments Accepted to the 1st workshop on Creative AI across Modalities in AAAI 2023

2212.10012 2022-12-21 cs.CL 83%

Language Modeling with Latent Situations

Belinda Z. Li, Maxwell Nye, Jacob Andreas

专题命中预训练与数据：language model(title,abstract)；prompting(abstract)；分类 cs.CL

Comments 13 pages, 3 figures, 7 tables

2210.01504 2022-12-20 cs.CL 83%

Knowledge Unlearning for Mitigating Privacy Risks in Language Models

Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, Minjoon Seo

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

2110.12010 2022-12-20 cs.CL 83%

ClimateBert: A Pretrained Language Model for Climate-Related Text

Nicolas Webersinke, Mathias Kraus, Julia Anna Bingler, Markus Leippold

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

2212.05195 2022-12-13 cs.LG 83%

Uniform Masking Prevails in Vision-Language Pretraining

Siddharth Verma, Yuchen Lu, Rui Hou, Hanchao Yu, Nicolas Ballas, Madian Khabsa, Amjad Almahairi

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.LG

2211.13594 2022-11-28 cs.CV cs.AI 83%

Self-supervised vision-language pretraining for Medical visual question answering

Pengfei Li, Gang Liu, Lin Tan, Jinying Liao, Shenjun Zhong

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.AI

Comments 5 pages, 3 figures

2211.00262 2022-11-02 cs.CL cs.CV 83%

Training Vision-Language Models with Less Bimodal Supervision

Elad Segal, Ben Bogin, Jonathan Berant

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments AKBC 2022

2210.15762 2022-10-31 cs.CL 83%

Nearest Neighbor Language Models for Stylistic Controllable Generation

Severino Trotta, Lucie Flek, Charles Welch

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted to GEM workshop at EMNLP 2022

2210.10951 2022-10-27 cs.CL 83%

Automatic Document Selection for Efficient Encoder Pretraining

Yukun Feng, Patrick Xia, Benjamin Van Durme, João Sedoc

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

2205.12697 2022-10-27 cs.CL 83%

PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generation

Ao Liu, Haoyu Dong, Naoaki Okazaki, Shi Han, Dongmei Zhang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments EMNLP'22

2210.12403 2022-10-26 cs.CL 83%

PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models

Yupeng Zhang, Hongzhi Zhang, Sirui Wang, Wei Wu, Zhoujun Li

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted by EMNLP 2022 main conference

2210.05836 2022-10-13 cs.CL 83%

CLIP also Understands Text: Prompting CLIP for Phrase Understanding

An Yan, Jiacheng Li, Wanrong Zhu, Yujie Lu, William Yang Wang, Julian McAuley

专题命中预训练与数据：prompting(title)；language model(abstract)；pretraining(abstract)；分类 cs.CL

Comments Work in progress

2208.13628 2022-10-06 cs.CV cs.LG 83%

Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment

Mustafa Shukor, Guillaume Couairon, Matthieu Cord

专题命中预训练与数据：pretraining(title,abstract)；foundation model(abstract)；分类 cs.LG

Comments BMVC 2022

2204.02311 2022-10-06 cs.CL 83%

PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, Noah Fiedel

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL

2209.11035 2022-09-28 cs.CL 83%

MonoByte: A Pool of Monolingual Byte-level Language Models

Hugo Abonizio, Leandro Rodrigues de Souza, Roberto Lotufo, Rodrigo Nogueira

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

2204.07834 2022-09-22 cs.CL 83%

Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understanding

Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, Dacheng Tao

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

2209.04595 2022-09-13 cs.CL 83%

OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue

Zhi Chen, Yuncong Liu, Lu Chen, Su Zhu, Mengyue Wu, Kai Yu

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments To appear at TACL, 16 pages, 8 figures

2112.06598 2022-09-13 cs.CL 83%

WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

Benjamin Minixhofer, Fabian Paischer, Navid Rekabsaz

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL

Comments NAACL 2022

2208.10442 2022-09-01 cs.CV cs.CL 83%

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, Furu Wei

专题命中预训练与数据：pretraining(title,abstract)；foundation model(abstract)；分类 cs.CL

Comments 18 pages

2205.15575 2022-07-05 cs.CL 83%

hmBERT: Historical Multilingual Language Models for Named Entity Recognition

Stefan Schweter, Luisa März, Katharina Schmid, Erion Çano

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Camera-ready HIPE-2022 Working Note Paper for CLEF 2022 (Conference and Labs of the Evaluation Forum (CLEF 2022))

2112.08696 2022-05-31 cs.CL 83%

Few-Shot Semantic Parsing with Language Models Trained On Code

Richard Shin, Benjamin Van Durme

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL

Comments NAACL 2022

2205.04810 2022-05-11 cs.CL 83%

The Importance of Context in Very Low Resource Language Modeling

Lukas Edman, Antonio Toral, Gertjan van Noord

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

2205.00619 2022-05-03 cs.CL 83%

POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection

Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nick Beauchamp, Lu Wang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments Findings of NAACL'22. The first two authors contribute equally

2204.12052 2022-04-27 cs.CL 83%

Pretraining Chinese BERT for Detecting Word Insertion and Deletion Errors

Cong Zhou, Yong Dai, Duyu Tang, Enbo Zhao, Zhangyin Feng, Li Kuang, Shuming Shi

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments 12 pages

2109.12573 2022-04-26 cs.CL 83%

XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge

Xiaoze Jiang, Yaobo Liang, Weizhu Chen, Nan Duan

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments AAAI-2022

1. 预训练与数据 12393 篇

Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval

Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners

Protein Representation Learning by Geometric Structure Pretraining

Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus

An Experimental Study on Pretraining Transformers from Scratch for IR

Language Models are Drummers: Drum Composition with Natural Language Pre-Training

Language Modeling with Latent Situations

Knowledge Unlearning for Mitigating Privacy Risks in Language Models

ClimateBert: A Pretrained Language Model for Climate-Related Text

Uniform Masking Prevails in Vision-Language Pretraining

Self-supervised vision-language pretraining for Medical visual question answering

Training Vision-Language Models with Less Bimodal Supervision

Nearest Neighbor Language Models for Stylistic Controllable Generation

Automatic Document Selection for Efficient Encoder Pretraining

PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generation

PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models

CLIP also Understands Text: Prompting CLIP for Phrase Understanding

Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment

PaLM: Scaling Language Modeling with Pathways

MonoByte: A Pool of Monolingual Byte-level Language Models

Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understanding

OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue

WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

hmBERT: Historical Multilingual Language Models for Named Entity Recognition

Few-Shot Semantic Parsing with Language Models Trained On Code

The Importance of Context in Very Low Resource Language Modeling

POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection

Pretraining Chinese BERT for Detecting Word Insertion and Deletion Errors

XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge