语言大模型 / LLM - arXivDaily 专题

2202.12024 2022-03-24 cs.CL 83%

NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better

Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, Xing Xie

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments ACL 2022

2103.01819 2022-01-04 cs.CL 83%

The Rediscovery Hypothesis: Language Models Need to Meet Linguistics

Vassilina Nikoulina, Maxat Tezekbayev, Nuradil Kozhakhmet, Madina Babazhanova, Matthias Gallé, Zhenisbek Assylbekov

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Journal ref Journal of Artificial Intelligence Vol. 72 (2021) 1343-1384

2107.00676 2021-12-24 cs.CL 83%

A Primer on Pretrained Multilingual Language Models

Sumanth Doddapaneni, Gowtham Ramesh, Mitesh M. Khapra, Anoop Kunchukuttan, Pratyush Kumar

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

2112.07254 2021-12-15 eess.AS cs.CL cs.SD 83%

Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model

Keqi Deng, Songjun Cao, Yike Zhang, Long Ma

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments ASRU2021

2112.02839 2021-12-07 cs.MM cs.AI 83%

MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering

Fangzhi Xu, Qika Lin, Jun Liu, Lingling Zhang, Tianzhe Zhao, Qi Chai, Yudai Pan

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.AI

Comments 9 pages, 6 figures

2111.06053 2021-11-12 cs.CL 83%

Improving Large-scale Language Models and Resources for Filipino

Jan Christian Blaise Cruz, Charibeth Cheng

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Resources are available at blaisecruz.com/resources

2109.04513 2021-10-26 cs.CL 83%

Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach

Koren Lazar, Benny Saret, Asaf Yehudai, Wayne Horowitz, Nathan Wasserman, Gabriel Stanovsky

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted to EMNLP 2021 (Main Conference)

2110.08182 2021-10-18 cs.CL cs.CV 83%

The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color

Cory Paik, Stéphane Aroca-Ouellette, Alessandro Roncone, Katharina Kann

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted to EMNLP 2021, 9 Pages

2110.07143 2021-10-15 cs.CL 83%

bert2BERT: Towards Reusable Pretrained Language Models

Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, Qun Liu

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL

2110.05301 2021-10-12 cs.CL 83%

On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias

Ting-Rui Chiang

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Work in progress

2007.08426 2021-09-28 cs.CL 83%

Investigating Pretrained Language Models for Graph-to-Text Generation

Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, Iryna Gurevych

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted as a long paper to NLP4ConvAI, EMNLP2021

2109.07301 2021-09-16 cs.CL 83%

What Vision-Language Models `See' when they See Scenes

Michele Cafagna, Kees van Deemter, Albert Gatt

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

2109.06605 2021-09-15 cs.CL 83%

MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model

Rasmus Kær Jørgensen, Mareike Hartmann, Xiang Dai, Desmond Elliott

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments Findings of EMNLP 2021

2109.00055 2021-09-14 cs.CL 83%

Sentence Bottleneck Autoencoders from Transformer Language Models

Ivan Montero, Nikolaos Pappas, Noah A. Smith

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

2012.15562 2021-09-13 cs.CL 83%

UNKs Everywhere: Adapting Multilingual Language Models to New Scripts

Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, Sebastian Ruder

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments EMNLP 2021

2109.03646 2021-09-09 cs.CL 83%

Sustainable Modular Debiasing of Language Models

Anne Lauscher, Tobias Lüken, Goran Glavaš

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted for EMNLP-Findings 2021

2109.03630 2021-09-09 cs.CL 83%

Discrete and Soft Prompting for Multilingual Models

Mengjie Zhao, Hinrich Schütze

专题命中预训练与数据：prompting(title,abstract)；language model(abstract)；分类 cs.CL

Comments EMNLP 2021

2101.00406 2021-09-06 cs.CL 83%

CDLM: Cross-Document Language Modeling

Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E. Peters, Arie Cattan, Ido Dagan

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments EMNLP 2021, findings

2108.12226 2021-08-30 cs.CL cs.SD eess.AS 83%

Injecting Text in Self-Supervised Speech Pretraining

Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Gary Wang, Pedro Moreno

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments submit to ASRU 2021

2106.05634 2021-06-11 cs.CL 83%

Exploring Unsupervised Pretraining Objectives for Machine Translation

Christos Baziotis, Ivan Titov, Alexandra Birch, Barry Haddow

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments Findings of ACL 2021

2012.15613 2021-06-03 cs.CL 83%

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, Iryna Gurevych

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments ACL 2021

2012.14763 2021-06-01 cs.CL 83%

CMV-BERT: Contrastive multi-vocab pretraining of BERT

Wei Zhu, Daniel Cheung

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments will add more detailed technical contents, and more detailed experiments

2105.00572 2021-05-04 cs.CL 83%

Larger-Scale Transformers for Multilingual Masked Language Modeling

Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, Alexis Conneau

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments 4 pages

2010.12858 2021-04-20 cs.CL 83%

When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models

Benjamin Muller, Antonis Anastasopoulos, Benoît Sagot, Djamé Seddah

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted at NAACL-HLT 2021

2103.10531 2021-04-15 cs.CL 83%

Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation

Alexandra Chronopoulou, Dario Stojanovski, Alexander Fraser

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted at NAACL 2021

2010.12547 2021-04-13 cs.CL 83%

Multilingual BERT Post-Pretraining Alignment

Lin Pan, Chung-Wei Hang, Haode Qi, Abhishek Shah, Saloni Potdar, Mo Yu

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments Accepted at NAACL2021

2011.08539 2020-11-18 cs.CL 83%

MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining

Wei Zhu

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

2011.04946 2020-11-11 cs.CL 83%

When Do You Need Billions of Words of Pretraining Data?

Yian Zhang, Alex Warstadt, Haau-Sing Li, Samuel R. Bowman

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments 10 pages, 6 figures

2011.03203 2020-11-09 cs.CL 83%

Unleashing the Power of Neural Discourse Parsers -- A Context and Structure Aware Approach Using Large Scale Pretraining

Grigorii Guz, Patrick Huber, Giuseppe Carenini

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments 10 pages, 1 figure, COLING 2020

2010.02480 2020-10-30 cs.CL 83%

Pretrained Language Model Embryology: The Birth of ALBERT

Cheng-Han Chiang, Sung-Feng Huang, Hung-yi Lee

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted to EMNLP 2020, short paper

1. 预训练与数据 12379 篇

NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better

The Rediscovery Hypothesis: Language Models Need to Meet Linguistics

A Primer on Pretrained Multilingual Language Models

Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model

MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering

Improving Large-scale Language Models and Resources for Filipino

Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach

The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color

bert2BERT: Towards Reusable Pretrained Language Models

On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias

Investigating Pretrained Language Models for Graph-to-Text Generation

What Vision-Language Models `See' when they See Scenes

MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model

Sentence Bottleneck Autoencoders from Transformer Language Models

UNKs Everywhere: Adapting Multilingual Language Models to New Scripts

Sustainable Modular Debiasing of Language Models

Discrete and Soft Prompting for Multilingual Models

CDLM: Cross-Document Language Modeling

Injecting Text in Self-Supervised Speech Pretraining

Exploring Unsupervised Pretraining Objectives for Machine Translation

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

CMV-BERT: Contrastive multi-vocab pretraining of BERT

Larger-Scale Transformers for Multilingual Masked Language Modeling

When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models

Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation

Multilingual BERT Post-Pretraining Alignment

MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining

When Do You Need Billions of Words of Pretraining Data?

Unleashing the Power of Neural Discourse Parsers -- A Context and Structure Aware Approach Using Large Scale Pretraining

Pretrained Language Model Embryology: The Birth of ALBERT