语言大模型 / LLM

Improving BERT Pretraining with Syntactic Supervision

Giorgos Tziafas, Konstantinos Kogkalidis, Gijs Wijnholds, Michael Moortgat

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments 4 pages, rejected by IWCS due to "not fitting the conference theme"

Progressive Generation of Long Text with Pretrained Language Models

Bowen Tan, Zichao Yang, Maruan AI-Shedivat, Eric P. Xing, Zhiting Hu

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments NAACL 2021, Code available at https://github.com/tanyuqian/progressive-generation

ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding

Dongling Xiao, Yu-Kun Li, Han Zhang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Accepted by NAACL-HLT 2021. Codes will be released at https://github.com/PaddlePaddle/ERNIE

ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding

Minjeong Kim, Gyuwan Kim, Sang-Woo Lee, Jung-Woo Ha

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments ICASSP 2021; 5 pages

From Universal Language Model to Downstream Task: Improving RoBERTa-Based Vietnamese Hate Speech Detection

Quang Huu Pham, Viet Anh Nguyen, Linh Bao Doan, Ngoc N. Tran, Ta Minh Thanh

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Published in 2020 12th International Conference on Knowledge and Systems Engineering (KSE)

Journal ref 2020 12th International Conference on Knowledge and Systems Engineering (KSE), Can Tho, Vietnam, 2020, pp. 37-42

Training Data Leakage Analysis in Language Models

Huseyin A. Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle, James Withers, Robert Sim

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Representation Matters: Offline Pretraining for Sequential Decision Making

Mengjiao Yang, Ofir Nachum

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

A Hybrid Task-Oriented Dialog System with Domain and Task Adaptive Pretraining

Boliang Zhang, Ying Lyu, Ning Ding, Tianhao Shen, Zhaoyang Jia, Kun Han, Kevin Knight

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.AI

Task Adaptive Pretraining of Transformers for Hostility Detection

Tathagata Raha, Sayar Ghosh Roy, Ujwal Narayan, Zubair Abid, Vasudeva Varma

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments To be published in: Proceedings of the First Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situation (CONSTRAINT) at AAAI 2021

Critical Thinking for Language Models

Gregor Betz, Christian Voigt, Kyle Richardson

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Pre-trained language models as knowledge bases for Automotive Complaint Analysis

V. D. Viellieber, M. Aßenmacher

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 5 pages

German's Next Language Model

Branden Chan, Stefan Schweter, Timo Möller

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Accepted by COLING2020

Unigram-Normalized Perplexity as a Language Model Performance Measure with Different Vocabulary Sizes

Jihyeon Roh, Sang-Hoon Oh, Soo-Young Lee

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Multimodal Pretraining for Dense Video Captioning

Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, Radu Soricut

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments AACL-IJCNLP 2020

E.T.: Entity-Transformers. Coreference augmented Neural Language Model for richer mention representations via Entity-Transformer blocks

Nikolaos Stylianou, Ioannis Vlahavas

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 10 pages, 4 figures, 5 tables, accepted at CRAC2020

Differentially Private Language Models Benefit from Public Pre-training

Gavin Kerrigan, Dylan Slack, Jens Tuyls

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

Seyone Chithrananda, Gabriel Grand, Bharath Ramsundar

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments Submitted to NeurIPS 2020 ML for Molecules Workshop

Text Classification Using Label Names Only: A Language Model Self-Training Approach

Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, Jiawei Han

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments EMNLP 2020. (Code: https://github.com/yumeng5/LOTClass)

Masked ELMo: An evolution of ELMo towards fully contextual RNN language models

Gregory Senay, Emmanuelle Salin

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

On the importance of pre-training data volume for compact language models

Vincent Micheli, Martin d'Hoffschmidt, François Fleuret

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments EMNLP 2020; typo corrected

Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes

Shuai Zheng, Haibin Lin, Sheng Zha, Mu Li

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments Technical Report (not under reviewed in any venue)

Current Limitations of Language Models: What You Need is Retrieval

Aran Komatsuzaki

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

The Go Transformer: Natural Language Modeling for Game Play

Matthew Ciolino, David Noever, Josh Kalin

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 8 Pages, 5 Figures, 1 Table, IEEE Format, Ai4i 2020

Variance-reduced Language Pretraining via a Mask Proposal Network

Liang Chen

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Language Modelling for Source Code with Transformer-XL

Thomas Dowdell, Hongyu Zhang

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Language Models as Fact Checkers?

Nayeon Lee, Belinda Z. Li, Sinong Wang, Wen-tau Yih, Hao Ma, Madian Khabsa

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted in FEVER Workshop (ACL2020)

Applying GPGPU to Recurrent Neural Network Language Model based Fast Network Search in the Real-Time LVCSR

Kyungmin Lee, Chiyoun Park, Ilhwan Kim, Namhoon Kim, Jaewon Lee

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 4 pages, 2 figures, Interspeech2015(Accepted)

BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining

Zachariah Zhang, Jingshu Liu, Narges Razavian

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Empirical Evaluation of Pretraining Strategies for Supervised Entity Linking

Thibault Févry, Nicholas FitzGerald, Livio Baldini Soares, Tom Kwiatkowski

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments 11 pages, 8 figures, appearing at AKBC 2020

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

Pengcheng Yin, Graham Neubig, Wen-tau Yih, Sebastian Riedel

专题命中 预训练与数据 ：pretraining(title)；language model(abstract)；分类 cs.CL、cs.LG

Comments To Appear at ACL 2020