语言大模型 / LLM - arXivDaily 专题

2203.12788 2022-03-25 cs.CL 79%

Evaluating Distributional Distortion in Neural Language Modeling

Benjamin LeBrun, Alessandro Sordoni, Timothy J. O'Donnell

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Journal ref International Conference on Learning Representations. 2022

2203.08928 2022-03-23 cs.CL 79%

C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References

Xiang Yue, Xiaoman Pan, Wenlin Yao, Dian Yu, Dong Yu, Jianshu Chen

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.CL

Comments ACL 2022 Main Conference

2203.11431 2022-03-23 cs.CL 79%

Task-guided Disentangled Tuning for Pretrained Language Models

Jiali Zeng, Yufan Jiang, Shuangzhi Wu, Yongjing Yin, Mu Li

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Findings of ACL 2022

2203.10753 2022-03-22 cs.CL 79%

Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability

Yoshinari Fujinuma, Jordan Boyd-Graber, Katharina Kann

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.CL

Comments ACL 2022 camera ready

2110.06852 2022-03-22 cs.CL 79%

Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects

Go Inoue, Salam Khalifa, Nizar Habash

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Accepted to Findings of ACL 2022

2108.13655 2022-03-21 cs.CL 79%

MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER

Ran Zhou, Xin Li, Ruidan He, Lidong Bing, Erik Cambria, Luo Si, Chunyan Miao

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Accepted at ACL 2022

2110.08484 2022-03-16 cs.CV cs.CL 79%

A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, Xiang Ren

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Accepted to ACL 2022

2110.08443 2022-03-10 cs.CL 79%

Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Wenxuan Zhou, Fangyu Liu, Ivan Vulić, Nigel Collier, Muhao Chen

专题命中预训练与数据：pretraining(title)；language model(abstract)；分类 cs.CL

Comments ACL 2022

2203.03582 2022-03-08 cs.CL cs.SD eess.AS 79%

Improving CTC-based speech recognition via knowledge transferring from pre-trained language models

Keqi Deng, Songjun Cao, Yike Zhang, Long Ma, Gaofeng Cheng, Ji Xu, Pengyuan Zhang

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments ICASSP 2022

2112.01529 2022-03-04 cs.CV cs.LG 79%

BEVT: BERT Pretraining of Video Transformers

Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, Lu Yuan

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.LG

Comments To Appear at CVPR 2022, code is available at https://github.com/xyzforever/BEVT

2202.09955 2022-02-24 cs.CL 79%

StyleBERT: Chinese pretraining by font style information

Chao Lv, Han Zhang, XinKai Du, Yunhao Zhang, Ying Huang, Wenhao Li, Jia Han, Shanshan Gu

专题命中预训练与数据：pretraining(title)；language model(abstract)；分类 cs.CL

2202.09452 2022-02-22 cs.CL 79%

From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French

Simon Gabay, Pedro Ortiz Suarez, Alexandre Bartz, Alix Chagué, Rachel Bawden, Philippe Gambette, Benoît Sagot

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments 8 pages, 2 figures, 4 tables

2202.03514 2022-02-09 cs.SD cs.LG eess.AS 79%

Maximizing Audio Event Detection Model Performance on Small Datasets Through Knowledge Transfer, Data Augmentation, And Pretraining: An Ablation Study

Daniel Tompkins, Kshitiz Kumar, Jian Wu

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.LG

2201.11990 2022-02-07 cs.CL 79%

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, Bryan Catanzaro

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Shaden Smith and Mostofa Patwary contributed equally

2201.08643 2022-01-24 cs.CL 79%

Text Style Transfer for Bias Mitigation using Masked Language Modeling

Ewoenam Kwaku Tokpo, Toon Calders

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments 9 pages, 3 figures, 5 tables

2201.05601 2022-01-19 cs.CL 79%

A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models

Vésteinn Snæbjarnarson, Haukur Barri Símonarson, Pétur Orri Ragnarsson, Svanhvít Lilja Ingólfsdóttir, Haukur Páll Jónsson, Vilhjálmur Þorsteinsson, Hafsteinn Einarsson

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

2112.08653 2021-12-17 cs.CL 79%

Reconsidering the Past: Optimizing Hidden States in Language Models

Davis Yoshida, Kevin Gimpel

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Findings of EMNLP version

Journal ref Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4099-4105

2112.07074 2021-12-15 cs.CV cs.LG 79%

Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text

Qing Li, Boqing Gong, Yin Cui, Dan Kondratyuk, Xianzhi Du, Ming-Hsuan Yang, Matthew Brown

专题命中预训练与数据：foundation model(title,abstract)；分类 cs.LG

Comments preliminary work

2112.06327 2021-12-14 cs.CL 79%

Improving Code-switching Language Modeling with Artificially Generated Texts using Cycle-consistent Adversarial Networks

Chia-Yu Li, Ngoc Thang Vu

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments 4 pages, 1 figure, Interspeech 2020

2109.04650 2021-11-30 cs.CL 79%

What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, Nako Sung

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Accepted to EMNLP2021 as a long paper. Fixed some typos

2101.12727 2021-11-30 cs.CV cs.LG 79%

Surprisingly Simple Semi-Supervised Domain Adaptation with Pretraining and Consistency

Samarth Mishra, Kate Saenko, Venkatesh Saligrama

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.LG

Comments Accepted to BMVC 2021

2111.11646 2021-11-25 cs.CV cs.AI q-bio.QM 79%

CytoImageNet: A large-scale pretraining dataset for bioimage transfer learning

Stanley Bryan Z. Hua, Alex X. Lu, Alan M. Moses

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.AI

Comments Accepted paper at NeurIPS 2021 Learning Meaningful Representations for Life (LMRL) Workshop

2106.10328 2021-11-24 cs.CL cs.CY 79%

Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

Irene Solaiman, Christy Dennison

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Both authors contributed equally. Accepted at NeurIPS 2021

2111.09509 2021-11-19 cs.CL 79%

How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN

R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, Asli Celikyilmaz

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments 10 pages, plus 39 pages of appendices

2102.01243 2021-11-18 cs.SD cs.LG eess.AS 79%

PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation

Yuan Gong, Yu-An Chung, James Glass

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.LG

Comments Published in IEEE/ACM Transactions on Audio Speech and Language Processing. Code at https://github.com/YuanGongND/psla

Journal ref in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3292-3306, 2021

2110.08887 2021-10-19 cs.CL 79%

Schrödinger's Tree -- On Syntax and Neural Language Models

Artur Kulmizev, Joakim Nivre

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments preprint, submitted to Frontiers in Artificial Intelligence: Perspectives for Natural Language Processing between AI, Linguistics and Cognitive Science

1911.01528 2021-10-19 cs.CL 79%

BAS: An Answer Selection Method Using BERT Language Model

Jamshid Mozafari, Afsaneh Fatemi, Mohammad Ali Nematbakhsh

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments 28 pages, 19 figures, 8 tables, Journal of Computing and Security

Journal ref Journal of Computing and Security 8 (2021) 1-18

2110.05896 2021-10-15 cs.CL 79%

LaoPLM: Pre-trained Language Models for Lao

Nankai Lin, Yingwen Fu, Chuwei Chen, Ziyu Yang, Shengyi Jiang

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

2108.07789 2021-10-04 cs.CL cs.SD eess.AS 79%

Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition

Xianrui Zheng, Chao Zhang, Philip C. Woodland

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments To appear in ASRU 2021

2105.09996 2021-10-04 cs.CV cs.CL 79%

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer

专题命中预训练与数据：language model(title)；pretraining(abstract)；分类 cs.CL

Comments 9 pages, ACL Findings 2021

1. 预训练与数据 12393 篇

Evaluating Distributional Distortion in Neural Language Modeling

C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References

Task-guided Disentangled Tuning for Pretrained Language Models

Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability

Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects

MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER

A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Improving CTC-based speech recognition via knowledge transferring from pre-trained language models

BEVT: BERT Pretraining of Video Transformers

StyleBERT: Chinese pretraining by font style information

From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French

Maximizing Audio Event Detection Model Performance on Small Datasets Through Knowledge Transfer, Data Augmentation, And Pretraining: An Ablation Study

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Text Style Transfer for Bias Mitigation using Masked Language Modeling

A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models

Reconsidering the Past: Optimizing Hidden States in Language Models

Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text

Improving Code-switching Language Modeling with Artificially Generated Texts using Cycle-consistent Adversarial Networks

What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

Surprisingly Simple Semi-Supervised Domain Adaptation with Pretraining and Consistency

CytoImageNet: A large-scale pretraining dataset for bioimage transfer learning

Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN

PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation

Schrödinger's Tree -- On Syntax and Neural Language Models

BAS: An Answer Selection Method Using BERT Language Model

LaoPLM: Pre-trained Language Models for Lao

Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding