语言大模型 / LLM

What Does it Mean for a Language Model to Preserve Privacy?

Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, Florian Tramèr

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 21 pages, 2 figures

Unified Scaling Laws for Routed Language Models

Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack Rae, Erich Elsen, Koray Kavukcuoglu, Karen Simonyan

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Fixing typos and affiliation clarity

HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic Analysis

Wenjun Qiu, Yang Xu

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Improving language models by retrieving from trillions of tokens

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, Laurent Sifre

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Fix incorrect reported numbers in Table 14

A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model

Xin Sun, Tao Ge, Shuming Ma, Jingjing Li, Furu Wei, Houfeng Wang

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, Geoffrey Irving

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments 120 pages

Formal Analysis of Art: Proxy Learning of Visual Concepts from Style Through Language Models

Diana Kim, Ahmed Elgammal, Marian Mazzone

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 23 pages, This paper is an extended version of a paper that will be published at the 36th AAAI Conference on Artificial Intelligence, to beheld in Vancouver, BC, Canada, February 22 - March 1, 2022

Supervised Graph Contrastive Pretraining for Text Classification

Samujjwal Ghosh, Subhadeep Maji, Maunendra Sankar Desarkar

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments A condensed version of this paper has been accepted to ACM SAC'22. DOI: https://doi.org/10.1145/3477314.3507194

RegionCLIP: Region-based Language-Image Pretraining

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, Jianfeng Gao

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Comments Technical report

DPRK-BERT: The Supreme Language Model

Arda Akdemir, Yeojoo Jeon

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Mitigating harm in language models with conditional-likelihood filtration

Helen Ngo, Cooper Raterink, João G. M. Araújo, Ivan Zhang, Carol Chen, Adrien Morisot, Nicholas Frosst

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Extraction of Medication Names from Twitter Using Augmentation and an Ensemble of Language Models

Igor Kulev, Berkay Köprü, Raul Rodriguez-Esteban, Diego Saldana, Yi Huang, Alessandro La Torraca, Elif Ozkirimli

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Proceedings of the BioCreative VII Challenge Evaluation Workshop

When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

Lijie Fan, Sijia Liu, Pin-Yu Chen, Gaoyuan Zhang, Chuang Gan

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Comments NeurIPS 2021. Code is available at https://github.com/LijieFan/AdvCL

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

Junyang Lin, An Yang, Jinze Bai, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Yong Li, Wei Lin, Jingren Zhou, Hongxia Yang

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments 14 pages, 4 figures

Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning

Shaohua Wu, Xudong Zhao, Tong Yu, Rongguo Zhang, Chong Shen, Hongli Liu, Feng Li, Hong Zhu, Jiangang Luo, Liang Xu, Xuanwei Zhang

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Pretraining & Reinforcement Learning: Sharpening the Axe Before Cutting the Tree

Saurav Kadavath, Samuel Paradis, Brian Yao

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Does Vision-and-Language Pretraining Improve Lexical Grounding?

Tian Yun, Chen Sun, Ellie Pavlick

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.AI

Comments Camera ready for Findings of EMNLP 2021

PalmTree: Learning an Assembly Language Model for Instruction Embedding

Xuezixiang Li, Qu Yu, Heng Yin

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.AI、cs.LG

Does Pretraining for Summarization Require Knowledge Transfer?

Kundan Krishna, Jeffrey Bigham, Zachary C. Lipton

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments Camera-ready for Findings of EMNLP 2021

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments To appear at EMNLP 2021; 26 pages total (9 main, 6 reference and 11 Appendix)

On the Multilingual Capabilities of Very Large-Scale English Language Models

Jordi Armengol-Estapé, Ona de Gibert Bonet, Maite Melero

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations

Josh Beal, Hao-Yu Wu, Dong Huk Park, Andrew Zhai, Dmitry Kislyuk

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted by WACV 2022

Analyzing Information Leakage of Updates to Natural Language Models

Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, Marc Brockschmidt

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining

Machel Reid, Mikel Artetxe

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments Preprint

Generative Pretraining for Paraphrase Evaluation

Jack Weston, Raphael Lenain, Udeepa Meepegama, Emil Fristed

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments Under review

Calibrate Before Use: Improving Few-Shot Performance of Language Models

Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, Sameer Singh

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments ICML 2021

Meta-learning for downstream aware and agnostic pretraining

Hongyin Luo, Shuyan Dong, Yung-Sung Chuang, Shang-Wen Li

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.AI

Comments Extended abstract

Journal ref Meta Learning and Its Applications to Natural Language Processing workshop at ACL 2021

MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models

Hari Sowrirajan, Jingbo Yang, Andrew Y. Ng, Pranav Rajpurkar

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted at Medical Imaging with Deep Learning (MIDL) Conference 2021

Benchmarking down-scaled (not so large) pre-trained language models

M. Aßenmacher, P. Schulze, C. Heumann

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 14 pages, 5 figures

Mitigating Political Bias in Language Models Through Reinforced Calibration

Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, Soroush Vosoughi

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments In proceedings of the 35th AAAI Conference on Artificial Intelligence