语言大模型 / LLM - arXivDaily 专题

2209.10372 2023-05-17 cs.CL cs.AI 84%

WeLM: A Well-Read Pre-trained Language Model for Chinese

Hui Su, Xiao Zhou, Houjin Yu, Xiaoyu Shen, Yuwen Chen, Zilin Zhu, Yang Yu, Jie Zhou

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2211.00151 2023-05-09 cs.CL cs.LG 84%

A Close Look into the Calibration of Pre-trained Language Models

Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, Heng Ji

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted to ACL 2023 main conference. Code is available at: https://github.com/lifan-yuan/PLMCalibration

2305.01550 2023-05-03 cs.CL cs.CR cs.LG 84%

Mitigating Approximate Memorization in Language Models via Dissimilarity Learned Policy

Aly M. Kassem

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2304.14275 2023-04-28 cs.CL cs.LG 84%

What's in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models through User-Provided Names in CAD Files

Peter Meltzer, Joseph G. Lambourne, Daniele Grandi

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2304.10946 2023-04-24 cs.CL cs.LG q-bio.BM 84%

CancerGPT: Few-shot Drug Pair Synergy Prediction using Large Pre-trained Language Models

Tianhao Li, Sandesh Shetty, Advaith Kamath, Ajay Jaiswal, Xianqian Jiang, Ying Ding, Yejin Kim

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.LG

2304.08865 2023-04-19 cs.CL cs.LG 84%

Romanization-based Large-scale Adaptation of Multilingual Language Models

Sukannya Purkayastha, Sebastian Ruder, Jonas Pfeiffer, Iryna Gurevych, Ivan Vulić

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 9 pages, 5 figures

2202.07206 2023-03-17 cs.CL cs.LG 84%

Impact of Pretraining Term Frequencies on Few-Shot Reasoning

Yasaman Razeghi, Robert L. Logan, Matt Gardner, Sameer Singh

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2110.03501 2023-03-15 stat.ML cs.CL cs.LG 84%

Pretrained Language Models are Symbolic Mathematics Solvers too!

Kimia Noorbakhsh, Modar Sulaiman, Mahdi Sharifi, Kallol Roy, Pooyan Jamshidi

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

2303.02995 2023-03-07 cs.CV cs.CL cs.LG 84%

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, Yongfeng Zhang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted at ICLR 2023

2202.07646 2023-03-07 cs.LG cs.CL 84%

Quantifying Memorization Across Neural Language Models

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, Chiyuan Zhang

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2206.07699 2023-02-20 cs.CV cs.CL cs.LG 84%

Write and Paint: Generative Vision-Language Models are Unified Modal Learners

Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, Jiawei Wang

专题命中预训练与数据：language model(title,abstract)；foundation model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2023

2210.01296 2023-02-17 cs.CL cs.AI 84%

Recitation-Augmented Language Models

Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, Denny Zhou

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICLR 2023. v2 adds the Codex results

2302.07735 2023-02-16 cs.CL cs.AI cs.CR 84%

Targeted Attack on GPT-Neo for the SATML Language Model Data Extraction Challenge

Ali Al-Kaswan, Maliheh Izadi, Arie van Deursen

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2212.14034 2022-12-29 cs.CL cs.LG 84%

Cramming: Training a Language Model on a Single GPU in One Day

Jonas Geiping, Tom Goldstein

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 22 pages, we provide code at https://github.com/JonasGeiping/cramming

2208.06049 2022-12-21 cs.CV cs.CL cs.LG 84%

MILAN: Masked Image Pretraining on Language Assisted Representation

Zejiang Hou, Fei Sun, Yen-Kuang Chen, Yuan Xie, Sun-Yuan Kung

专题命中预训练与数据：pretraining(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments add new experiments and improved results. provide repo link

2202.06539 2022-12-21 cs.CR cs.CL cs.LG 84%

Deduplicating Training Data Mitigates Privacy Risks in Language Models

Nikhil Kandpal, Eric Wallace, Colin Raffel

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments ICML 2022 Camera Ready Version

2211.17201 2022-12-01 cs.CL cs.LG math.OC 84%

ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

Rui Pan, Shizhe Diao, Jianlin Chen, Tong Zhang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2112.10668 2022-11-11 cs.CL cs.AI 84%

Few-shot Learning with Multilingual Language Models

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li

专题命中预训练与数据：language model(title,abstract)；prompting(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2022; 34 pages

2201.06910 2022-11-01 cs.LG cs.CL 84%

ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization

Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, Zhilin Yang

专题命中预训练与数据：pretraining(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments 18 pages

2111.10962 2022-10-20 cs.CL cs.AI 84%

Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples

Linlin Liu, Xin Li, Ruidan He, Lidong Bing, Shafiq Joty, Luo Si

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP 2022

2208.02812 2022-10-13 cs.CV cs.AI cs.LG 84%

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, Jiwen Lu

专题命中预训练与数据：prompting(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments Accepted to NeurIPS 2022, project page: https://p2p.ivg-research.xyz

2202.04538 2022-10-13 cs.CL cs.LG 84%

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

Yu Meng, Jiaxin Huang, Yu Zhang, Jiawei Han

专题命中预训练与数据：language model(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2022. (Code: https://github.com/yumeng5/SuperGen)

2209.01712 2022-09-07 cs.LG cs.AI q-bio.BM 84%

ChemBERTa-2: Towards Chemical Foundation Models

Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, Bharath Ramsundar

专题命中预训练与数据：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments ELLIS Machine Learning for Molecule Discovery Workshop

2111.04130 2022-07-25 cs.CL cs.LG 84%

NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework

Xingcheng Yao, Yanan Zheng, Xiaocong Yang, Zhilin Yang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 14 pages, 5 figures

Journal ref Proceedings of the 39th International Conference on Machine Learning, PMLR 162:25438-25451, 2022

2207.10245 2022-07-22 cs.CL cs.AI 84%

The Birth of Bias: A case study on the evolution of gender bias in an English language model

Oskar van der Wal, Jaap Jumelet, Katrin Schulz, Willem Zuidema

专题命中预训练与数据：language model(title,abstract)；small language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at the 4th Workshop on Gender Bias in Natural Language Processing (NAACL, 2022)

2206.14355 2022-06-30 cs.CV cs.CL cs.LG 84%

EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering

Violetta Shevchenko, Ehsan Abbasnejad, Anthony Dick, Anton van den Hengel, Damien Teney

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2205.01863 2022-06-27 cs.CL cs.CR cs.LG 84%

Provably Confidential Language Modelling

Xuandong Zhao, Lei Li, Yu-Xiang Wang

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments NAACL 2022

2204.05939 2022-04-13 cs.CL cs.AI 84%

Mining Logical Event Schemas From Pre-Trained Language Models

Lane Lawley, Lenhart Schubert

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments To appear at ACL SRW 2022

2111.04909 2022-04-12 cs.CL cs.AI 84%

FPM: A Collection of Large-scale Foundation Pre-trained Language Models

Dezhou Shen

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments 12 pages, 4 tables

2204.03243 2022-04-08 cs.CL cs.LG 84%

Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators

Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2022. (Code and Models: https://github.com/microsoft/AMOS)

1. 预训练与数据 12393 篇

WeLM: A Well-Read Pre-trained Language Model for Chinese

A Close Look into the Calibration of Pre-trained Language Models

Mitigating Approximate Memorization in Language Models via Dissimilarity Learned Policy

What's in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models through User-Provided Names in CAD Files

CancerGPT: Few-shot Drug Pair Synergy Prediction using Large Pre-trained Language Models

Romanization-based Large-scale Adaptation of Multilingual Language Models

Impact of Pretraining Term Frequencies on Few-Shot Reasoning

Pretrained Language Models are Symbolic Mathematics Solvers too!

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Quantifying Memorization Across Neural Language Models

Write and Paint: Generative Vision-Language Models are Unified Modal Learners

Recitation-Augmented Language Models

Targeted Attack on GPT-Neo for the SATML Language Model Data Extraction Challenge

Cramming: Training a Language Model on a Single GPU in One Day

MILAN: Masked Image Pretraining on Language Assisted Representation

Deduplicating Training Data Mitigates Privacy Risks in Language Models

ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

Few-shot Learning with Multilingual Language Models

ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization

Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

ChemBERTa-2: Towards Chemical Foundation Models

NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework

The Birth of Bias: A case study on the evolution of gender bias in an English language model

EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering

Provably Confidential Language Modelling

Mining Logical Event Schemas From Pre-Trained Language Models

FPM: A Collection of Large-scale Foundation Pre-trained Language Models

Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators