语言大模型 / LLM - arXivDaily 专题

2407.09985 2024-10-25 cs.AI cs.LG 86%

A Training Data Recipe to Accelerate A* Search with Language Models

Devaansh Gupta, Boyang Li

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted to Findings of EMNLP, 2024

2407.05841 2024-10-23 cs.CL cs.LG 86%

An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models

Nandini Mundra, Aditya Nanda Kishore, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments CONLL 2024 (EMNLP 2024)

2410.08196 2024-10-11 cs.CL cs.AI cs.CV 86%

MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li

专题命中预训练与数据：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments https://github.com/mathllm/MathCoder2

2410.05355 2024-10-10 cs.CL cs.AI 86%

Falcon Mamba: The First Competitive Attention-free 7B Language Model

Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, Hakim Hacid

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2407.17467 2024-10-08 cs.CL cs.LG 86%

CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models

Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, Fei Tan

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments EMNLP 2024 main conference

2406.08116 2024-10-07 cs.CL cs.AI 86%

Supportiveness-based Knowledge Rewriting for Retrieval-augmented Language Modeling

Zile Qiao, Wei Ye, Yong Jiang, Tong Mo, Pengjun Xie, Weiping Li, Fei Huang, Shikun Zhang

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

2311.17041 2024-10-04 cs.CV cs.AI cs.CL 86%

Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties

Keunwoo Peter Yu, Zheyuan Zhang, Fengyuan Hu, Shane Storks, Joyce Chai

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；prompting(abstract)；分类 cs.CL、cs.AI

Comments 16 pages, LaTeX; Accepted to EMNLP 2024 Main

2408.10468 2024-09-06 cs.LG cs.CL cs.CR 86%

Tracing Privacy Leakage of Language Models to Training Data via Adjusted Influence Functions

Jinxin Liu, Zao Yang

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

2408.15650 2024-08-29 cs.CL cs.AI 86%

Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings

Lingyu Gao

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments PhD thesis

2407.12869 2024-07-29 cs.CL cs.AI 86%

Bilingual Adaptation of Monolingual Foundation Models

Gurpreet Gosal, Yishi Xu, Gokul Ramakrishnan, Rituraj Joshi, Avraham Sheinin, Zhiming, Chen, Biswajit Mishra, Natalia Vassilieva, Joel Hestness, Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Onkar Pandit, Satheesh Katipomu, Samta Kamboj, Samujjwal Ghosh, Rahul Pal, Parvez Mullah, Soundar Doraiswamy, Mohamed El Karim Chami, Preslav Nakov

专题命中预训练与数据：foundation model(title)；LLM(abstract)；large language model(abstract)；language model(abstract)

2407.09855 2024-07-16 cs.CL cs.AI 86%

Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi

Shantipriya Parida, Shakshi Panwar, Kusum Lata, Sanskruti Mishra, Sambit Sekhar

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted as a book chapter in the book Title "APPLIED SPEECH AND TEXT PROCESSING FOR LOW RESOURCE LANGUAGES"

2312.12112 2024-07-02 cs.LG cs.AI 86%

Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes

Nabeel Seedat, Nicolas Huynh, Boris van Breugel, Mihaela van der Schaar

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Presented at the 41st International Conference on Machine Learning (ICML) 2024. *Seedat & Huynh contributed equally

2401.03003 2024-06-25 cs.SE cs.CL cs.LG 86%

AST-T5: Structure-Aware Pretraining for Code Generation and Understanding

Linyuan Gong, Mostafa Elhoushi, Alvin Cheung

专题命中预训练与数据：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 15 pages; ICML 2024: https://icml.cc/virtual/2024/poster/33601

2406.09412 2024-06-14 cs.CV cs.AI cs.LG cs.MM 86%

Explore the Limits of Omni-modal Pretraining at Scale

Yiyuan Zhang, Handong Li, Jing Liu, Xiangyu Yue

专题命中预训练与数据：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Project Website: https://invictus717.github.io/MiCo/

2406.06563 2024-06-12 cs.CL cs.AI 86%

Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models

Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lü, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, Xiaokun Wang, Yutuan Ma, Rui Hu, Shuicheng Yan, Han Fang, Yahui Zhou

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2406.05955 2024-06-12 cs.LG cs.CL 86%

Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters

Yixin Song, Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, Haibo Chen

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

2406.01333 2024-06-04 cs.CL cs.AI 86%

Probing Language Models for Pre-training Data Detection

Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Haonan Lu, Bing Liu, Wenliang Chen

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL-2024 main conference

2402.10379 2024-05-29 cs.CL cs.LG 86%

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows

Ajay Patel, Colin Raffel, Chris Callison-Burch

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Published in ACL 2024

2405.15640 2024-05-27 cs.CL cs.AI 86%

GECKO: Generative Language Model for English, Code and Korean

Sungwoo Oh, Donggyu Kim

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2405.11446 2024-05-21 cs.CL cs.LG 86%

MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning

Sanchit Sinha, Yuguang Yue, Victor Soto, Mayank Kulkarni, Jianhua Lu, Aidong Zhang

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments KDD 2024, 11 pages(9 main, 2 ref, 1 App) Openreview https://openreview.net/forum?id=JwecLNhWDy&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DKDD.org%2F2024%2FResearch_Track%2FAuthors%23your-submissions)

2404.17806 2024-04-30 cs.SD cs.CL cs.LG eess.AS 86%

T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining

Yi Yuan, Zhuo Chen, Xubo Liu, Haohe Liu, Xuenan Xu, Dongya Jia, Yuanzhe Chen, Mark D. Plumbley, Wenwu Wang

专题命中预训练与数据：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Preprint submitted to IEEE MLSP 2024

2404.18228 2024-04-30 cs.CL cs.LG 86%

TextGram: Towards a better domain-adaptive pretraining

Sharayu Hiwarkhedkar, Saloni Mittal, Vidula Magdum, Omkar Dhekane, Raviraj Joshi, Geetanjali Kale, Arnav Ladkat

专题命中预训练与数据：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted at SPELLL 2023

2404.17790 2024-04-30 cs.CL cs.AI 86%

Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, Naoaki Okazaki

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

2404.08865 2024-04-16 cs.CL cs.LG 86%

LLM In-Context Recall is Prompt Dependent

Daniel Machlab, Rick Battle

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

2403.17661 2024-04-16 cs.CL cs.AI 86%

Language Models for Text Classification: Is In-Context Learning Enough?

Aleksandra Edwards, Jose Camacho-Collados

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；prompting(abstract)；分类 cs.CL、cs.AI

Comments Accepted at LREC-COLING 2024

2311.14648 2024-03-21 cs.CL cs.AI 86%

Calibrated Language Models Must Hallucinate

Adam Tauman Kalai, Santosh S. Vempala

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；post-training(abstract)；分类 cs.CL、cs.AI

Comments In Proceedings of the 56th Annual ACM Symposium on Theory of Computing (STOC) 2024

2403.04811 2024-03-11 cs.SE cs.CL cs.LG 86%

Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models

Martin Riddell, Ansong Ni, Arman Cohan

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

2402.12151 2024-03-05 cs.CL cs.AI 86%

Transformer-based Causal Language Models Perform Clustering

Xinbo Wu, Lav R. Varshney

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Added new experimental results and fixed some errors

2402.12102 2024-02-20 cs.CL cs.AI 86%

Is It a Free Lunch for Removing Outliers during Pretraining?

Baohao Liao, Christof Monz

专题命中预训练与数据：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 5 pages, 3 figures, 1 table

2402.09841 2024-02-16 cs.CL cs.CV cs.LG 86%

LAPDoc: Layout-Aware Prompting for Documents

Marcel Lamott, Yves-Noel Weweler, Adrian Ulges, Faisal Shafait, Dirk Krechel, Darko Obradovic

专题命中预训练与数据：prompting(title)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Under review at ICDAR2024

1. 预训练与数据 12365 篇

A Training Data Recipe to Accelerate A* Search with Language Models

An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models

MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

Falcon Mamba: The First Competitive Attention-free 7B Language Model

CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models

Supportiveness-based Knowledge Rewriting for Retrieval-augmented Language Modeling

Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties

Tracing Privacy Leakage of Language Models to Training Data via Adjusted Influence Functions

Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings

Bilingual Adaptation of Monolingual Foundation Models

Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi

Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes

AST-T5: Structure-Aware Pretraining for Code Generation and Understanding

Explore the Limits of Omni-modal Pretraining at Scale

Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models

Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters

Probing Language Models for Pre-training Data Detection

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows

GECKO: Generative Language Model for English, Code and Korean

MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning

T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining

TextGram: Towards a better domain-adaptive pretraining

Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

LLM In-Context Recall is Prompt Dependent

Language Models for Text Classification: Is In-Context Learning Enough?

Calibrated Language Models Must Hallucinate

Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models

Transformer-based Causal Language Models Perform Clustering

Is It a Free Lunch for Removing Outliers during Pretraining?

LAPDoc: Layout-Aware Prompting for Documents