语言大模型 / LLM

DataDecide: How to Predict Best Pretraining Data with Small Experiments

Ian Magnusson, Nguyen Tai, Ben Bogin, David Heineman, Jena D. Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A. Smith, Pang Wei Koh, Jesse Dodge

机构 * Allen Institute for AI ； Paul G. Allen School of Computer Science \& Engineering, University of Washington ； University of Pennsylvania 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ICML 2025

Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation

Guanting Dong, Xiaoxi Li, Yuyao Zhang, Mengjie Deng

机构 * Renmin University of China（中国人民大学） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at SIGIR 2025 LiveRAG Workshop (Oral Presentation)

OAT-Rephrase: Optimization-Aware Training Data Rephrasing for Zeroth-Order LLM Fine-Tuning

Jikai Long, Zijian Hu, Xiaodong Yu, Jianwen Xie, Zhaozhuo Xu

机构 * Stevens Institute of Technology（史蒂文斯理工学院） ； Lambda, Inc.（Lambda公司） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback

Yongjin Yang, Sinjae Kang, Juyong Lee, Dongjun Lee, Se-Young Yun, Kimin Lee

专题命中 预训练与数据 ：language agent(title)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Preprint, under review

Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model

Chong Li, Yingzhuo Deng, Jiajun Zhang, Chengqing Zong

机构 * State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院，北京） ； School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（人工智能学院，中国科学院大学，北京） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments ACL 2025, our codes and models are available at https://github.com/ZNLP/DMoE

Scaling Laws for Upcycling Mixture-of-Experts Language Models

Seng Pei Liew, Takuya Kato, Sho Takase

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments ICML 2025. 16 figures, 8 tables. Code available at https://github.com/sbintuitions/sparse-upcycling-scaling-laws

Relational Schemata in BERT Are Inducible, Not Emergent: A Study of Performance vs. Competence in Language Models

Cole Gawin

机构 * Cole Gawin（独立研究者） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 4 figures, 3 tables

A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets

Ryan Lagasse, Aidan Kierans, Avijit Ghosh, Shiri Dori-Hacohen

机构 * University of Connecticut Storrs（康涅狄格大学斯托尔斯分校） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

Wenqi Zhang, Hang Zhang, Xin Li, Jiashuo Sun, Yongliang Shen, Weiming Lu, Deli Zhao, Yueting Zhuang, Lidong Bing

机构 * College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院） ； DAMO Academy, Alibaba Group（阿里巴巴集团达摩院） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；LLM(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Under review

Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis

Albérick Euraste Djiré, Abdoul Kader Kaboré, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé

机构 * ucl.ac.uk（伦敦大学学院） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

MIND: Math Informed syNthetic Dialogues for Pretraining LLMs

Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro

专题命中 预训练与数据 ：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 31 pages, 5 figures, 14 tables

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

Nikhil Sardana, Jacob Portes, Sasha Doubov, Jonathan Frankle

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments 16 pages, 7 figures, In the 41st International Conference on Machine Learning, 2024

CroissantLLM: A Truly Bilingual French-English Language Model

Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, Pierre Colombo

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Improving Pretraining Data Using Perplexity Correlations

Tristan Thrush, Christopher Potts, Tatsunori Hashimoto

专题命中 预训练与数据 ：pretraining(title,abstract)；LLM(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2025

LLM-Empowered Class Imbalanced Graph Prompt Learning for Online Drug Trafficking Detection

Tianyi Ma, Yiyue Qian, Zehong Wang, Zheyuan Zhang, Chuxu Zhang, Yanfang Ye

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Large, Small or Both: A Novel Data Augmentation Framework Based on Language Models for Debiasing Opinion Summarization

Yanyue Zhang, Pengfei Li, Yilong Lai, Deyu Zhou, Yulan He

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；small language model(abstract)；分类 cs.CL、cs.AI

Journal ref COLING2025

LLM-based MOFs Synthesis Condition Extraction using Few-Shot Demonstrations

Lei Shi, Zhimeng Liu, Yi Yang, Weize Wu, Yuyang Zhang, Hongbo Zhang, Jing Lin, Siyu Wu, Zihan Chen, Ruiming Li, Nan Wang, Zipeng Liu, Huobin Tan, Hongyi Gao, Yue Zhang, Ge Wang

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Few-shot LLM Synthetic Data with Distribution Matching

Jiyuan Ren, Zhaocheng Du, Zhihao Wen, Qinglin Jia, Sunhao Dai, Chuhan Wu, Zhenhua Dong

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 10 pages, 5 figures, accepted at www 2025

Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers

Siddharth Singh, Prajwal Singhania, Aditya Ranjan, John Kirchenbauer, Jonas Geiping, Yuxin Wen, Neel Jain, Abhimanyu Hans, Manli Shu, Aditya Tomar, Tom Goldstein, Abhinav Bhatele

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

KBLaM: Knowledge Base augmented Language Model

Xi Wang, Taketomo Isazawa, Liana Mikaelyan, James Hensman

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models

Yimu Wang, Shuai Yuan, Bo Xue, Xiangru Jian, Wei Pang, Mushi Wang, Ning Yu

专题命中 预训练与数据 ：foundation model(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments NAACL 2025

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies

Ehsaneddin Asgari, Yassine El Kheir, Mohammad Ali Sadraei Javaheri

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles

Munachiso Nwadike, Zangir Iklassov, Toluwani Aremu, Tatsuya Hiraoka, Velibor Bojkovic, Benjamin Heinzerling, Hilal Alqaubeh, Martin Takáč, Kentaro Inui

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Chain-of-Translation Prompting (CoTR): A Novel Prompting Technique for Low Resource Languages

Tejas Deshpande, Nidhi Kowtal, Raviraj Joshi

专题命中 预训练与数据 ：prompting(title,abstract)；LLM(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted at PACLIC 38 (2024)

What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement

Xisen Jin, Xiang Ren

专题命中 预训练与数据 ：language model(title,abstract)；instruction tuning(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments ICML 2024 (Spotlight)

Scaling Law for Language Models Training Considering Batch Size

Xian Shuai, Yiding Wang, Yimeng Wu, Xin Jiang, Xiaozhe Ren

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.LG

LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemble

Yujeong Lee, Sangwoo Shin, Wei-Jin Park, Honguk Woo

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Findings of EMNLP-2024 Camera Ready Version

Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)

Tim Hartill

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Warmstarting for Scaling Language Models

Neeratyoy Mallik, Maciej Janowski, Johannes Hog, Herilalaina Rakotoarison, Aaron Klein, Josif Grabocka, Frank Hutter

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition

Ziyang Ma, Wen Wu, Zhisheng Zheng, Yiwei Guo, Qian Chen, Shiliang Zhang, Xie Chen

专题命中 预训练与数据 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments This work has been submitted to the IEEE for possible publication