语言大模型 / LLM

Variance-Adaptive Muon: Accelerating LLM Pretraining with NSR-Modulated and Variance-Scaled Momentum

方差自适应缪子：通过NSR调制和方差缩放的动量加速大语言模型预训练

Jingru Li, Yibo Fan, Huan Li

机构 * College of Artificial Intelligence, Nankai University（人工智能学院，南开大学） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；pretraining(title,abstract)；large language model(abstract)；language model(abstract)

AI总结 本文提出缪子-NSR和缪子-VS两种方法，通过方差自适应归一化和NSR调制或方差缩放，加速大语言模型预训练，提升收敛速度并降低验证损失。

Nationality and Region Prediction from Names: A Comparative Study of Neural Models and Large Language Models

从名字预测国籍：神经模型和大语言模型的比较研究

Keito Inoshita

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；prompting(abstract)

AI总结 本研究比较神经模型与大语言模型在国籍预测中的表现，发现LLMs在所有粒度级别均优于神经模型，且在低频国籍上表现下降，错误类型差异显著。

Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining

打破冻结子空间：用于大语言模型预训练低秩优化的重要性采样

Haochen Zhang, Junze Yin, Guanchu Wang, Zirui Liu, Lin F. Yang, Tianyi Zhang, Anshumali Shrivastava, Vladimir Braverman

机构 * Rice University（Rice大学） ； University of North Carolina at Charlotte（北卡罗来纳州立大学） ； University of Minnesota Twin Cities（明尼苏达大学双城分校） ； University of California, Los Angeles（加州大学洛杉矶分校） ； Johns Hopkins University（约翰霍普金斯大学） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；pretraining(title,abstract)；large language model(abstract)；language model(abstract)

AI总结 本文提出了一种具有可证明收敛性的低秩优化方法，用于大语言模型预训练，以解决主导子空间在预训练中冻结的问题，并在实验中表现出优越的性能。

Comments NeurIPS 2025

Nanbeige4-3B Technical Report: Exploring the Frontier of Small Language Models

Nanbeige4-3B 技术报告：探索小型语言模型的前沿

Chen Yang, Guangyue Peng, Jiaying Zhu, Ran Le, Ruixiang Feng, Tao Zhang, Wei Ruan, Xiaoqi Liu, Xiaoxue Cheng, Xiyun Xu, Yang Song, Yanzipeng Gao, Yiming Jia, Yun Xing, Yuntao Wen, Zekai Wang, Zhenwei An, Zhicong Sun, Zongchao Chen

机构 * Nanbeige LLM Lab（纳布吉LLM实验室） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；small language model(title,abstract)；post-training(abstract)；SFT(abstract)

AI总结 Nanbeige4-3B通过FG-WSD调度器、DPD蒸馏和强化学习技术，实现了小型语言模型在性能和扩展定律上的突破。

CoSP: Reconfigurable Multi-State Metamaterial Inverse Design via Contrastive Pretrained Large Language Model

CoSP: 通过对比预训练大语言模型实现可重构多状态超材料逆向设计

Shujie Yang, Xuzhe Zhao, Yuqi Zhang, Yansong Tang, Kaichen Dong

机构 * Center of Double Helix, Tsinghua Shenzhen International Graduate School, Tsinghua University（双螺旋中心，清华大学深圳国际研究生院，清华大学） ； Intelligent Passive Thermal Control Center, Research Institute of Tsinghua University in Shenzhen（智能被动热控制中心，清华大学深圳研究院） 纠错

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；pretraining(abstract)

AI总结 CoSP通过对比预训练大语言模型实现可重构多状态超材料的逆向设计，能高效生成具有目标光学性能的材料结构。

Comments 5 pages, 6 figures

Synthetic Data Generation Using Large Language Models: Advances in Text and Code

利用大语言模型生成合成数据：文本与代码领域的进展

Mihai Nadas, Laura Diosan, Andreea Tomescu

机构 * Faculty of Mathematics and Computer Science, Babeş-Bolyai University（巴贝什-博耶亚大学数学与计算机科学系） ； KlusAI Research Lab（KlusAI研究实验室） 纠错

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；instruction tuning(abstract)

AI总结 本文探讨了利用大语言模型生成合成数据在文本和代码领域的新进展，分析了其在低资源任务和代码应用中的潜力及挑战。

Comments 24 pages, 6 tables, 1 figure, 64 references

Journal ref IEEE Access 13, 134615-134633 (2025)

Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering

Vlad Negoita, Mihai Masala, Traian Rebedea

机构 * National University of Science and Technology（科学与技术国家大学） ； POLITEHNICA Bucharest（布加勒斯特理工大学） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；pretraining(title,abstract)；large language model(abstract)；language model(abstract)

Cognitive-Aligned Spatio-Temporal Large Language Models For Next Point-of-Interest Prediction

Penglong Zhai, Jie Li, Fanyi Di, Yue Liu, Yifang Yuan, Jie Huang, Peng Wu, Sicong Wang, Mingyang Yin, Tingting Hu, Yao Xu, Xin Li

机构 * AMAP, Alibaba Group（阿里集团AMAP） 纠错

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；pretraining(abstract)；SFT(abstract)

Comments 12 pages, 5 figures

On Relation-Specific Neurons in Large Language Models

Yihong Liu, Runsheng Chen, Lea Hirlimann, Ahmad Dawar Hakimi, Mingyang Wang, Amir Hossein Kargaran, Sascha Rothe, François Yvon, Hinrich Schütze

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；pretraining(abstract)

Comments EMNLP 2025

From Anchors to Answers: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Models

Yanbiao Ji, Chang Liu, Xin Chen, Dan Luo, Mei Li, Yue Ding, Wenqing Lin, Hongtao Lu

机构 * Shanghai Jiao Tong University（上海交通大学） ； The Chinese University of Hong Kong（香港中文大学） ； Lehigh University（莱斯大学） ； Tencent（腾讯） 纠错

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；pretraining(abstract)

Comments Accepted by CIKM 2025

NovoMolGen: Rethinking Molecular Language Model Pretraining

Kamran Chitsaz, Roshan Balaji, Quentin Fournier, Nirav Pravinbhai Bhatt, Sarath Chandar

机构 * Mila – Quebec AI Institute（魁北克AI研究所） ； Wadhwani School of Data Science and AI, IIT Madras（数据科学与人工智能学院，印度理工学院马德拉斯分校） ； The Centre for Integrative Biology and Systems medicinE (IBSE)（整合生物学与系统医学中心） ； IIT Madras Zanzibar（印度理工学院马德拉斯分校（赞赞比尔）） ； Polytechnique Montréal Canada（蒙特利尔理工学院，加拿大） ； CIFAR AI Chair（CIFAR人工智能主席） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；large language model(abstract)；foundation model(abstract)

Can Large Language Models (LLMs) Describe Pictures Like Children? A Comparative Corpus Study

Hanna Woloszyn, Benjamin Gagl

机构 * Self-Learning Systems Lab, Faculty of Human Sciences, Department of Special Education and Rehabilitation（自主学习系统实验室，人文科学学院，特殊教育与康复系） 纠错

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；prompting(abstract)

Craw4LLM: Efficient Web Crawling for LLM Pretraining

Shi Yu, Zhiyuan Liu, Chenyan Xiong

机构 * Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系） ； Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；pretraining(title,abstract)；large language model(abstract)；language model(abstract)

Checkpoint Merging via Bayesian Optimization in LLM Pretraining

Deyuan Liu, Zecheng Wang, Bingning Wang, Weipeng Chen, Chunshan Li, Zhiying Tu, Dianhui Chu, Bo Li, Dianbo Sui

专题命中 预训练与数据 ：LLM(title,abstract)；pretraining(title,abstract)；large language model(abstract)；language model(abstract)

Retrieval Augmented Generation based Large Language Models for Causality Mining

Thushara Manjari Naduvilakandy, Hyeju Jang, Mohammad Al Hasan

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；prompting(abstract)

Comments 13 pages, 6 figures, published in knowledgeNLP-NAACL2025

QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining

Fengze Liu, Weidong Zhou, Binbin Liu, Zhimiao Yu, Yifan Zhang, Haobin Lin, Yifeng Yu, Bingni Zhang, Xiaohuan Zhou, Taifeng Wang, Yong Cao

机构 * ByteDance（字节跳动） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；pretraining(title,abstract)；large language model(abstract)；language model(abstract)

Large Language Model for Verilog Generation with Code-Structure-Guided Reinforcement Learning

Ning Wang, Bingkun Yao, Jie Zhou, Xi Wang, Zhe Jiang, Nan Guan

机构 * City University of Hong Kong（香港城市大学） ； Southeast University（东南大学） 纠错

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；instruction tuning(abstract)

Ontology Matching with Large Language Models and Prioritized Depth-First Search

Maria Taboada, Diego Martinez, Mohammed Arideh, Rosa Mosquera

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；prompting(abstract)

Improving Consistency in Large Language Models through Chain of Guidance

Harsh Raj, Vipul Gupta, Domenic Rosati, Subhabrata Majumdar

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；prompting(abstract)

Comments Accepted at Transactions of Machine Learning Research (TMLR) 2025

MetaOpenFOAM 2.0: Large Language Model Driven Chain of Thought for Automating CFD Simulation and Post-Processing

Yuxuan Chen, Xu Zhu, Hua Zhou, Zhuyin Ren

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；post-training(abstract)

Comments 16 pages,11 figures

CoddLLM: Empowering Large Language Models for Data Analytics

Jiani Zhang, Hengrui Zhang, Rishav Chakravarti, Yiqun Hu, Patrick Ng, Asterios Katsifodimos, Huzefa Rangwala, George Karypis, Alon Halevy

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；foundation model(abstract)；post-training(abstract)

Foundational Large Language Models for Materials Research

Vaibhav Mishra, Somaditya Singh, Dhruv Ahlawat, Mohd Zaki, Vaibhav Bihani, Hargun Singh Grover, Biswajit Mishra, Santiago Miret, Mausam, N. M. Anoop Krishnan

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；pretraining(abstract)

Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language

Raphaël Merx, Aso Mahmudi, Katrina Langford, Leo Alberto de Araujo, Ekaterina Vylomova

专题命中 预训练与数据 ：LLM(title,abstract)；prompting(title,abstract)；large language model(abstract)；language model(abstract)

Hidden Entity Detection from GitHub Leveraging Large Language Models

Lu Gan, Martin Blum, Danilo Dessi, Brigitte Mathiak, Ralf Schenkel, Stefan Dietze

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；pretraining(abstract)

Comments accepted by KDD2024 workshop DL4KG

Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation

Heeseung Kim, Soonshin Seo, Kyeongseok Jeong, Ohsung Kwon, Soyoon Kim, Jungwhan Kim, Jaehong Lee, Eunwoo Song, Myungwoo Oh, Jung-Woo Ha, Sungroh Yoon, Kang Min Yoo

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；pretraining(abstract)

Comments NeurIPS 2024, Project Page: https://unifiedsdm.github.io/

Information Anxiety in Large Language Models

Prasoon Bajpai, Sarah Masud, Tanmoy Chakraborty

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；pretraining(abstract)

Reminding Multimodal Large Language Models of Object-aware Knowledge with Retrieved Tags

Daiqing Qi, Handong Zhao, Zijun Wei, Sheng Li

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；foundation model(abstract)；instruction tuning(abstract)

Comments Main Conference at EMNLP 2024

Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extraction

Sizhe Zhou, Yu Meng, Bowen Jin, Jiawei Han

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；small language model(abstract)；SLM(abstract)

Comments 25 pages, 20 Tables, 9 Figures; Accepted to EMNLP 2024

Statically Contextualizing Large Language Models with Typed Holes

Andrew Blinn, Xiang Li, June Hyung Kim, Cyrus Omar

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；prompting(abstract)

Comments To appear at OOPSLA2024

Journal ref Proceedings of the ACM on Programming Languages, Volume 8, 2024; Issue OOPSLA2 Article No.: 288, Pages 468 - 498

Iterative Data Generation with Large Language Models for Aspect-based Sentiment Analysis

Qihuang Zhong, Haiyun Li, Luyao Zhuang, Juhua Liu, Bo Du

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；prompting(abstract)