ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
ExpertFlow:通过预测专家缓存和令牌调度实现高效的混合专家推理
Xin He, Shunkang Zhang, Kaijie Tang, Shaohuai Shi, Yuxin Wang, Zihao Zeng, Zhenheng Tang, Xiaowen Chu, Haiyan Yin, Ivor W. Tsang, Yew Soon Ong
机构
*
CFAR, Agency for Science, Technology and Research (A*STAR), Singapore(新加坡科技研究局CFAR)
;
The Hong Kong University of Science and Technology, Hong Kong(香港科技大学)
;
Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳))
;
Hong Kong Baptist University, Hong Kong(香港浸会大学)
;
Nanyang Technological University, Singapore(南洋理工大学)
;
The Hong Kong University of Science and Technology (Guangzhou), China(香港科技大学(广州))