Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference
面向低延迟边缘-云LLM推理的延迟自适应推测控制
机构 * Shanghai Key Laboratory of Integrated Administration Technologies for Information Security, School of Computer Science, Shanghai Jiao Tong University(上海信息安全集成管理技术重点实验室,上海交通大学计算机科学学院) ; School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(深圳大学理学院,香港中文大学(深圳))
AI总结 针对分布式边缘-云推理中推测解码的草稿长度在线控制问题,提出延迟单调阈值最优策略,并设计在线算法UCB-SpecStop,在真实测试台上验证了相变预测,延迟降低达22.4%。
Comments 16 pages, 9 figures, submitted to an IEEE journal for possible publication