Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
通过预处理稀疏性实现近oracle的KV选择用于长上下文推理
机构 * University College London(伦敦大学学院) ; Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS)(深圳先进技术研究院,中国科学院) ; Chinese University of Hong Kong(香港中文大学) ; College of Computing & Data Science, Nanyang Technological University(computing 与数据科学学院,南洋理工大学)
AI总结 PrHS通过预处理稀疏性在长上下文推理中实现近oracle的KV选择,有效降低计算和带宽成本,提升推理效率。
Comments An effective method for accelerating LLM's inference via selective KV processing