DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
机构 * University of Freiburg(弗赖堡大学) ; ELLIS Institute Tübingen(图宾根ELLIS研究所) ; Microsoft Research(微软研究院) ; Prior Labs(先驱实验室) ; CSML, Istituto Italiano di Tecnologia(CSML,意大利技术研究所) ; AI Centre, University College London(伦敦大学学院人工智能中心)
Comments v5: Characterization of DeltaProduct's state-tracking ability. Analysis of hidden state's effective rank. Improved scaling analysis. v6: Added analysis for products of RWKV-7 matrices, v6: Accepted at NeurIPS 2025