This project compares two neural models: S4 and LRU, evaluating their performance across accuracy. We compare in task Key-Word Spotting. Audio is a long time-series format and good fit for this models.
Neural models play a crucial role in various applications, and understanding their performance characteristics is essential for making informed decisions. This project focuses on comparing two neural models, S4 and LRU, to help users choose the most suitable model for their specific needs.
S4 is a neural model designed for long time series. Structured State Space sequence model (S4) based on a new parameterization for the SSM, and show that it can be computed much more efficiently than prior approaches while preserving their theoretical strengths.
LRU has careful design of deep RNNs using standard signal propagation arguments can recover the impressive performance of deep SSMs on long-range reasoning tasks, while also matching their training speed. To achieve this, we analyze and ablate a series of changes to standard RNNs including linearizing and diagonalizing the recurrence, using better parameterizations and initializations, and ensuring proper normalization of the forward pass.