Resonator-Gated RNNs

Detecting repetitive and periodic temporal patterns is essential for accurate predictions and informed decision-making in various domains of sequence learning. In RNN-based approaches to sequence learning, gated RNNs, such as long short-term memory networks (LSTMs) and gated recurrent units (GRUs), are the de facto standard for these predictions. While adept at capturing longer-term dependencies, gated RNNs still sometimes struggle with periodic data components because their gating mechanism is designed to prioritize retaining transient relevant information. As a result, these networks are often challenged by periodicity in the data. We present a novel memory unit that incorporates a simple resonator circuit. The circuit facilitates the recognition of periodic data patterns, focusing on data-specific time scales and respective frequencies. Moreover, it enables the forward propagation of information through resonating dynamics, while stably channeling the gradient backwards. We show that our resonator-gated RNN (RG-RNN) accelerates the training convergence on multiple sequence classifications tasks. Moreover, it significantly outperforms vanilla LSTMs on four out of five benchmark tasks in terms of accuracy. We conclude that resonator-based gating offers a new inductive bias to gated-RNNs, focusing learning on the detection and processing of periodic data patterns.