Neural ordinary differential equations (Neural ODEs) are deep models in which ODEs described by neural networks determine the transformation from inputs to outputs. The continuous nature of transformations obtained by Neural ODEs makes them suitable for a variety of applications. The dynamics of standard Neural ODEs employed in practice are known to be generally non-oscillatory and stable, converging to an equilibrium point. Conversely, it is known in neuroscience that periodic and oscillatory dynamics of neural networks in brains play an important role. In this paper, we propose the neural almost invariant set (Neural AIS), a novel class of Neural ODEs whose dynamics are periodic and oscillatory and that utilize those properties of dynamics as features via almost invariant sets using the spectra of the corresponding transfer operator. And, we provide a practical way of implementing the above idea using random Fourier features. We evaluate Neural AISs empirically against existing Neural ODEs in the image classification task. We also investigate the behavior of Neural AISs through spectral analysis.