HM-W2V: Word Embedding from Hippocampus Spiking Neural Network

The spiking neural network (SNN) is biologically interpretable, but faces slow progress in NLP due to limitations like word embeddings. To cultivate a suitable word vector space for SNN, we draw inspiration from the memory processes in the human brain's hippocampus, using analog-to-digital conversion to abstract the memory mechanism, and propose an interactive word embedding space for the temporal lobe and hippocampus, with a prefrontal lobe module for integrating information processing. Building upon this foundation, we propose a novel hippocampus spiking neural network (HM-SNN) and train the word embeddings (HM-w2v) on the OpenWebText dataset. The word vectors in HM-w2v simulate the membrane potential changes in the brain when perceiving word information. To assess the quality of HM-w2v, we compare the performance in word similarity and text classification tasks with word2vec and FlyVec. Experimental findings indicate that, when the model's LIF (Leaky Integrate-and-Fire) Neuron outputs pulses, our word embeddings outperform word2vec and exhibit robust competitiveness against FlyVec. Additionally, when adopting the membrane potential as the output of LIF Neuron, HM-w2v demonstrates pronounced advantages. At last, we engage in a discussion regarding the threshold for pulse firing in LIF Neuron, revealing that optimal word embedding quality is achieved when the number of activated neurons in the pulse vector approximates half of the vector length.