With the increasing interest in applying neural networks (NNs) to safety-critical problems like autonomous driving or unmanned aircrafts, ensuring the reliability of these NNs becomes essential. Therefore, several verification techniques have been proposed in recent years. Additionally, various abstraction techniques have been developed to enable verification of larger NNs. As the area develops, different surveys on verification of NNs are being published. However, we are missing a systematic summarization of knowledge through the lens of formal methods. In this literature review, we provide a systematic overview of techniques for verification and abstraction of NNs published in well-known formal verification conferences during the last ten years.