Neonatal encephalopathy is one of the most common problems in newborns leading to severe complications and even death. Although current diagnosis is mainly performed by clinicians, artificial intelligence-based systems could potentially aid them in the diagnose. To this end, this work employs 73 videos of affected newborns from the ‘Hospital Universitario de Burgos’ to develop several convolutional neural network models capable of detecting one of the main signs of neonatal encephalopathy: lethargy. The detection of this sign has been divided into 3 items: Eye opening, Mouth contraction and Eye frowning. For each item, several models were trained and evaluated using the frames of these videos. The results obtained were positive especially for the Mouth contraction detection, with a balanced accuracy value of 0.822 and a recall of 0.85. Further improvements need to be carried out in order to make these models available as an aid-diagnostic tool in clinical practice.