The shift of computing capabilities towards edge sensing elements in image classification tasks is leading to the substitution of cameras in some industrial and consumer tasks with ultra-low-resolution (ULR) Time-of-Flight (ToF) sensors, thanks to their compactness and ultra-low power consumption. To effectively integrate classification capabilities into edge devices, a hardware-aware design of the classification algorithm and a careful custom Hardware (HW) post-processing core are needed. This work proposes a new compact and ultra-low power Neural Network (NN) that enables the implementation of a HW post-processing core for real-time classification tasks inside the sensor packaging. The resulting HW architecture has been prototyped on an AMD Xilinx Artix-7 FPGA, reaching an Energy per Inference consumption of 65.6 nJ and a power consumption of 1.095 \(\upmu W\) at the maximum Output Data Rate of the sensor. Moreover, the implementation in Skywater 130 nm technology reveals an area occupation below 1 \({\text {mm}}^2\) , with a total power compared to the sensor consumption of about 11%. These results are significantly lower than the typical energy and power consumption of the sensor, enabling real-time post-processing of depth images and encouraging research in integrating classification capabilities next to sensing elements.