The interest in capsule networks, recently proposed as an alternative to convolutional neural networks (CNNs), has seen a steady increase in recent years. This is mainly due to their ability to recognize variations in pose and deformations while requiring less training data compared to classic convolutional neural networks (CNNs). In addition, from an explainability perspective, this novel architecture also shows the potential of being more explainable and interpretable due to its hierarchical, internal representation of learned concepts and its ability to encode class characteristics as pose parameters in the class capsules. However, existing work has mainly focused on studying the first capsule network architecture, while newer architectures, such as Matrix Capsules with EM-Routing, have not received the same attention. Here we conduct a preliminary study of the inner-workings of Matrix Capsule architectures with EM-Routing and perform an analysis of the aspects that differentiate it from regular CNNs. At the same time, we focus our analysis on their interpretability and explainability properties.