Training Matters: Unlocking Potentials of Deeper Graph Convolutional Neural Networks

The performance limit of deep Graph Convolutional Networks (GCNs) are pervasively thought to be caused by the inherent limitations of the GCN layers, such as their insufficient expressive power. However, if this were true, modifying only the training procedure for a given architecture would not likely to enhance performance. Contrary to this belief, our paper demonstrates several ways to achieve such improvements. We begin by highlighting the training challenges of GCNs from the perspective of graph signal energy loss. More specifically, we find that the loss of energy in the backward pass during training hinders the learning of the layers closer to the input. To address this, we propose several strategies to mitigate the training problem by slightly modifying the GCN operator, from the energy perspective. After empirical validation, we confirm that these changes of operator lead to significant decrease in the training difficulties and notable performance boost, without changing the composition of parameters. With these, we conclude that the root cause of the problem is more likely the training difficulty than the others.