Let \((K,H_V)\) be a Heisenberg Gelfand pair and \(G:=K\ltimes H_V\) be its associated semidirect product. Here, K is a compact Lie group acting smoothly on the Heisenberg group \(H_V:=V\times \mathbb {R},\) where V is a finite-dimensional complex vector space. Let \(\widehat{G}\) be the unitary dual of G equipped with the Fell topology. We say that \(\pi \in \widehat{G}\) is a spherical representation of G if the restriction \(\pi |_K\) of \(\pi \) to the subgroup K has a one-dimensional space of K-fixed vectors. Boidol, Ludwig and Müller have introduced the notion of the so-called small representations of G, that is all \(\pi \in \widehat{G}\) that cannot be Hausdorff separated from the trivial one-dimensional representation \(1_G\) of G. Using a parametrization of the set of irreducible spherical representations due to work of Benson-Ratcliff, we show that any non-trivial spherical representation of G cannot be a small representation. Furthermore, we prove that the converse of this statement is false in the setting of the Heisenberg motion group \(G_d:=U(d)\ltimes H_{\mathbb {C}^d}, d\in \mathbb {N}^\times .\)