Recent studies have demonstrated that spiking neural networks designed under the principle of efficient coding process information efficiently with properties close to that of real networks. However, there is no analytical study of whether efficient spiking networks impose a finite bound on neural firing rates and on the representational error, and whether the objective of improving the neural representation with every spike is realized in biophysically realistic neural implementations. Here, we demonstrate that such networks are guaranteed to keep firing rates finite, and to achieve a finite representational error. Further, we show that in a model with optimal parameters, the vast majority of spikes improve the representation and carry the information about the stimulus. Thus, efficient spiking networks are well-defined mathematically and realize their objectives with biologically realistic spiking dynamics.