This paper presents two strategies to prevent the learned pitch representation from worsening so as to improve pitch and pitch class distributions in symbolic music generation. The first strategy is to switch the input pitch representation from the flat MIDI number representation to a hierarchical representation consisting of pitch class (chroma) and octave, which forces musically similar pitches to share part of the embedding vectors. The second strategy freezes the pitch embeddings during training according to the proposed evaluation metric of the pitch embedding space, maintaining the robustness of the embedding obtained in the first strategy. The experiments show that, when both strategies were applied to training an auto-regressive neural network for melody generation, the generated samples exhibited significant improvement in pitch class entropy (from 19% to 34% overlapping with the test dataset), and a modest but still significant improvement on pitch entropy (from 24% to 28%).