We present new machine learning methods for computing brain state sleep trajectories from EEG data and objective insomnia detection using these trajectories, without the intermediate step of sleep stage scoring. To efficiently compute the brain state sleep trajectory for new patients and deal with the cold start problem, we explore three ensemble-based approaches (M0, M1 and M2), combining individual, neural network-based, surrogate models of BrainTrak for the other subjects. All three methods performed well, demonstrating that it is possible to accurately estimate the brain state sleep trajectory of a new patient based on the trajectories of the other subjects. We then propose three methods for detecting insomnia, which use the computed sleep trajectories. Two of these methods (D1 and D2) are based on voting ensembles, while the third (D3) employs a convolutional neural network on a novel representation: 2D-image of 3D-volume occupancy features, to construct a brain state trajectory signature for the patient. All three insomnia detection methods performed well, achieving accuracy of 88.89-96.83% and AUC of 90.33-96.58%, with D2 and D3 performing the best. By leveraging machine learning methods for computing brain state sleep trajectories and insomnia detection, our approach highlights a promising pathway toward scalable and objective tools, to assist clinicians in their diagnosis.