Epilepsy is a serious neurological disorder resulting in brain seizures. Predicting its onset at an earlier stage can provide the necessary golden period for treatment. In the earlier work, a cross-reference learning-based epileptic seizure detection was proposed using multi-modal features. Cross-correlation information is learned from multi-modal EEG signals, and a Long-Short-Term-Memory (LSTM) predictor is trained to classify epileptic seizures. Though this method improved accuracy, optimizing the feature selection and predictor process can further improve the model. Toward this goal, this work integrates optimization strategies to cross-reference learning. Optimization is done in two stages: feature selection and prediction. Unlike the multi-head dot product attention, clustering-based attention is applied in the feature selection stage. In the second stage, hyperparameters of LSTM are optimized with the objective of reduction of prediction error. With this two-stage optimization, the accuracy of the reference learning model increased by 1%, and false positives reduced by 2%.