Today, internet users are becoming increasingly vulnerable to phishing attacks. Phishing is a five-star crime involving stealing sensitive information from internet users, leading to heavy economic losses. Many recent reports suggest that the number of phishing attacks doubles yearly. Therefore, effective and efficient methods are needed to detect these phishing attacks. This paper proposes a novel approach to detect phishing websites with high-performance solutions. We developed a framework that extracts embeddings of Uniform Resource Locators (URL) generated using several embedding creation techniques. Later, the existing features and URL embeddings were concatenated and used to train several deep learning models (FFNN, RNN, and LSTM). Our proposed RoBERTa embeddings based LSTM model for phishing website detection achieved an accuracy of 99.98% and an F1-score of 99.991%. We also focused on model explainability and performed the faithfulness check to validate model reliability. We used the integrated gradient algorithm to explain the performance of the designed models to various stakeholders. The faithfulness of the model was evaluated by removing the top 5 attributes with the highest attribution scores iteratively and observing the performance change in accuracy.