Transformer models have garnered significant attention for various tasks, such as sentiment analysis. From basic transformer architectures to generative models, numerous approaches have been applied and tested for sentiment analysis. However, the limitations and challenges associated with these models have not yet been adequately assessed. In this study, we aim to explore both the potential and the limitations of two types of transformer-based models (RoBERTa-XLM and GPT-2) for aspect-based sentiment analysis. Our evaluation is conducted using data from the Swiss Sleep Database. We conclude, that techniques like fine-tuning, data balancing, expert-driven normalization, and negation-aware processing techniques are essential to improve their performance in medical contexts.