Accurate citation context extraction is essential for understanding the impact and relevance of cited works, enhancing scientometric tasks such as citation analysis, literature review automation, and knowledge graph construction. In this paper we leverage a large language model (RoBERTa) and fine-tune it for span extraction in a question-answering framework, to effectively extract citation contexts from scientific texts. Traditional citation context extraction methods use fixed-window lengths and often fail to capture the nuanced nature and variable lengths of citation contexts. We leverage the capabilities of context-aware transformer models such as RoBERTa to overcome these limitations and improve the efficacy of citation context extraction. We propose a novel approach that fine-tunes a RoBERTa model by treating the downstream task as a span extraction problem similar to a question-answering task. This method allows the model to dynamically identify and extract relevant citation contexts based on the specific in-text citation, thereby capturing a more precise and contextually relevant text snippet. We evaluated our model against four baseline methods that use a fixed-window approach on two benchmark datasets. Our model outperforms all baseline methods, achieving F1 scores of 95% and 92% on each dataset. Its adaptability to the variable nature of citation contexts led to more accurate extractions, demonstrating the effectiveness of the span extraction approach within a question-answering framework. These results highlight the potential of transformer-based models to significantly enhance citation context extraction from scientific literature.