As Large Language Models (LLMs) have become robust and universal in various tasks and increasingly popular and widely used in society, their political bias is becoming more and more critical. Hence, to probe the political bias of LLMs, several studies have proposed frameworks that ask political questions and estimate the political stance of LLMs by using their answers. However, these approaches suffer from the effect of the question’s wording and the randomness of answer generation algorithms, and do not take into account information selection bias in the context of summarization tasks. In this work, we summarize news articles describing the same news event with different political stances by using the LLMs and evaluate the political stance of the output summaries by comparing the output and input texts. We probed some LLMs and found a potential political bias in word choice. Additionally, our result indicates that the bias can be mostly dissipated by generation algorithms.