Large Language Models (LLMs) have gained significant popularity for their application in various everyday tasks such as text generation, summarization, and information retrieval. As the widespread adoption of LLMs continues to surge, it becomes increasingly crucial to ensure that these models yield responses that are politically impartial, with the aim of preventing information bubbles, upholding fairness in representation, and mitigating confirmation bias. In this paper, we propose a cost-effective and automated pipeline to generate prompts and train a classifier to explore bias within LLM. This process was used to produce BERTPOL – a political ideology classifier. We also propose a classification of bias in LLMs into first-level bias and second-level bias to determine whether a model is generally biased (first-level) or whether it exhibits bias only when portraying a persona or attribute (second-level). We then discuss a few significant findings: 1) responses from LLMs on politically controversial topics are mainly left-leaning, 2) Across different industries and occupational roles, Healthcare and Education are the most liberal-leaning, 3) White Americans are portrayed as more politically neutral as compared to Asian, African and Hispanic Americans, which are significantly more liberal as seen by LLMs.