Large Language Models (LLMs) are increasingly used in collaboration analytics to identify roles and analyze team dynamics. However, concerns about bias and stereotypes in LLM-based role identification remain underexplored. This pilot study examines whether GPT-4o exhibits gender bias in leadership identification within collaborative problem-solving (CPS) contexts. Using two CPS datasets, we analyze how GPT-4o assigns leadership roles under agentic (dominant, assertive) and communal (collaborative, supportive) leadership definitions and whether these assignments change when gender information is provided. Results show that providing gender information leads to systematic shifts in the LLM’s leadership attribution, with male students more frequently assigned agentic leadership and female students more frequently assigned communal leadership. Further logistic regression analysis reveals that linguistic patterns in student discourse do not explain these changes, suggesting that gender information may drive the leadership identification. These findings have critical implications for the AIED community, highlighting the need for rigorous bias evaluation before applying LLMs in collaboration analytics.