Artificial Intelligence (AI), particularly generative AI (GenAI) tools like ChatGPT, has gained significant visibility in education due to its ability to simulate natural language interactions. While AI offers opportunities to enhance learning - such as personalized tutoring, research assistance, and code debugging - it also presents risks, including over-reliance, the spread of misinformation, and reduced development of critical thinking. To ensure ethical and effective use, students must develop AI literacy, which involves understanding how AI systems work, recognizing their limitations, and critically evaluating their outcomes. While existing researches rely on self-assessment questionnaires to identify AI competencies, this approach can be subject to bias and lacks insights into actual user behavior. To address this limitation, this research explores an alternative method by analyzing student interaction traces with ChatGPT, combined with self-assessment data, to identify AI competencies in an educational context. Interaction traces are defined as the entire history of interactions between students and the AI tool during the execution of a task. This paper presents the results of a pilot study involving undergraduate students in computer science-related courses. The analysis highlights discrepancies between self-perceived and demonstrated competencies, emphasizing the need for more practical assessment strategies. The results suggest that an understanding of AI competencies can be made more comprehensive if there is another form of assessment besides a self-assessment questionnaire, thus allowing students to improve themselves and become critical citizens in the use of AI.