The growing prevalence of LLM-based conversational agents in everyday applications has led to an increasing risk of users disclosing sensitive personal information. Understanding how effectively different tools can identify such disclosures, and therefore protect users, is critical to mitigate privacy risks in human-agent interactions. This paper aims to evaluate the effectiveness of different methods to detect personal information in human-agent conversations. In particular, we compare the potential of several out-of-the-box LLMs as detection agents to more traditional approaches such as Microsoft Presidio. To do so, we use a labeled dataset containing various human interactions with conversational agents. We show that both approaches have strengths and weaknesses, and that none of them on their own seem effective enough to detect personal information in human-agent interactions in uncontrolled, real-world environments.