We propose a vision-language model-based framework for image recognition that operates by first generating natural language descriptions of each class from only a few example images, and then using these descriptions to guide the recognition process. Deploying machine learning systems in real-world settings is often hindered by challenges such as (i) variability in class-defining features across individuals or over time, (ii) the need for annotations that require deep domain expertise, and (iii) the demand for intuitive and explainable predictions by end users. Our description-driven approach addresses all of these challenges and is evaluated on the challenging task of estimating the emotional state of children with profound intellectual and multiple disabilities (PIMD). To assess the effectiveness of the proposed framework, we conducted comfort and discomfort state estimation using video data of a single PIMD child, who expresses emotional states primarily through facial expressions. Experimental results show that our method, guided by facial-expression descriptions distilled from just 24 images, achieves F1 scores comparable to conventional approaches, without requiring input from parents or AI engineers.