The immune system plays a critical role protecting the body from pathogens and foreign substances. Food allergies produce disorders in the immune system. Young children (between one to three years old) start exploring the world by consuming any object that comes to their attention. Exploration skills being underrecognized pose a primary threat to a child’s life. Children with allergies require increased oversight in the growth stage. This paper proposes a smart surveillance device called “FoodPAL” that can monitor toddlers’ consumption. A hybrid approach for object detection is employed to optimize performance efficiency and resource utilization. A high-quality dataset was developed and used to train two state-of-the-art algorithms for the purpose of identifying allergenic foods. The objective of this research is for protecting the immune systems of children from dietary allergies.