Protecting crops from unauthorized individuals and undesired animals is a never-ending problem in agriculture. Conventional surveillance techniques sometimes fail to distinguish between dangerous and innocuous intrusions, and in certain situations, they even endanger people. Through the use of computer vision and sophisticated detection techniques, this research presents a Smart Agricultural Surveillance System that employs smart technology to identify and respond to various things in real-time. Our system can categorize humans and animals into three groups: less harmful, more harmful, and harmless. It does this by using the standard YOLOv9 (You Only Look Once) concept on a tiny and reasonably priced Raspberry Pi 4 device. The system reacts in a way that helps safeguard the crops and ensures worker safety based on these categories. Innocent people, less dangerous animals (such as birds, cows, sheep, and monkeys), and more dangerous creatures (such as wild boars, deer, bears, elephants, and foxes) can all be distinguished by the method. The system uses an LED to indicate that the electric fence has been triggered for more hazardous animals, sounds an alarm for less dangerous animals to frighten them away, and does nothing for humans, depending on what it detects. This technology offers farmers an effective, automated method of safeguarding their crops while guaranteeing the safety of both people and animals by employing intelligent classification and only initiating reactions when required.