The Indian economy is predominantly based on agriculture which provides employment to most of the people in the country and contributes a lot to the GDP of the country. Cotton forms a pivotal cash crop that provides income for innumerable farmers and sectors around the textile and medical industries. On the downside, cotton crops are prone to pest and disease attacks which creates a risk for yield and income for farmers in India. Identifying pest and disease outbreaks promptly is very important in protecting crops from excessive use of pesticides and wastage of crops and resources. This paper proposes a new pest detection system for cotton crops using image processing and machine learning techniques. The system consists of pest detection on R-CNN-based networks which uses the transfer learning technique for feature extraction and uses SVM for pest classification. The implementation of these techniques contributes to the improvement of early due detection which will be helpful for farmers in taking the appropriate steps on time to avoid losses on their crops. Moreover, it encourages the implementation of better pest control strategies in the country hence encouraging less agricultural losses.