Agriculture plays an important role in global food security, yet crop disease significantly impacts yield and economic stability. Traditional centralized machine learning approaches for crop disease classification, though effective, pose challenges like data privacy, security, and high communication cost. To overcome these issues, this research introduces AgriFed, a federated learning based framework that enables decentralized training while ensuring confidentiality of data and reduction in data sharing cost. We analyze the performance of number of Convolutional Neural Network based architectures, such as VGG16, GoggleNet, DenseNet, ResNet, MobileNet, and EfficientNet for crop disease classification using TOM2024 dataset, which includes images of tomato, onion and maize crops. The results highlight that federated learning not only preserves data privacy but also achieves comparable or better results than centralized learning. Furthermore, this research explores the balance between model complexity and performance, demonstrating that federated learning is a scalable and efficient method for real-time disease crop detection. These findings illustrate the potential of federated learning in modernizing the agriculture technology while ensuring the privacy of data.