As enterprises transition toward cloud-native architectures, the integration of Apache Spark with Kubernetes offers a powerful, flexible foundation for scalable distributed data processing. This combination supports a wide range of workloads—from high-throughput batch jobs, real-time streaming pipelines to scalable machine learning (ML) training and inference jobs – all within a unified, containerized environment. However, running Spark on Kubernetes introduces several operational and performance challenges, including resource allocation, pod scheduling, fault tolerance, and cluster optimization. This paper identifies the key differences between the traditional Spark deployments using YARN as Resource manager and deploying Spark on Kubernetes. The paper identifies not just the benefits of running Spark on Kubernetes, but also the challenges and mitigation strategies. We also investigate architectural considerations, tuning strategies, and deployment best practices for running diverse Spark workloads efficiently in Kubernetes environments. We explore dynamic resource allocation, executor and driver optimization, pod affinity rules, and integration with persistent and object storage solutions. Additionally, we address multi-tenant configurations that require fair scheduling, namespace isolation, and secure resource boundaries to support concurrent data teams and use cases. To operationalize Spark in production, we present DevOps-aligned best practices using Helm charts, GitOps workflows, and CI/CD pipelines to manage versioned, repeatable Spark deployments. Our findings serve as a comprehensive guide for data engineers, ML practitioners, and platform architects seeking to build robust Spark-on-Kubernetes pipelines for batch, streaming, and ML workload.