Vision-based tactile sensors can provide robots with multiple tactile modalities simultaneously, representing a crucial solution of multimodal tactile perception for general-purpose robots. This study focuses on enhancing object softness discrimination capability through an improved elastomer design for vision-based tactile sensors. We observed that vision-based tactile sensors employing conventional homogeneous elastomers may exhibit sensitivity degradation in softness discrimination under specific conditions, especially when the object is smooth and homogeneous. To address this issue, we established a mechanical model of softness perception, theoretically revealing both the necessary and sufficient conditions for this sensitivity degradation phenomenon. Based on this theoretical framework, we developed quantitative sensitivity metrics to evaluate softness discrimination sensitivity and subsequently designed a gradient-stiffness elastomer through optimizing the sensitivity metric. Both finite element simulations and prototype experiments demonstrate that the improved elastomer significantly enhances the sensor’s softness discrimination performance.