<p>Diffeological spaces firstly introduced by J. M.&#xa0;Souriau in the 1980&#xa0;s are a natural generalization of smooth manifolds but optimization techniques are only known on manifolds so far. Generalizing these techniques to diffeological spaces is very challenging because of several reasons. One of the main reasons is that there are various definitions of tangent spaces which do not coincide. Additionally, one needs to deal with a generalization of a Riemannian space in order to define gradients which are indispensable for optimization methods. One main aim of this paper is a suitable definition of a tangent space in view to optimization methods. Based on this definition, we present a diffeological Riemannian space and a diffeological gradient, which we need for the formulation of an optimization algorithm on diffeological spaces. Moreover, in order to be able to update the iterates in an optimization algorithm on diffeological spaces, we present a diffeological retraction and the Levi-Civita connection on diffeological spaces. This paper also illustrates the novel objects by examples. Finally, we formulate the steepest descent method on diffeological spaces and apply it to an example.</p>