In this paper we introduce an inverse semigroup \(\mathcal {S}(E,C)\) associated to a separated graph (E, C) and describe its internal structure. In particular we show that it is strongly \(E^*\) -unitary and can be realized as a partial semidirect product of the form \(\mathcal {Y}\rtimes \mathbb {F}\) for a certain partial action of the free group \(\mathbb {F}=\mathbb {F}(E^1)\) on the edges of E on a semilattice \(\mathcal {Y}\) realizing the idempotents of \(\mathcal {S}(E,C)\) . In addition we also describe the spectrum as well as the tight spectrum of \(\mathcal {Y}\) . We then use the inverse semigroup \(\mathcal {S}(E,C)\) to describe several “tame” algebras associated to (E, C), including its Cohn algebra, its Leavitt-path algebra, and analogues in the realm of algebras, like the tame algebra \({\mathcal {O}}(E,C)\) and its Toeplitz extension \(\mathcal {T}(E,C)\) , proving that these algebras are canonically isomorphic to certain algebras attached to \(\mathcal {S}(E,C)\) . Our structural results on \(\mathcal {S}(E,C)\) imply that these algebras can be realized as partial crossed products, revealing a great portion of their structure.