In recent years, distributional reinforcement learning has produced many state-of-the-art results in typical reinforcement learning benchmarks, such as the suite of Atari games. Increasingly sample-efficient distributional algorithms for the discrete action domain have been developed over time, which vary primarily in the way they parameterize their approximations of value distributions, and how they quantify the differences between those distributions. In this work, we transfer three of those algorithms - Quantile Regression Deep Q-Network (QR-DQN), Implicit Quantile Networks (IQN) and Fully Parameterized Quantile Function (FQF) - to the continuous action domain by extending two powerful actor-critic algorithms - Twin Delayed Deep Deterministic policy gradient (TD3) and Soft Actor-Critic (SAC) - with distributional critics. We investigate whether the relative performance of the methods for the discrete action space translates to the continuous case. To that end, we compare them empirically on a set of continuous control tasks (Ant, HalfCheetah, Hopper, Humanoid and Walker2D). Our results indicate qualitative invariance regarding the number and placement of distributional atoms in the deterministic, continuous action setting.