The quality of a rare earth element (REE) concentrate can be negatively affected by the presence of fluorite. This study aims to better understand the high-intensity magnetic fluorite separation process from a finely ground REE concentrate to purify a REE-flotation concentrate. Using a flotation concentrate produced during mini-piloting, particle size and suspected agglomeration were first investigated using laser diffraction particle sizing. Laboratory-scale magnetic separation tests were carried out using a factorial design with magnetic induction intensity (10,000–20,000 Gauss) and the solids fraction of the pulp (10–20% w/w) as variables. The material tested comprised monazite and bastnaesite as REE-bearing minerals, as well as Fe-bearing dolomite, ankerite, apatite, and fluorite, the latter two being assumed to be diamagnetic. Results showed significant particle agglomeration in the concentrate before ultrasonication. Chemical and mineralogical analyses allowed for description of the recovery of the REE and gangue minerals in terms of their size, magnetic properties, and liberation. While magnetic separation was effective at rejecting fluorite, a significant portion was carried over to the magnetic fraction through association with paramagnetic ferroan dolomite and REE minerals. While a REE recovery to the magnetic product of 80% has been achieved, REE losses mainly resulted from difficulty in attracting fine (<5 μm) particles, even after multiple passes. Liberation may also hamper REE recovery which increased upon an HCl preleaching step. This study provides a better understanding of the impact of particle size and liberation on high-intensity magnetic separation of REE-bearing minerals from diamagnetic minerals.