Traditional Machine Learning methods rely on expert-provided labels for training, which can be costly and time-consuming. Crowdsourcing platforms have emerged as a possible solution, offering access to non-expert annotators at a cheaper cost. These come with some limitations, since they may be unreliable. Active Learning is a technique proposed to reduce the need for labels by selecting the most informative samples for labeling. However, traditional Active Learning assumes an unrealistic scenario, operating within the assumption of a perfect oracle that always provides the correct label. This scenario does not reflect real-world conditions, where there is no infallible source of ground truth. Moreover, there might be multiple annotators involved, each one with varying levels of expertise and reliability. In this paper, we propose a new approach to reduce the dependence on ground truth or domain-expert labels. The proposed approach is capable of learning the expertise and bias of unreliable annotators without using ground truth labels. Our approach applies Active Learning to the multiple annotator scenario, by not only selecting the most informative sample but also leveraging annotator expertise to choose the annotator that is most likely to provide the correct annotation for a given instance. We describe three Active Learning annotator selection methods that aim to maximize the number of correct annotations by selecting the most appropriate annotator for a given instance. The results demonstrate that these methods outperform baseline approaches, resulting in higher-quality annotations and, in turn, increased model accuracy.