We consider an auction-based crowdsourcing system. A requester is faced with a binary choice question and decides to hire workers to answer the question. The workers can ask prices for answering the question and the requester can choose to hire which workers based on their skills and ask prices. We model the problem as a mechanism design problem and characterize the optimal hiring policy. We show that the problem of computing the accuracy of a given set of workers is #P-hard. However, we prove that choosing at most k workers into committee can achieve at least \(1/\lceil n/k \rceil \) of the optimal utility. Finally, we also provide a polynomial algorithm for computing the optimal hiring strategy when the number of workers’ skill levels is constant.