Advances in information extraction have enabled the automatic construction of large knowledge graphs (e.g. Yago, Wikidata, or Google KG), which are widely used in many applications like semantic search or data analytics. However, due to their semi-automatic construction, KGs are often incomplete. Rule learning methods, concerned with extracting frequent patterns from KGs and casting them into rules, can be applied to predict potentially missing facts. A crucial step in this process is rule ranking. Ranking rules is especially challenging over highly incomplete or biased KGs (e.g. those predominantly storing facts about famous people), as biased rules might fit the data best and be ranked at the top based on standard statistical metrics like rule confidence. To address this issue, we propose an innovative framework that integrates pre-trained Language Models (LMs) into rule ranking. Unlike prior works that rely solely on KG embeddings, our approach uses LMs to evaluate the predictive quality of rules, leading to a hybrid ranking system that combines statistical metrics with LM-based scores. Preliminary experiments on the Wiki44K dataset demonstrate the potential of this method, achieving higher precision in rule-based predictions. This study highlights a novel application of LMs for rule learning and ranking, advancing the field by providing a flexible and robust approach for handling incomplete and biased KGs.