Local decision explainability is critical for the adoption and acceptance of AI in medical domains such as polypharmacy. In this paper, we will apply LIME for decision explanation of AI_Polypharmacy models. LIME will rely on perturbing patient data to generate new individuals similar to the one for which the diagnosis is made. We outline the challenges we face in this study of LIME applied to polypharmacy: the size of our dataset, which is very large; the specificity of the data, where all features are binary and unbalanced; and the mismatch of LIME with these pharmacy data. We address these challenges by first identifying the LIME limitations we face, and then proposing an effective strategy for using the synthetic pharmacy data to form an information-rich dataset for building locality in LIME. To do this, we formulate the problem as a two-step data balancing exercise so that fidelity in LIME can be ensured on our data and on our polypharmacy AI model. The challenge in this step is how to balance a massive, highly unbalanced database while respecting the constraints and goals of the polypharmacy domain that LIME will use to generate valid explanations. In addition, we choose an appropriate similarity distance for patient records to handle both the imbalance of our data and its binary categorical nature. The results of this experiment show that the proposed approach applied to massive, unbalanced binary data such as those in our domain, and overcomes the fidelity problem of LIME. As a result, it was able to generate valid explanations in the context of polypharmacy.