Large Language Models (LLMs) have shown impressive capabilities in solving complex mathematical problems, making them valuable tools for education, research, and automated tutoring. However, top-performing models on benchmarks like MATH500, such as GPT-4 and DeepSeek-R1, are often large, proprietary, and costly to use, limiting their accessibility. In contrast, smaller open-source models are more affordable and easier to deploy locally but typically underperform in mathematical reasoning tasks. In this work, we explore the math-problem-solving potential of six small-scale, open-source LLMs (all under 10 billion parameters): Arithmo-Mistral-7B, MAmmoTH-7B, MAmmoTH-8B, MetaMath-7B, MetaMath-Llemma-7B, and MetaMath-Mistral-7B, on the MATH500 benchmark. To enhance their accuracy, we apply two “test-time” ensemble strategies: (1) Intra-model ensemble, where each model generates five independent outputs and the most frequent prediction is selected; and (2) Inter-model ensemble, where 2-level majority voting is performed: first at the intra-level, then across all the models in the ensemble. Our results show that the Intra-model ensemble consistently improves performance over individual runs, and combining outputs across models yields further gains. An ensemble of all six models achieves 38% accuracy on MATH500, outperforming each model’s standalone accuracy (typically 20–30%). Furthermore, a smaller ensemble of the top three models, MetaMath-Llemma-7B, MAmmoTH-7B, and MAmmoTH-8B, achieves a best result of 39.4%. These findings demonstrate that lightweight ensemble techniques can significantly boost math problem solving performance in small LLMs without additional training or expensive computational resources.