The rise of large language models (LLMs) has opened new opportunities in Recommender Systems (RSs) by enhancing user behavior modeling and content understanding. However, current approaches that integrate LLMs into RSs solely utilize either LLMs or conventional recommender models (CRMs) to generate final recommendations, without considering which data segments LLMs or CRMs excel in. To fill in this gap, we conduct experiments on MovieLens-1M and Amazon-Books datasets, and compare the performance of a representative CRM (DCNv2) and an LLM (LLaMA2-7B) on various groups of data samples. Our findings reveal that LLMs excel in data segments where CRMs exhibit lower confidence and precision, while samples where CRMs excel are relatively challenging for LLMs, requiring substantial training data and a long training time for comparable performance. This suggests potential synergies in the combination between LLMs and CRMs. Motivated by these insights, we propose Collaborative Recommendation with conventional Recommender and Large Language Model (dubbed CoReLLa). In this framework, we first jointly train LLMs and CRMs and address the issue of decision boundary shifts through alignment loss. Then, the resource-efficient CRMs, with a shorter inference time, handle simple and moderate samples, while LLMs process the small subset of challenging samples for CRMs. Our experimental results demonstrate that CoReLLa outperforms state-of-the-art CRMs and LLMs methods significantly, underscoring its effectiveness in recommendation tasks.