Most existing methods integrating large language models (LLMs) into recommender systems directly employ textual side information as input. However, this practice often leads to three issues: stringent length constraints, inclusion of irrelevant noisy tokens, and inadequate utilization of collaborative signals. Moreover, most existing methods rely on manually crafted instruction prompts composed of natural language tokens, whose effectiveness heavily depends on careful design. In this paper, we propose PromptLLM4Rec, an innovative approach that improves LLM-based recommendation with Curriculum Prompt Learning and Cross-Model Semantic Alignment (CMSA). First, we introduce CMSA, a task that aligns a smaller language model with the recommendation-oriented LLM. This enables compression of lengthy text into compact vector representations while maximizing semantic preservation. Second, we propose a Dual-Perspective Feature Fusion module that integrates collaborative and semantic information to achieve holistic item representations. Finally, we design two types of soft prompts and employ a curriculum learning strategy with progressively intensified prompt perturbations. We conduct comprehensive experiments on three real-world datasets, and the results demonstrate the effectiveness and superiority of our proposed method.