Despite significant advancements in general Image Captioning (IC), captions are most effective when they transcend obvious descriptions and engage the user’s interest. Personalized image captioning (PIC) addresses this need by generating descriptions that incorporate the user’s prior knowledge, such as active vocabulary, writing style, and other user-specific information. This task is complicated by the large number of users on social networks and the evolving nature of user preferences for image captions. This paper proposes a hybrid model that combines a pre-trained model with a retrieval-based memory mechanism to tackle the Personalized Image Captioning problem. Our method involves two main phases: (1) Constructing User Memory (UM), and (2) generating image descriptions using the pre-trained model and UM. We create external user memories containing word-level knowledge in textual and image contexts, referred to as user historical context. During the second phase, we utilize a retrieval method to measure the similarity between historical and current contexts, retrieving relevant knowledge from the UM to feed into the pre-trained model to generate descriptions word by word. Our framework does not require additional parameters, supports expansion, and enables explicit and interpretable memorization of user knowledge. Furthermore, our method outperforms baseline models and achieves results comparable to state-of-the-art methods.