The increase in the number of parameters in Large Language Models (LLMs) is accompanied by a significant rise in memory consumption, computational resource requirements, and energy costs during both pre-training and fine-tuning. A major contributing factors to these limitations is the use of the backpropagation algorithm, which necessitates the global propagation of gradients and the storage of intermediate activations at each layer. This report shows an approach to mitigate these costs by combining localized learning based on Restricted Boltzmann Machine (RBM) with the Low-Rank Adaptation (LoRA) method. RBM provides training using localized weight updates that are independent of the global error signal, and can efficiently approximate the distribution of input data without having to save activations for later gradient calculations, which in turn reduces the amount of memory required for fine-tuning the network. The proposed method reduces memory consumption by 9% compared to LoRA, while demonstrating only a slight decrease in performance for modern LLMs architectures.