Place recognition plays an important role in computer vision, robotics, and autonomous driving, which aims to determine whether an autonomous vehicle has reached a previously visited scene. Most existing methods rely on a single observation and are limited to a specific sensor modality, which hinders their applicability in real-world scenarios. In this work, we propose a novel RGB-D place recognition framework, called FlexPRNet, that can flexibly work on different sensor modalities by employing a newly designed two-branch (RGB and point-cloud) encoder to exploit spatio-temporal information of adjacent observations to better generate descriptors of places in real-world dynamic environments. Noteworthy, our encoder can work on both indoor and outdoor scenarios. These descriptors are then fed into Graph Convolutional Neural Networks (GCNN) optimized with a triplet loss for place recognition. From the experimental results, our method outperforms the state-of-the-art methods on average up to 3% recall@1 in several place recognition benchmarks. The implementation of our method will be made publicly available.