Testing Internet of Things (IoT) platforms is essential for assessing their reliability, scalability and potential performance issues. Such tests are particularly challenging to set up when these IoT platforms are managing large fleets of devices. A promising approach to address this is to use test frameworks that employ simulated IoT devices to interact with the target IoT platform. This work introduces Fleet sImulator for Scalability Tests (FIST), an extendable test framework that simulates a variety of realistic IoT scenarios, including large fleets of devices and realistic network conditions. The framework’s flexibility is demonstrated through the evaluation of the open-source IoT platform ThingsBoard. We evaluate the capabilities of this platform in dealing with multiple device connections and firmware update tasks. Overall, this study contributes to the development of robust and reliable IoT platforms by providing a comprehensive test framework for performance evaluations.