Conventional Large Language Model (LLM) benchmarking methods, primarily reliant on static Question-Answer (QA) pairs fall short in accurately assessing these models’ ability to perform tasks that involve multiple steps of planning, information seeking and interactive problem solving. To address these limitations, we introduce a novel interactive benchmark designed for LLMs with image support. Our benchmark presents a virtual escape room for the model to solve through interacting with and navigating the environment using textual commands. This approach emphasizes active information seeking and seeks to assess whether models performing well in isolated tasks generalizes to compound tasks consisting of multiple subtasks, in addition to mitigating concerns about data contamination. We tested the benchmark on five models, with only GPT o1 being able to fully complete it. To demonstrate that the benchmark can be used to test specific hypotheses on capabilities of LLMs, we also prepared a variant that substitutes instances of Latin letters with invented symbols in order to test whether the image based reasoning capabilities of the tested models rely on familiarity with the symbols being used, with the results suggesting that is indeed the case. We conclude that our benchmark is capable of offering insights into the true reasoning abilities of MLLMs and providing a scalable and extensible framework for keeping up with future developments in the field. To facilitate this, we have released the source code of the benchmark on github.