Multi-agent reinforcement learning is a promising approach to solve decentralized control problems. However, if states of agents are dependent on other agents’ states, knowing the whole system state is crucial for decision-making. Exchanging state information between agents requires a communication infrastructure, which in practice might be unreliable. In this paper, we focus on a setting where a central controller is present that can store and forward state information of agents. We propose a reconstruction approach that allows agents to predict the state of other agents based on their own observations. During episodes of lost communication, agents can still act according to a previously learned joint policy. We show that the proposed approach is able to achieve a similar performance as a centralized approach, even when the communication between agents is unreliable.