In recent years, electronic health records (EHRs) have been widely applied in medical-related tasks, such as assisting physicians in diagnosis and enabling researchers to conduct analytical and predictive tasks, signifying their crucial significance. However, due to data privacy concerns and limited data samples, obtaining a sufficient amount of high-quality EHR data remains a challenge in real life. Synthetic EHR data can address this issue, but due to data tight interrelations (e.g., hypertension patients tend to have higher blood pressure readings) and the need to adhere to real-world logic (e.g., heart rates cannot be below 0), existing methods struggle to capture label correlations and generate data with constraints. In this paper, we propose a multi-labeled time-series GAN with constraints (LTGAN) for EHR data generation to address these issues. It consists of multi-label and time-series data generation modules, with added constraints to limit the range and distribution of the synthetic EHR data. The overall evaluation of our model includes resemblance evaluation and utility evaluation. Using curated EHR dataset eICU, the experimental results demonstrate that our proposed method outperforms existing models in EHR data generation problem.