Our natural environment has witnessed severe issues over the past decades caused by the construction industry. Cement manufacturing and gypsum drywall disposal are two of the most debatable topics in this domain due to their direct or indirect negative impact on the natural environment such as air pollution, water contamination, climate change, and soil degradation to name a few. To address these challenges, several solutions have been introduced by researchers to date and one of them is replacing the cement in concrete with recycled gypsum from waste drywall. The resulting concrete (gypsum concrete) has been evaluated in terms of short-term compressive strength (up to 90) days and the result has supported the theory of gypsum concrete functionality to some extent. In this paper, the long-term physical and mechanical properties of gypsum concrete as well as the overall durability of this novel material under two environmental exposures have been evaluated. Concrete cylinders with dimensions of 100 × 200 mm were prepared. Three different mix designs for cementitious material were considered. 0, 10, and 20% of cement were replaced by gypsum powder (fine gypsum). Specimens were planned to be tested on day 7 and day 28. The cylinders were exposed to airdry and freshwater conditions. Afterward, the compressive strength of cylinders was tested at 1000, 3000, and 6000 h after the curing period to evaluate the durability of specimens. Moreover, the alterations in the volume and weight of specimens over time are noted and analyzed.