An exploitation of limited natural stone as main coarse aggregate for concrete production adversely influences living environment. Fly ash, a waste formed from coal burning process at several power plants, is a potential alternative to the natural stone. The present study aims to investigate the influence of pelletized aggregate (PA) manufactured from a mixture of cement, fly ash, and CaSO4·2H2O for replacing crushed stone on bulk density and compressive strength of hardened concrete. The PA with a water-to-binder ratio corresponding to 20% by mass, a fly ash-to-binder ratio of 85% by mass, and a CaSO4·2H2O-to-binder ratio of 2% by mass was fabricated through a pelletizing technique by using a pelletizer machine at laboratory. For concrete production, four PA replacements were 0, 25, 50, and 100% by volume of coarse aggregate. Experimental results exhibited that bulk density corresponding to hardened concrete containing PA reduced by 3.3–14.1% as compared with that of reference concrete with 100% crushed stone. The compressive strength corresponding to 3, 7, and 28 days of hardened concretes with PA replacements decreased by 10.0–33.2%, 8.7–39.0%, and 8.6–36.8%, respectively, as compared with that of reference concrete. The higher the PA amount, the lower the bulk density and compressive strength of hardened concrete. The compressive strength had a positive relationship with bulk density of hardened concrete. It can be considered that the optimum replacement of PA was 50% by volume to produce the concrete with the 28-day compressive strength of higher than 40 MPa and the bulk density of 2250 kg/m3, meeting a specification of heavyweight concrete as per TCVN 5574.