Conserving natural resources and addressing environmental concerns are significant motivations for repurposing waste materials in pavement construction. Aggregates, fillers, and asphalt binders are conventionally amalgamated to formulate the asphalt mixture for constructing the pavement's surface layer. In the Northeastern region of India, the availability of conventional aggregates is scanty. Therefore, it creates a platform to work on using different building demolition wastes to replace traditional fillers in asphalt mixtures. This research looks at using three types of building demolition waste, such as brick dust, plaster dust, and concrete block dust, along with conventional fillers, i.e., stone dust and lime dust. To determine the physical and chemical properties of employed fillers, tests such as uncompacted bulk mass, particle size distribution, methylene blue, X-ray diffraction, and specific gravity are performed. Fillers and asphalt binder are combined in three distinct filler/binder (F/B) ratios (0.6, 0.8, and 1) to create asphalt mastic. On these mastics, rheological characteristics (complex modulus, storage modulus, loss modulus, phase angle, and complex shear modulus) are examined. To further examine the performance of the asphalt mix, moisture susceptibility, creep tests, and cantabro abrasion tests are performed. The results of the entire study indicate that, aside from traditional filler, or stone dust, plaster dust performs better, with an F/B ratio of 0.8 due to its cementitious properties.