<p>This paper presents a hardware-aware bit-allocation framework for pre-trained 2D and 3D CNNs to simplify volumetric data processing. The proposed scheme explores tradeoffs between prediction accuracy and hardware limitations by forming a multi-objective optimization problem that is solved using a modified Particle Swarm Optimization (PSO) algorithm. Efficient convergence of the proposed algorithm produces a set of optimized word-lengths for weights and activations for each layer of CNNs that is conducive to an efficient hardware design. The proposed bit-allocation framework satisfies three conflicting objectives of latency, resource utilization, and inference error. The selection of the final word-lengths is from a Pareto-optimal set which is based on given preferred parameters for the target CNN models. The quantization framework has been evaluated for three modern 3D CNNs, I3D, C3D and R(2+1)D. Experimental results have shown a 48%, 45%, and 44% lower resource utilization, and around 20% to 30% reduction in latency for I3D, C3D, and R(2+1)D, respectively, with a negligible loss in accuracy when moving from fixed 8-bit integer implementation to the proposed mixed-precision bit-allocation policy. To demonstrate its scalability and efficacy, the proposed bit-allocation framework has also been tested on a popular 2D CNN, ResNet-50. The results show that our generalized framework gives performance comparable to the state-of-the-art mixed-precision bit-allocation frameworks designed and tuned specifically for 2D CNNs as well.</p>