Model quantization is a widely-used technique to compress and accelerate neural networks. Among various quantization approaches, Data-Free Quantization (DFQ) has emerged as a promising solution, as it eliminates the need for original data during the quantization process. This characteristic makes DFQ particularly suitable for privacy-sensitive and confidential scenarios. However, a critical limitation of existing DFQ methods is their reliance on synthetic data for model calibration, which not only compromises accuracy but also introduces significant computational overhead and time costs. To address these challenges, we propose FQuant, an efficient quantization framework capable of completing the quantization process within seconds. Specifically, FQuant is optimized for deployment on inference-dedicated devices with constrained computational and memory resources. Through extensive analysis, we observe a notable divergence in activation distributions across different intermediate layers during inference, while the distributions within the same layer remain highly consistent. Leveraging this insight, we employ a minimal set of test images as a calibration dataset to capture layer-wise activations, thus eliminating the need for private data. Subsequently, the quantization levels for each layer are determined using the K-Means algorithm, and the corresponding quantization parameters for both weights and activations are computed to finalize the quantization process. Remarkably, our method achieves an average accuracy improvement of 3.9% compared to state-of-the-art DFQ methods under 4-bit quantization, without requiring fine-tuning or synthetic data.