Improving the balance between accuracy, speed, and memory usage remains a central challenge in visual perception. Existing models are usually task or modality specific, limiting scalability. We propose EffiPerception, a plug-and-play framework that enhances efficiency across 2D and 3D perception tasks without redesigning architectures. It consists of three lightweight modules: (1) Efficient Feature Extractors that reduce redundancy in images and point clouds while preserving geometric cues, (2) Efficient Layers including Sparse Down-Sampling (SDS) and Global Spatial Aggregation (GSA) to refine features and improve cross-modal context, and (3) EffiOptim, an 8-bit optimizer that reduces training memory overhead. Our framework is task-agnostic and architecture-compatible, enabling integration into existing pipelines. Experiments on COCO, KITTI, and SemanticKITTI show consistent gains in accuracy, faster inference, lower memory usage, and improved robustness under noise. EffiPerception thus provides a unified, generalizable solution for efficient perception, complementing task-specific architectures and paving the way toward resource-aware deployment in real-world systems.