Multi-modal object Re-identification (ReID) matches objects between images captured by cameras of different modalities, showing great potentials for practical applications like security monitoring. Despite progress achieved in recent years, existing works have not fully filtered out irrelevant background information. They may also suffer from the problem of information loss when suppressing features in different modalities to the same modality-shared feature. To address these issues, we propose a novel method named FIE for multi-modal ReID, which includes filtering, inference and enhancement stage. In our method, image masking is firstly performed to filter out background-related parts in the input image. After feature extraction, a Feature Filtering Module (FFM) is designed to simultaneously enhance foreground and filter out background information. Afterwards, we present the proposed Cross-modality Inference Module (CIM) to infer modality-shared feature from single-modality feature to retain modality-specific information. To further improve the feature representation, a Multi-scale Enhancement Module (MEM) is proposed to fuse information of different scales. Experiments on three benchmarks demonstrate the superiority of our method compared with the state of the arts.