Recently, Object detection has been dominated by end-to-end object detectors based on Transformer, which eliminates the hand designed anchor and NMS components. However, there are still problems such as network complexity and slow training convergence. In the paper, we construct a lightweight network. It uses all Lightweight Pooling Transformer (LPT) from the backbone network to head detection. The Pooling Backbone is first adopted to get a significant parameter reduction and less memory usage. Then we design a Hybrid Pooling Transformer, which includes a channel and position attention mechanism. The channel attention mechanism uses a spatial pooling pyramid network and the positional attention mechanism uses an adaptive pooling module to focus on the positional information. Experiments in MS COCO 2017 and PASCAL VOC 2012 show that the proposed method is competitive with some state-of-the-art methods. The source code can be obtained at https://github.com/fanghe123/LPT .