Roadside perception systems have significant potential in comprehending complex urban traffic scenarios, due to its high perspective and easy deployment. However, single-infrastructure roadside perception hardly perceives a wide range of blind spots and occlusions in restricted traffic area. Due to the lack of cooperative interaction with adjacent roadside sensors. Meanwhile, previous collaboration methods are vehicle-centric which overlook the inherent attribute of roadside sensors (like broader view, pitch, roll, yaw angle, height). Orienting area-coverage scene understanding, we construct an efficient lidar-based roadside cooperative tracker, dubbed as RCTracker. Specifically, deformable sparse fusion (DSF) is developed to capture long-range spatial dependence by adaptive aggregating interest area information, and roadside attention fusion (RAF) further contributes to learn global-local associations among infrastructures in the vicinity. In this pattern, we could reduce large-scale message transmission budgets and dynamic motion blur. Quantitative and qualitative experiments are conducted on publicly real-word RCooper benchmarks to validate the effectiveness of our RCTracker framework. The results report the state-of-the-art cooperative perception performance compared to advanced vehicle-centric method, which also demonstrates the advancement of our tailored cooperative pattern for real-word roadside perception.