The popularization of mobile phones has promoted research on human activity recognition (HAR) based on smartphone inertial sensors. Traditional centralized approaches achieve remarkable accuracy performance, but bring unacceptable latency and raise the risk of privacy leakage. Federated learning (FL) emerges and rises to mitigate these issues by collaboratively training DNN models with edge devices without transmitting raw data, which is communicational inefficient due to the transmission of a large volume of parameters. As a light-weight, fast, and robust learning method, hyperdimensional computing (HDC) offers an alternative for classification tasks. In this work, we propose CoR-FHD, a communication-efficient and robust FL system training hyperdimensional (HD) model for smartphone-based HAR applications. In CoR-FHD, we binarize the real-valued HD models to minimize communication costs and increase robustness against network fluctuations. However, information loss attributed to binarization causes unavoidable accuracy drop. To settle this problem, a self-assisted approach is proposed to update the HD model on each node. Moreover, we point out the disadvantages of the current HDC retraining method and redesign the retraining mechanism to further improve the recognition accuracy. Experimental results based on two real-world HAR datasets demonstrate that CoR-FHD saves communication cost by 8-16 \(\times \) and 30-88 \(\times \) compared to previous work of federated HDC and DNN learning, and performs highly robustness against unreliable networks while achieving considerable accuracy.