In order to provide more comprehensive medical services and personalized health monitoring according to individual needs, Body Area Networks (BANs) have been extensively studied by many researchers. As BANs involve the transmission of personal private data, the security of the communication is of utmost importance. Unfortunately, existing encryption techniques cannot be directly applied with the limited power or computation ability of the sensors in BANs. An alternative direction is to use physiological signals for key agreement. However, many of the current physiological signal-based key agreement schemes either have high overheads or are vulnerable to certain security issues. In this paper, we proposed a novel secret sharing and Bloom filter based key agreement scheme that balances overhead and security for BANs. Also, we use an approximate hash table to free ourselves from the problem of dependence on the ordering of features, which has been neglected by previous researchers. We validate the security of our scheme with based on real datasets and compare overheads with other solutions.