Cross-lingual sentence representation learning involves mapping text to a vector space, ensuring that the distances between vectors representing semantically similar texts are equally close, and enabling the sharing of a common semantic space among vectors from diverse languages. Existing methods predominantly utilize cross-language datasets centered around English for training sentence representation models, with evaluation primarily focused on cross-language retrieval tasks. This approach often leads to suboptimal performance when these models are applied to monolingual semantic text similarity tasks in non-English settings. We propose a strategy that integrates heterogeneous data from various languages, computes perplexity for each sample using a generative model, and employs focus learning in loss calculation to fully utilize the training data. By training on Chinese-English data, our model (Bi-CSE) achieves state-of-the-art results in bilingual sentence representation, getting a 14% absolute increase in the Spearman correlation coefficient on the Chinese semantic textual similarity task (STS), showcasing the efficacy of our approach. We will release our code, data, and models to the public.