Brain graph/network analysis provides an effective way for automated diagnosis of brain disorders such as autism spectrum disorder (ASD) and major depressive disorder (MDD), by exploring the interactions among brain regions from resting-state functional magnetic resonance imaging (rs-fMRI) data. Although deep learning models like graph neural networks (GNNs) and Transformers have increasingly been applied to learn brain graph representations, their effectiveness is often constrained by the scarcity of brain graph data with the difficulties of acquisition and annotation. To address this issue, in this paper, we propose a novel brain graph adaptive co-contrastive learning (BrainGACCL) framework with universum samples for brain disease diagnosis. Rather than relying on the predefined contrastive views or task-independent contrastive learning process as in the existing methods, BrainGACCL automatically produces contrastive views with diverse topologies but similar semantics via task-aware dual-view augmentors. BrainGACCL captures perturbation-invariant representations at both the attribute and structural levels of the graph through a cross-view collaborative contrastive mechanism, as well as incorporates universum samples as additional negative samples to guide contrastive learning. The effectiveness of our BrainGACCL has been validated for diagnosing ASD and MDD across different sites on two real rs-fMRI datasets, using limited labeled data. The implementation code is available at https://github.com/doris-aviva/BrainGACCL .