Self-supervised learning has become the de facto method for 3D representation in many vision tasks. This calls for 3D understanding without reliance on manual annotations. However, point clouds are inherently discrete and material-insensitive, making it difficult for them to capture the complexity of the 3D world and bridge the modality gap with dense 2D images. To address this issue, we argue that 3D Gaussian ellipsoids are a better representation than point clouds. In this paper, we propose a General Gaussian Model (GGM) that seamlessly integrates autoencoding and autoregressive tasks in a single transformer. We decouple the encoder and decoder architectures to mitigate the impact of task-irrelevant decoder components on the training objective. Furthermore, we introduce an auxiliary regressor to enhance feature extraction capabilities by aligning the encoder’s mask tokens with those generated by the regressor. Considering the significant differences between Gaussian ellipsoids and point clouds in terms of parameters and centroid distribution, we design an innovative fine-tuning scheme for GGM. Extensive experiments indicate that GGM not only surpasses its point cloud counterparts but also shares the merits of both autoencoding and autoregressive, e.g., achieving 88.2% accuracy on ScanObjectNN and 94.00% accuracy on ModelNet40.