Synthetic data has become an increasingly popular means of sharing data without revealing sensitive information. Though Membership Inference Attacks (MIAs) are widely considered the gold standard for empirically assessing the privacy of a synthetic dataset, for simplicity, practitioners and researchers often rely on proxy metrics such as Distance to Closest Record (DCR). These metrics estimate a synthetic dataset’s privacy by measuring the similarity between the training data and generated synthetic data. This similarity can also be compared against the similarity between the training data and a disjoint holdout set of real records to construct a binary privacy test. If the synthetic data is not more similar to the training data than the holdout set is, the synthetic dataset passes the privacy test and is considered private. In this work, we show that, while computationally inexpensive, DCR and other distance-based metrics fail to identify privacy leakage. Across multiple datasets and both classical models, such as Baynet and CTGAN, and more recent diffusion models, we show that datasets deemed private by proxy metrics are highly vulnerable to MIAs. We similarly find both the binary privacy test and the continuous measure based on these metrics to be uninformative of actual membership inference risk. We further show that these failures are consistent across different metric hyperparameter settings and record selection methods. Finally, we argue that DCR and other distance-based metrics are flawed by design and show an example of a simple leakage they miss in practice. With this work, we hope to motivate practitioners to move away from proxy metrics to MIAs as the rigorous, comprehensive standard of evaluating privacy of synthetic data, in particular to make claims of datasets being legally anonymous.