Less than Necessary or More than Sufficient: Validating Probing Dataset Size

The vast body of research is dedicated to interpreting language models, particularly probing them for linguistic properties. As in many other NLP fields, probing works tend to reuse existing datasets, resulting in more and more specialized findings. Introducing new datasets, although necessary for truly typologically diverse studies, requires labor-intensive data annotation. Meanwhile, models become heavier, probing methods inventory enriches, and the cost of probing experiments grows accordingly. To minimize the amount of work annotating new data, and reduce the computational cost of experimenting with the existing data, it will be beneficial to assess dataset size. We propose fractions probing, a novel method of validating probing dataset size. It includes data redundancy test to review existing datasets and data sufficiency test to provide guidance when collecting new ones. We illustrate the method’s applicability with SentEval probing suite, finding that it can be safely reduced. Our experiments are conducted for two models, BERT and RoBERTa, showing the latter to consistently require more data. Fractions probing can be used to analogously investigate other datasets and models.