Lena Voita
Information-Theoretic Probing with Minimum Description Length
What happens when relationships between Alice and Bob have gone too far.
How can you know whether a model has learned to encode a linguistic property? The most popular approach to measure how well pretrained representations encode a linguistic property is to use the accuracy of a probing classifier (probe). However, such probes often fail to adequately reflect differences in representations, and they can show different results depending on probe hyperparameters. As an alternative to standard probing, we propose information-theoretic probing which measures minimum description length (MDL) of labels given representations. In addition to probe quality, the description length evaluates “the amount of effort” needed to achieve this quality. We show that (i) MDL can be easily evaluated on top of standard probe-training pipelines, and (ii) compared to standard probes, the results of MDL probing are more informative, stable, and sensible.
Elena (Lena) Voita is a PhD student interested in (mostly document-level) neural machine translation, as well as understanding what and how neural models learn. Previously, she spent 4 years having fun at different parts of Yandex; 2.5 of them (those with most fun) as a research scientist at Yandex Research side by side with the Yandex Translate team.