Probing as a Technique to Understand Abstract Spaces

Abstract

Machine learning models, while very powerful, have their operation obfuscated behind millions of parameters. This obfuscation can make deriving a human meaningful process from a machine learning model very difficult. However, while the intermediate states of a machine learning model are similarly obfuscated, using probing, we can start to explore looking at possible structure in those intermediate states. Large language models are a prime example of this obfuscation, and probing can begin to allow novel experimentation to be performed.