I'm developing a neuroscience-inspired interpretation of how Transformers work. This is one of the first steps. GitHub repo has the code, and the paper is available on arXiv (also accepted to ICML 2024).