Language models can explain neurons in language models

OpenAI has published research where GPT-4 was used to automatically generate and score explanations for the behavior of individual neurons in large language models. The work focuses on GPT-2, with the team releasing a dataset containing explanations and scores for every neuron in that model. This approach aims to scale interpretability by using one language model to analyze another, addressing the challenge of understanding how LLMs internally represent and process information. While the explanations are acknowledged to be imperfect, the method provides a way to generate hypotheses about neuron function at a scale that would be impossible with manual analysis. For developers building AI workflows, this research hints at future tools that could offer automated insights into model behavior, potentially aiding in debugging and improving system transparency. The dataset serves as a resource for the community to explore and refine interpretability techniques.