Refining LLM self-evaluation
LLMs sometimes reach correct answers for the wrong reasons. Estimating confidence for each piece of supporting evidence, rather than for the answer alone, makes such spurious correctness detectable.
Our research asks how far we can trust AI that appears to read between the lines, and how we can tell. We organize our work around three themes:
Can we tell whether a model understands? Fluent output is not proof of understanding. This line of work spells out the conditions a model that truly reads between the lines should satisfy: being right for the right reasons, recognizing why an argument is flawed, and asking the question that exposes its weakness. It then builds tests that reveal how far current models meet those conditions.
LLMs sometimes reach correct answers for the wrong reasons. Estimating confidence for each piece of supporting evidence, rather than for the answer alone, makes such spurious correctness detectable.
Labeling an argument as fallacious says little about why it is flawed. Templates and slot-filling capture the underlying logic of fallacious arguments, which current models struggle to recover.
A good Socratic question targets a specific weakness in an argument. FOCUS is a benchmark for generating such questions, grounded in specific spans of the argument.
What happens inside a model when it seems to understand? Behavior alone cannot tell genuine inference from a convenient shortcut. By probing and intervening on a model's internals, this line of work identifies the computation behind in-context learning, multilingual processing, task representation in vision–language models, and a model's own judgment of what it knows, so that success on the outside can be checked against what happens inside.
In-context learning lets a model pick up a task from a few demonstrations, but how it does so remains unclear. Decomposing the process into a few interpretable steps, we explain how models extract, and discard, task information from demonstrations.
Multilingual language models appear to process many languages in a shared latent space. Our analysis traces how they move between language-specific and shared spaces, and shows that specific "transfer neurons" play a key role in these transitions.
Vision–language models encode each task they perform as an internal representation. This work examines whether such task representations compose additively.
Can a language model tell whether it knows the answer to a question? This work investigates the internal mechanisms behind such judgments.
Can we make models understand, not just seem to? Diagnosis should lead to repair. Building on what testing and dissection reveal, this line of work intervenes on a model's internals to remove knowledge it should not have and to make its reasoning sounder, and checks that the change is real rather than cosmetic.
Machine unlearning aims to make a model forget harmful knowledge, but it is unclear whether that knowledge is truly erased or merely hidden. With this question in mind, we develop unlearning methods that are efficient and robust.
Even with chain-of-thought prompting, language models still make logical errors. Intervening on attention heads improves their chain-of-thought logical reasoning.
Past grants are listed on Prof. Inoue’s website.