Interpretability · EMNLP 2024 Main

MMNeuron

Discovering neuron-level, domain-specific mechanisms inside multimodal language models.

Jiahao Huo^1,3, Yibo Yan^1,2, Boren Hu^1,2, Yutao Yue^1,2, and Xuming Hu^1,2

¹ The Hong Kong University of Science and Technology (Guangzhou) · ² The Hong Kong University of Science and Technology · ³ Tongji University

Paper / arXiv ↗ Code ↗ Motivation & method ↓

Project overview

MMNeuron neuron-level interpretation illustration

MMNeuron traces how multimodal information travels through domain-specific neurons in an MLLM.

Motivation

Domain competence should be traced to internal computational mechanisms.

Aggregate benchmark scores reveal what a multimodal model can do, but not where specialized visual knowledge is represented or whether identified components have a causal role.

Multimodal large language models operate across heterogeneous domains such as documents, medicine, commerce, advertising, and remote sensing. Existing interpretability analyses commonly describe attention patterns or global representations, leaving the neuron-level organization of domain-specific behavior insufficiently characterized.

MMNeuron studies whether a sparse subset of neurons consistently distinguishes individual domains, how those neurons influence intermediate predictions, and whether intervening on them changes downstream accuracy. The analysis connects activation statistics, layer-wise mechanisms, and controlled interventions within one framework.

VenueEMNLP 2024 Main

LensNeuron-level analysis

DomainsFive visual domains

Main findings

Evidence for sparse domain specialization

<1%

Only a small fraction of neurons are identified as domain-specific in the reported analysis.

5 domains

Specialized behavior is studied across document, medical, commerce, advertising, and remote-sensing examples.

10%

Reported accuracy can change by up to this amount when domain-specific neurons are manipulated.

Methodology

Identification, interpretation, and intervention

The framework moves from statistical discovery to mechanistic analysis and finally to causal validation.

Domain-specific neuron identification

Activation statistics are collected for examples from five visual domains. Neurons with consistently selective responses are identified by contrasting within-domain activation against the remaining domains across model layers.

Layer-wise mechanism analysis

The discovered neurons are organized into a three-stage processing account. Logit-lens projections map intermediate activations into the output vocabulary, revealing how domain evidence emerges and propagates toward task predictions.

Causal intervention

Selected neuron activations are enhanced or suppressed during inference. Measuring domain-specific accuracy before and after intervention tests whether the neurons contribute functionally to behavior rather than merely correlate with it.

Visual results

Mechanistic analysis across domains

From model anatomy to domain examples, the figures make an internal phenomenon concrete.

MMNeuron framework — **Neuron-level analysis framework**The pipeline discovers domain-specific neurons, explains their mechanism, and validates their causal influence.

MMNeuron neuron distribution PCA — **Domain separation**Projection of neuron representations reveals structure shared by and specific to different visual domains.

MMNeuron logit lens analysis — **Logit-lens mechanism**Intermediate neuron activity can be followed into the model’s domain-specific prediction space.

MMNeuron document domain example — **Documents**Domain-specific responses on document-style visual inputs.

MMNeuron medical domain example — **Medicine**Specialized visual knowledge is reflected in medical examples.

MMNeuron commerce domain example — **Commerce**Commerce-oriented visual reasoning activates a different internal subset.

MMNeuron advertising domain example — **Advertising**Domain cues in advertising images provide another test of specialization.

MMNeuron remote sensing domain example — **Remote sensing**Neuron specialization extends to remote-sensing visual inputs.

Citation

BibTeX

@inproceedings{huo2024mmneuron,
  title     = {MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model},
  author    = {Huo, Jiahao and Yan, Yibo and Hu, Boren and Yue, Yutao and Hu, Xuming},
  booktitle = {Proceedings of EMNLP},
  year      = {2024}
}