Explainable AI

Shapley Values & SHAP

Cooperative game theory as a unified framework for feature attribution.

The origin: a fair division problem

Lloyd Shapley (1953) asked: given a coalition of players that together earn some payout, how should the payout be split fairly? He defined four axioms (efficiency, symmetry, dummy, linearity) and proved there exists a unique attribution rule satisfying all four. That rule is the Shapley value.

The translation to machine learning, due to Štrumbelj & Kononenko (2014) and made canonical by Lundberg & Lee's SHAP (NeurIPS 2017), is:

The formula

The Shapley value of feature $i$ is the weighted average of its marginal contributions across all possible coalitions:

Read this as: for every subset $S$ that does not contain feature $i$, measure how much adding $i$ changes the value ($v(S \cup \{i\}) - v(S)$), and average these marginal contributions, weighted so each "coalition size" contributes equally.

The four axioms (and why uniqueness matters)

1. Efficiency. $\sum_i \phi_i = v(N) - v(\emptyset)$. The attributions exactly account for the prediction relative to the empty baseline.
2. Symmetry. If two features always contribute equally ($v(S \cup \{i\}) = v(S \cup \{j\})$ for all $S$), then $\phi_i = \phi_j$.
3. Dummy. A feature that never changes the prediction gets attribution zero.
4. Linearity. Attributions are additive across composable games: $\phi_i(v_1 + v_2) = \phi_i(v_1) + \phi_i(v_2)$.

These axioms uniquely determine the formula above. Any feature-attribution method satisfying all four must compute Shapley values. This is the central appeal of SHAP: it isn't one of many possible attribution methods; among methods with these properties, it is the only one.

Why we can't just compute it

The sum has $2^{n-1}$ coalitions. For a model with 50 features that is $\approx 5.6 \times 10^{14}$ evaluations per prediction. Worse, evaluating $v(S)$ requires marginalizing out the missing features, which typically means averaging the model's prediction over a sample of values for the absent features — adding another factor.

All practical SHAP variants are approximations or model-specific exact algorithms:

The baseline problem

"Marginalize out missing features" is harder than it sounds. SHAP needs a notion of $v(S)$ when only the features in $S$ are present — but neural networks don't accept partial inputs. Two common choices:

• Marginal expectation: $v(S) = \mathbb{E}_{X_{\bar S}}[f(X_S, X_{\bar S})]$ over a background distribution. Honest but requires a representative dataset.
• Conditional expectation: $v(S) = \mathbb{E}_{X_{\bar S} \mid X_S}[f(X_S, X_{\bar S})]$. More accurate but requires modeling the conditional, which is itself hard.

Janzing et al. (2020) argue that the choice of baseline encodes implicit causal assumptions, and different baselines yield meaningfully different attributions. This is not a solved problem.

XAI for Computer Vision

Saliency maps, class activation, integrated gradients, occlusion.

For a vision model, the natural form of a local explanation is a heatmap over the input image highlighting which pixels (or regions) drove the prediction. The methods below differ in how they compute that heatmap, and crucially, in whether the heatmap actually reflects the model's reasoning.

Vanilla saliency (Simonyan et al., 2013)

The simplest idea: compute the gradient of the predicted class score with respect to the input pixels. Pixels whose small perturbations most change the output are the "salient" ones.

Cheap (one backward pass). Often noisy. Tends to highlight edges in the image rather than semantically meaningful regions.

Grad-CAM (Selvaraju et al., 2017)

Currently the standard baseline for CNN interpretability. The insight: instead of using gradients with respect to pixels, use gradients with respect to the last convolutional layer's feature maps. Those maps still carry spatial structure (a 14×14 grid for ResNet-50 with 224×224 input) and are semantically richer than raw pixels.

Let $A^k \in \mathbb{R}^{H \times W}$ be the $k$-th feature map of the target conv layer, and $y^c$ be the score for class $c$. Compute channel weights by global-average-pooling the gradients:

Then form a weighted sum of feature maps, gated by ReLU to keep only positively-contributing regions:

The result is a low-resolution heatmap that you upsample to image size. Class-discriminative: different classes produce different maps. Survives the sanity check in Adebayo et al. (2018).

Integrated Gradients (Sundararajan et al., 2017)

A theoretically principled feature-attribution method for any differentiable model. Given an input $x$ and a baseline $x'$ (typically a black image), integrate the gradient along the straight-line path from $x'$ to $x$:

In practice the integral is approximated by a Riemann sum over 20–50 steps. IG satisfies completeness ($\sum_i \text{IG}_i = F(x) - F(x')$, the same property as Shapley efficiency) and implementation invariance (functionally equivalent models give the same attributions). The choice of baseline matters: zero image, blur, or random noise yield different attributions.

LIME for images (Ribeiro et al., 2016)

Partition the image into ~50 superpixels (e.g., via SLIC). Generate ~1000 perturbed versions by randomly masking subsets of superpixels with a neutral color. Get the model's predictions on all perturbations, then fit a sparse linear model that predicts these outputs from the binary mask. The coefficients of the linear model are the superpixel attributions.

LIME is model-agnostic and gives intuitive region-level explanations. It's also slow (1000 forward passes per explanation) and unstable: re-running LIME on the same image with a different perturbation seed can produce visibly different heatmaps.

Occlusion sensitivity (Zeiler & Fergus, 2014)

Slide an opaque gray patch over the image; for each position, record how much the predicted class probability drops. Drops in probability map directly to importance. Intuitive, no gradients needed, very slow (one forward pass per patch position), and conflates the model's reliance on a region with the OOD-ness of the occluded image.

The honest comparison

For most CNN interpretability work today, the default is Grad-CAM for quick visual checks and Integrated Gradients when you need theoretical guarantees. SHAP (via the DeepExplainer or GradientExplainer) is also viable and ties the analysis to the Shapley framework introduced in Tab 2.

Code: Grad-CAM in PyTorch

A minimal-but-real implementation in ~50 lines.

The implementation below loads a pretrained ResNet-50, registers forward and backward hooks on its last convolutional block, runs Grad-CAM on an input image, and returns a normalized heatmap. No external XAI libraries — just PyTorch and torchvision.

import torch
import torch.nn.functional as F
from torchvision import models, transforms
from PIL import Image
import numpy as np

# 1. Load a pretrained model and set it to eval mode.
model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
model.eval()

# 2. Pick the target layer. For ResNet-50, the last conv block is layer4.
target_layer = model.layer4[-1]

# 3. Register hooks to capture the layer's activations and gradients.
activations = {}
gradients   = {}

def fwd_hook(module, inp, out):
    activations['value'] = out

def bwd_hook(module, grad_in, grad_out):
    gradients['value'] = grad_out[0]

target_layer.register_forward_hook(fwd_hook)
target_layer.register_full_backward_hook(bwd_hook)

# 4. Preprocess an input image (ImageNet normalization).
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

img = Image.open('cat.jpg').convert('RGB')
x   = preprocess(img).unsqueeze(0)                  # shape [1, 3, 224, 224]

# 5. Forward pass; pick the predicted class.
logits     = model(x)
pred_class = logits.argmax(dim=1).item()

# 6. Backward pass on the predicted class score.
model.zero_grad()
logits[0, pred_class].backward()

# 7. Grad-CAM: weight feature maps by channel-averaged gradients, ReLU, upsample.
A      = activations['value']                       # [1, 2048, 7, 7]
grad_A = gradients['value']                         # [1, 2048, 7, 7]

alpha  = grad_A.mean(dim=(2, 3), keepdim=True)      # [1, 2048, 1, 1]
cam    = F.relu((alpha * A).sum(dim=1, keepdim=True))   # [1, 1, 7, 7]

cam    = F.interpolate(cam, size=(224, 224),
                       mode='bilinear', align_corners=False)
cam    = cam.squeeze().detach().numpy()
cam    = (cam - cam.min()) / (cam.max() - cam.min() + 1e-8)

# `cam` is a 224x224 array in [0, 1]. Overlay on the image with matplotlib:
# plt.imshow(img); plt.imshow(cam, alpha=0.5, cmap='jet')

What's happening in each step

• Hooks (steps 2–3). Hooks let you grab intermediate tensors without rewriting the model. forward_hook captures activations during the forward pass; full_backward_hook captures gradients during backprop.
• Backward on a scalar (step 6). logits[0, pred_class] is a scalar — the logit for the predicted class. Calling .backward() on it populates grad_A with $\partial y^c / \partial A^k_{ij}$ for every spatial position and every channel.
• Channel weights (step 7). grad_A.mean(dim=(2,3)) implements $\alpha_k^c = \frac{1}{Z}\sum_{i,j} \partial y^c / \partial A^k_{ij}$ — global average pooling of the gradients per channel.
• ReLU. We only care about features whose presence increases $y^c$. Features pushing the prediction away from the target class are zeroed.

Production-ready alternative

For real projects, use a maintained library rather than rolling your own:

from pytorch_grad_cam import GradCAM
from pytorch_grad_cam.utils.model_targets import ClassifierOutputTarget

cam = GradCAM(model=model, target_layers=[model.layer4[-1]])
heatmap = cam(input_tensor=x,
              targets=[ClassifierOutputTarget(pred_class)])[0]

The pytorch-grad-cam package also implements Grad-CAM++, ScoreCAM, AblationCAM, EigenCAM, and others — useful for comparing methods on your own model.

Using SHAP on a CNN

import shap

# Pick a small background set (50–100 images from the training set)
background = train_images[:100]

# DeepExplainer uses DeepLIFT-style propagation with Shapley baselines
explainer   = shap.DeepExplainer(model, background)
shap_values = explainer.shap_values(test_images[:5])

# Visualize per-class pixel attributions
shap.image_plot(shap_values, test_images[:5])

Can SHAP Be Used With LLMs?

Yes — with significant caveats, and not as the right tool for every question.

The short answer: SHAP is well-defined for LLMs, the shap library has dedicated explainers for text models, and it produces useful attributions for classification-style tasks. But the obstacles that make SHAP awkward on images become severe on language, and for the deepest questions about LLM behavior the field is moving toward different techniques entirely.

Why LLMs are hard for SHAP

1. Input dimensionality. A 2000-token prompt with a 100k vocabulary is a vastly higher-dimensional input than even a 224×224×3 image. The coalition sum has $2^{2000}$ terms.
2. Cost per coalition. Each coalition evaluation is a full LLM forward pass, which already costs hundreds of milliseconds. Sampling enough coalitions for stable attributions costs seconds to minutes per explanation.
3. The baseline problem, intensified. What is a "missing" token? Replace with [MASK], with a random token from the vocabulary, or simply delete it and shorten the sequence? Each choice yields different SHAP values and encodes different counterfactual semantics.
4. High-dimensional output. For generation, the "output" isn't a single score but a distribution over thousands of vocabulary items at every position. Which scalar do you attribute? The probability of the actual completion? The logit of a specific token?
5. Compositionality and context dependence. A token's contribution depends heavily on neighboring tokens. The Shapley axiom of linearity is violated in spirit — meaning isn't additive over tokens.

Where SHAP works well on text

Short-text classification with a frozen model. Sentiment analysis, NLI, hate-speech detection, intent classification. The shap library's PartitionExplainer (the default for text) uses hierarchical Shapley: tokens are grouped, attributions are computed at the group level, and the tree is recursively refined where attributions are large. This brings $2^n$ down to something tractable.

import shap
import transformers

# A short-text classification pipeline
pipe = transformers.pipeline(
    "sentiment-analysis",
    model="distilbert-base-uncased-finetuned-sst-2-english"
)

explainer   = shap.Explainer(pipe)
shap_values = explainer(["The film was a delight, but ran far too long."])

# Renders an HTML visualization with each token colored by attribution
shap.plots.text(shap_values)

This works well, runs in a few seconds, and produces interpretable per-token attributions. It is the right tool when the question is "which words in this short input drove this classification?"

Where SHAP struggles or breaks

• Open-ended generation. "Why did the model write this paragraph?" doesn't reduce to single-token attribution because the answer is a structured object.
• Long contexts. 32k or 128k token windows make even PartitionSHAP impractical. You can sample, but variance becomes unacceptable.
• Behavioral questions. "Does this model have a sycophancy bias?" is a global property, not a per-token attribution. SHAP doesn't address it.
• Causal reasoning. SHAP attributions are correlational. They tell you what the model uses; they don't tell you what would happen if you intervened on a feature.

What practitioners reach for instead

Gradient-based attribution on embeddings

Integrated Gradients still works on LLMs if applied to token embeddings rather than tokens themselves. The captum library (PyTorch) and transformers-interpret wrap this conveniently. Faster than SHAP, often comparable quality for short inputs.

Attention attribution

Transformers expose attention weights for free. Visualizing them is tempting and common — but Jain & Wallace (2019) and Wiegreffe & Pinter (2019) demonstrated that attention weights are not reliable explanations: alternative attention patterns can produce the same output. Treat attention visualizations as suggestive heuristics, not faithful explanations.

Mechanistic interpretability

The emerging research frontier for LLMs. Rather than asking "which input tokens mattered," ask "which internal computations does the model perform, and what do they encode?" Active sub-areas:

• Probing — train a small classifier on intermediate activations to detect whether the model encodes a given property (syntax, factuality, sentiment). Limited: presence of information doesn't mean the model uses it.
• Activation patching / causal tracing — replace activations at specific layers/positions with those from a different input; observe how the output changes. Reveals which components are causally involved in a behavior.
• Circuit analysis — identify small sub-networks of attention heads and MLP neurons that implement a specific capability (e.g., the "indirect object identification" circuit in GPT-2 small, mapped by Wang et al. 2022).
• Sparse autoencoders (SAEs) — train an over-complete sparse dictionary on the model's hidden states. The learned features are often monosemantic and human-interpretable. Anthropic's interpretability team has scaled this approach to extract millions of features from Claude-class models (Templeton et al., 2024).

Influence functions

"Which training examples most affected this prediction?" Mathematically elegant but computationally brutal; recent approximations (TRAK, Park et al. 2023) make it feasible on large models.

Self-explanation and chain-of-thought

Ask the model to explain itself. Plausible and cheap. Often unfaithful: stated reasons may not match actual computation. Recent work (Turpin et al., 2023) shows LLMs can produce convincing chain-of-thought explanations that have no relation to their actual decision process when an artificial bias is introduced.