This page shows visualizations of feature importance generated by applying the technique of integrated gradients to the Inception (v1) object recognition network.
In a nutshell, we examine the gradients of certain counterfactual images obtained by uniformly scaling the pixel intesities in the image at hand, and aggregate these gradients together. The scaled down images are essentially points along the straight line path from the actual image to the black image.
A different summarization of the gradients of scaled down images is to directly visualize them as an animation (GIF), which shows how the gradient of the image changes as the pixel intensities are scaled up.
The integrated gradients method can be implemented in a few lines of code (see attributions.py), is applicable to a variety of deep networks and has some nice mathematical properties. We recommend reading our paper, presently under submission to ICML 2017, for more details.
Below are some visualizations of interior gradients (as a GIF) and integrated gradients for some images from the ImageNet object recognition dataset. For comparison, we also show a visualization of the gradients at the actual image.