Record layer activations aggregated by image categories at all timestamps #8
Description
Imagine that at every timestamp, we had access to the cumulative responses of every component of the network for all images in each category.
This would let us could color components of layers based on which image categories most strongly activated them. Hue could indicate the most strongly-activating class, while saturation could indicate the precision of this activation. For example, output neurons that are finely-tuned to particular classes would be heavily saturated, while we might expect earlier layers to show less saturation since they probably capture more low-level visual features that aren't specific to any particular class (such as lines at a particular orientation). When moving the timeline slider, shifts in hue and saturation would show how different parts of the model become tuned to particular input classes.
This could be difficult to implement. The data storage requirements aren't huge if we aggregate the responses by image class, since we would be storing k extra copies of the model at each timestamp (one per class). Ideally, we would compute these aggregates during regular model training and testing. A naive approach would be to process each image through the network at each timestep in an offline batch job. A better approach would be to segment the training images by class, then pass them through a modified testing pipeline that aggregates all intermediate values.
We'd also have to decide whether the coloring will be based on the images' predicted or true classes.