Hierarchical abstraction of information in Deep Neural Networks

We develop a theoretical framework for how hierarchical representation of features in input data emerges from progressive renormalization and sparse-coding done using convolutional layers. At each level new degrees of freedom appear, which are low-lying energy states, separated by a gap from a pool of high energy states. This separation defines a natural way for sparse encoding of training data. Repeating this renormalization procedure results in a hierarchical representation of the data. We show that trained filter in popular image processing deep neural nets are consistent with such a hierarchical representation.