Show Tag: sparse-coding

Select Other Tags

Sparse coding is a compromise between the simplicity of grandmother-cell-type codes and the efficiency and ability for generalization of combinatorial (dense) codes.

The trigger feature hypothesis in principle postulates combinatorial codes (or even sparse coding).

Weber presents a continuous Hopfield-like RNN as a model of complex cells in V1. This model receives input from a sparse coding generative Helmholtz machine, described earlier as a model of simple cells in V1, and which produces topography by coactivating neighbors in its "sleep phase". The complex cell model with its horizontal connections is trained to predict the simple cells' activations, while input images undergo small random shifts. The trained network features realistic centre-surround weight profiles (in position- and orientation-space) and sharpened orientation tuning curves.

Zhao et al.'s model develops both disparity selection and vergence control in an effort to minimize reconstruction error.

It uses a form of sparse-coding to learn to approximate its input and a variation of the actor-critic learning algorithm called natural actor critic reinforcement learning algorithm (NACREL).

The teaching signal to the NACREL algorithm is the reconstruction error of the model after the action produced by it.

Since combinatorial and sparse codes are known to be efficient, it would make sense if the states of generative models would usually be encoded in them.

That would preclude redundant population codes—except if we use the notion of redundancy in our idea of efficiency.

The complexity of features (or combinations of features) neurons in the ventral pathway react to increases to object level. Most neurons react to feature combinations which are below object level, however.

By optimizing sparseness (or coding efficiency) of functions for representing natural images, one can arrive at tuning functions similar to those found in in simple cells. They are

  • spatially localized
  • oriented
  • band-pass filters with different spatial frequencies.

LGN cells respond whitened---ie. efficiently---to natural images, but they respond non-white to white noise, eg. They are thus well-adapted to natural images from the efficient coding point of view.

One hypothesis about early visual processing is that it tries to preserve (and enhance) as much information about the visual stimuli (with as little effort) as possible. Findings about efficiency in visual processing seem to validate this hypothesis.

A complete theory of early visual processing would need to address more aspects than coding efficiency, optimal representation and cleanup. Tasks and implementation would have to be taken into account.

Redundancy reduction, predictive coding, efficient coding, sparse coding, and energy minimization are related hypotheses with similar predictions. All these theories are reasonably successful in explaining biological phenomena.