Artificial Neural Networks with Interacting Output Units

The simple perceptron has just one output unit whose response is considered the truth value of a binary predicate. Even if many output units act in parallel on a common body of input and hidden layers, they are distinguished only by their specific teacher signals but are otherwise independent of each other. In this chapter, we turn to models in which the activity and learning of one output neuron directly affects the activity and learning of all other outputs. The simplest case is the matrix model of associative learning which stores input–output associations in a distributed or “holographic” way; it is based on covariance matrices and the outer product learning rule. A biologically more realistic form of learning is competitive learningCompetitive learning in which the eigenvectors of the covariance matrix of the input data set play an important role. It allows the self-organization of efficient data representations including decorrelation and the formation of continuous feature maps. Sparse coding is a related scheme that optimizes representations not by redundancy reduction but by the identification of meaningful sub-patterns. Finally, we discuss the formation of continuous field attractors or activity “bumps” moving on continuous feature maps. Such attractors are used as models of working memory.