Lecture 2

• Most common in practical applications.
• With 1 input layer, 1 output layer and x hidden layers.
• With more that 1 layer of hidden units we get a deep neural network.

• Very difficult to train.
• More biologically realistic.
• Would be more appropriate to predict a stock price.

• Connections between units are symmetrical (same weight in both directions)
• John Hopfield realized they are much easier to analyze than recurrent networks.
• More restricted in what they can do.
• Symmetrically connected net without hidden units are called Hopfield nets.

• We select features manually
• Then we learn the weights of the features.
• We compare the hypothesis with a threshold to decide the final target.
• Perceptrons use binary threshold neurons

• For each test case:
  • If the output is correct, leave the weights as is.
  • If 0 is predicted instead of 1, we add the input vector to the weight vector.
  • If 1 is predicted instead of 0, we substract the input vector from the weight vector.

• Weight-space: space with one dimension per weight.
• A point in the weight space represents a particular settings of all the weights.
• If we eliminate the threshold, each training case can be represented as a hyperplane though the origin.

⇒ The weights must lie on one side of this hyper-plane to get the answer correct.

• The training case hyperplane is perpendicular to the input vector of that training case. The right side of the plane to get the right answer is given by the direction of the input vector and the target answer (0 or 1).

• The average of 2 solutions is also a solution in this weight space. ⇒ The problem is convex.

• We assume there exist a feasible vector of weights in the weight space (eg. vector to get the right answer for all inputs).
• After a finite number of mistakes the weight vector must lie in the feasible region if this region exists.

• If we have the correct features then we can do almost anything.
• Now we define a data-space where each point is a data point.
• The weight vector defines a plane in this data space.