Now that we have covered the neural network basics and what a deep network is, let’s go through the layers of the deep network.
Dense layer / Fully connected layer
Let’s start with the simplest one. This is basically the layer we talked about in the neural network. The neurons of a dense layer are connected to every neuron of its preceding layer. Something like the image of standard neural nets below.
Advantages
- Changes the dimension of the vector i.e. number of neurons changes at every layer changing the dimension of the input vector at every stage.
- Adds non-linear property (through activation function). The activation function allows neural networks to identify non-linear relations between the independent and dependent variables.
- Produces output using every node in the previous layer (learn from every feature).
- Used for rotation, scaling, and translation.
Dropout layer
Dropout [1] randomly sets the outgoing edges of hidden units to 0 at each update of the training phase. Each unit is retained with a fixed probability p independent of other units. For input layer p = 1. Note in the dropout layer below, that’s not how the network looks throughout training. After each epoch different edges are connected and dropped.
While training without a dropout layer, the weights of neurons are tuned for specific features providing some specialization. By using dropout, neurons are randomly dropped out of the network during training, the other neurons will have to step in and handle the representation required to make predictions for the missing values.
Advantages
- It prevents overfitting.
- It prevents the co-adaption of features [2]. Co-adaption of a feature means a feature detector is helpful only in the context of several other features.
- Combines different networks efficiently.