What Is Forward And Backward Propagation?

• Forward Propagation
  • I will show the working in a stepwise fashion.
• Backward Propagation
• INTERVIEW TOPICS

Forward Propagation

• It is the process of passing input from input layer to output layer through hidden layer. Following steps fall under forward propagation:
  • Weight initialization
  • Apply activation Function
  • Adding Dropout layer
  • Take output.

  

  Photo Credit–>Rpubs

  

  Photo Credit–>Sattyajit Patnaik

  I will show the working in a stepwise fashion.

  

  Photo Credit–>Data Flair

  • STEP 1: The input layer takes the input data.
  • STEP 2: The hidden layer takes the input from the input layer, perform a complex operation to find a pattern in a data and gives the output to the output layer. Here in this layer, summation of the product of input from the input layer and the weights of the hidden layer is performed and activation function is applied to it.
  • 

  

  

  • STEP 3: The output layer takes the output from the hidden layer and gives the output. Output expression is given as:

  

Backward Propagation

• Backpropagation is very much a reason why neural network training works. It is a method of optimizing the weights of a neural network based on the cost function obtained in the previous epoch (iteration). Proper tuning of these weights allows us to reduce error and make the model reliable by increasing its accuracy. Following steps fall under backward propagation:
  • Calculate the error
    • Error is calculated as:
      • Error = (Predicted Output - Actual Output)^2
  • Update the weights

    • There are different optimizer to update a weights and here weights are updated using gradient decent optimizer which is done as:

      

      • new_weight = the new weight of Xi
      • old_weight = the old weight of Xi
      • learning_rate = the learning rate of the network
      • dE/dW is the partial derivative of the error for each of the Xs. It is the rate of change of the error to the change in weight.

    • Let’s see how back propagation happens by taking an eaxmple:

      

      • So it is done by using Chain Rule of Derivatives.
      • After this again weight is updates as before.
      • Similarly all the weights are updated like this if they are making more error.
  • Apply activation Function
  • Take output.
  • Beautiful illustration of backpropagation:

  

  gif-credit

INTERVIEW TOPICS

### 1. What is the difference between Forward Propagation and Backward Propagation?

• Forward Propagation is the process of taking the input and passing it through the network to get the output. Each hidden layer accepts the input data, processes it as per the activation function, and passes it to the successive layer.
• Back propagation is the practice of fine-tuning the weights of the neural network based on the error rate obtained from the previous epoch. Proper tuning of the weights ensures low error rates, making the model more reliable.

### 2. Why do we need backpropagation?

• It is used to calculate the derivative/gradient of the loss function with respect to all the weights in the network.
• Backpropagation minimizes the cost/loss function by updating the weights with the gradient descent/optimization method.
• Produces learning by modifying the weights of the connected nodes during the process of training.

### 3. What are the advantages of backpropagation?

• It’s fast, simple, and easy to program.
• It has no parameters to tune apart from the number of inputs.
• It is a flexible method as it does not require prior knowledge about the network.
• It does not need any special mention of the features of the function to be learned.

### 4. What are the disadvantages of backpropagation?

• The matrix-based approach is used instead of a mini-batch, hence taking longer and be less efficient in comparison