To differentiate the loss function in a Neural Network, there are four options

  1. Manual differentiation: It is labor intensive and often it is hard to calculate the closed form solution especially for complex function
  2. Symbolic differentiation: Like manual, it is also hard for complex function
  3. Numerical differentiation: Can handle complex function but may cause numerical issues
  4. Automatic differentiation: In Deep Learning, most libraries use automatic differentiation