nn-components.md

Neural Networks Components

This is a table that lists components of a neural networks with examples.

Category	Examples
Data
Data Preprocessing
Model initialisation	Weight initialisation Glorot Gaussian Truncated Gaussian Bias initalisation All zeroes Slightly negative
Input
Input layer
Hidden Layers: (Usually linear) op	Linear: WX + b Convolutional layer RNN layer
Hidden Layers: Activation	ReLU Tanh Sigmoid
Connections	Skip connections
Regularisation between layers	Dropout
Data preprocessing between layers	Batch normalisation
Output layer: convert hidden layer output to predictions	Linear Softmax (classification)
Output
Loss function: data loss	MSE
Loss function: regularisation loss	L1 norm L2 norm
Optimiser	SGD: Stochastic Gradient Descent SGD with learning rate tuning: Adam RMSprop Adagrad
Parameter update method	For SGD: Vanilla update Momentum update Nesterov momentum update
Gradient calculation	Backpropagation Synthetic gradients