Neural network foundations
Syllabus
Detailed content
Tensors and computational graphs
Tensors generalize scalars, vectors, and matrices to higher dimensions. Neural network frameworks record operations in a computational graph so gradients can be calculated. Track tensor shape, device, and dtype at every layer. Shape mismatches and unintended broadcasting are frequent sources of silent bugs.
Perceptrons and activation functions
A neuron combines weighted inputs and a bias, then applies a nonlinear activation. Stacking linear layers without nonlinearities still produces a linear function, so activations give a network expressive power. ReLU is common for hidden layers; output activations depend on whether the task is regression, binary, or multiclass classification.
Loss, gradients, and backpropagation
A loss function measures prediction error. Backpropagation applies the chain rule through the computational graph to compute how each parameter affects that loss. Gradient descent updates parameters in the direction that reduces the objective. Check gradients and loss behavior on a tiny example before scaling to large data.