"Mathematics of Neural Networks" is an excellent set of lecture notes for anyone who wants to study modern neural networks from a mathematical perspective. It covers supervised learning, artificial neurons and activation functions (ReLU, sigmoid, tanh, swish, softmax), shallow and deep neural networks, stochastic gradient descent, weight initialization, vanishing and exploding gradients, convolutional neural networks, automatic differentiation, backpropagation, adaptive optimization algorithms (Adagrad, RMSProp, Adam), and concludes with modern geometric concepts such as manifolds, Lie groups, equivariance, group convolutions, and rotation-translation equivariant CNNs. I recommend keeping it in your bookmarks as a useful reference whenever you need to explore the mathematics behind neural networks. arxiv.org/abs/2403.04807