Training deep networks

Deep networks are trained on losses with long valleys, many minima and saddle points. This topic speeds descent with momentum and adaptive step sizes, looks at the shape of the loss, and restrains a network that memorises its training data.