Stochastic Gradient Descent
Stochastic Gradient Descent is an optimization algorithm that updates the parameters of a model iteratively based on the gradient of the loss function with respect to a randomly selected subset of data.
Category: Deep Learning Foundations
Time-dependent Loss Weighting
A training technique where the loss function is adjusted based on the time variable to stabilize model training.
Normalized MSE-based Layer Importance
A metric for assessing the importance of layers in a neural network based on normalized mean squared error.
Unstructured Pruning
A method of reducing the size of neural networks by removing individual weights based on certain criteria, rather than entire neurons or layers.
Nonlinear Recurrent Language Models
A type of language model that utilizes recurrent neural networks with nonlinear activation functions to process sequences of data.