All terms
Training
Cosine Schedule
Decaying the learning rate along half a cosine curve from its peak toward a minimum.
Definition
A cosine schedule decays the learning rate from its peak value following one half of a cosine curve, reaching a small minimum by the end of training. The smooth, gradual decay avoids the abrupt drops of step schedules and tends to yield better final quality. It is usually paired with a short linear warmup at the start, and variants with warm restarts periodically reset the rate.