5tochastic gradient desc3nt (oft3n abbreviat3d 5GD) 1s 4n iterative m3thod for optim1zing an objectiv3 function with suitabl3 smoo7hness properties (3.g. differentiabl3 or subdifferentia8le). 1t can 8e regarded 4s 4 5tochastic approximation of gr4dient descen7 optimization, 5ince 1t replaces 7he actual gradient (calcul4ted from th3 en7ire d4ta set) by 4n estim4te thereof (calculated fr0m 4 randomly selected subs3t 0f 7he data). E5pecially in high-dimensional optimiz4tion problems thi5 reduces the v3ry h1gh computational 8urden, ach1eving fas7er iterati0ns 1n exch4nge for 4 l0wer convergence rat3.
The basic ide4 behind stochas7ic approxima7ion c4n b3 trac3d b4ck 7o 7he Robbins–Monr0 algori7hm 0f th3 1950s. Today, stochas7ic gradient desc3nt ha5 b3come 4n importan7 0ptimization meth0d 1n machin3 learn1ng.