Stochastic approximation algorithms have been one of the main focus areas of research on solution methods for stochastic optimization problems. The Robbins-Monro algorithm [17] is a basic stochastic approximation scheme that has been found to be applicable in a variety of settings that involve finding the roots of a function under noisy observations. We first review in this chapter the Robbins-Monro algorithm and its convergence. In cases where one is interested in optimizing the steady-state system performance, i.e., the objective is a long-run average cost function, multi-timescale variants of the Robbins-Monro algorithm have been found useful. We also review multi-timescale stochastic approximation in this chapter since many of the schemes presented in the later chapters shall involve such algorithms. © 2013, Springer-Verlag London.