Present work aims to investigate the unique microstructural development during severe plastic deformation (SPD) of aluminum and its alloys under cryogenic temperature and during the post cryo-SPD annealing treatment. The influence of alloy chemistry and stacking fault energy (SFE) on recovery, recrystallization and grain growth phenomenon has been examined by various characterization techniques. Commercially pure Al alloy, non-heat treatable Al-4.5%Mg alloy and heat treatable Al-4.5%Cu alloy exhibiting varied SFE have been selected for this investigation. First, a significant variation in the cryo-SPDed microstructure has been observed between pure aluminum and its alloys. It has been attributed to the ‘polyslip’ deformation mechanism prevalent only in pure aluminum due to its very high SFE. Second, due to this inherent difference in their cryo-SPDed microstructures and influence of alloying elements, both the Al alloys show a substantial difference in their microstructural response during annealing in comparison with pure Al. For pure aluminum, grain rotation and coalescence is the dominating restoration mechanism. However, for aluminum alloys, rapid vacancy-assisted recovery, polygonization and recrystallization of equiaxed, high-angled, sub-micrometric grains have been observed. The grain growth mechanism of these recrystallized grains is heavily influenced by solute drag and particle drag effects. For non-heat treatable Al-4.5%Mg alloy, abnormal grain growth has been observed. In heat treatable Al-4.5%Cu alloy, nano-particles have precipitated in the grain boundaries, thereby retarding the grain growth process and inducing high microstructural thermal stability. © 2017 Elsevier B.V.