Convolutional neural networks achieve state of the art results for a variety of tasks. However, this improved performance comes at the cost of performing convolutional operations throughout the entire image. Resizing of images to manageable levels is one of the often used techniques so as to reduce this computational overhead. On medical images, lesions are represented by a minuscule proportion of pixels and resizing may lead to loss of information. Recurrent attention mechanism based network aid in reducing computational overhead while performing convolutional operations on high resolution images. The proposed technique was tested on 2 distinct classification task viz; classification of brain tumors from Magnetic Resonance images predicting the severity of diabetic macular edema from fundus images. For the former task (n=300), the technique achieved state of the art accuracy of 97%. While on the latter (n=89), the proposed model achieved an accuracy of 93.37% © 2019 IEEE.