Advances in medical imaging technologies have led to the generation of large databases with high-resolution image volumes. To retrieve images with pathology similar to the one under examination, we propose a content-based image retrieval framework (CBIR) for medical image retrieval using deep Convolutional Neural Network (CNN). We present retrieval results for medical images using a pre-Trained neural network, ResNet-18. A multi-modality dataset that contains twenty-Three classes and four modalities including (Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Mammogram (MG), and Positron Emission Tomograph (PET)) are used for demonstrating our method. We obtain an average classification accuracy of 92% and the mean average precision of 0.90 for retrieval. The proposed method can assist in clinical diagnosis and training radiologist. © 2019 SPIE.