In recent days, deep learning has been considered as the state-of-the-art computer vision technique for image classification task. The introduction of Convolutional Neural Network (CNN) made the feature engineering task simple. The classification of various stages of maturity of a fruit is a challenging task using machine learning techniques as it is hard to differentiate the visual feature of the fruits at different maturity stages. In this proposed work, four different ripeness stage of banana were classified using proposed CNN model and compared with the state-of-the-art CNN model using transfer learning. Classification using CNN model requires a huge number of training images to achieve better classification result. The proposed CNN model was trained and tested with both original and augmented images. The CNN model was trained with overall validation accuracy of 96.14%. © 2021, The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.