The field of agriculture is one of the most profitable fields for a country. The produce of this industry i.e., fruits and vegetables, is tremendous and thus the quality insurance of these products are of utmost importance. Evaluation of fruits can be done manually but due to inconsistent results and huge time consumption, it is necessary to have the automated systems to perform the quality tests. In this study, computer vision has been used to build an architecture that is competent to detect whether the fruit is rotten or fresh. VGG16 CNN (Convolutional Neural Network) model is employed to extract the features from the images of Apples, Bananas, Guava and Oranges. With the help of extracted features, the classification is performed through Decision Tree, Support Vector Machines (SVM), and Logistic regression models. Support Vector Machine performed the best classification with an accuracy of 99%.