The growing quality and safety concern about fresh agricultural produce among consumers have led to the development of non-destructive quality assessment and testing techniques of fruits and vegetables. Humans judge the quality of fruits based on sensory attributes like taste, aroma etc. The shape, size, color, presence of defects which are external to fruits also influence the degree of consumer acceptability of produce. The traditional time consuming, manual fruit quality inspection is replaced by automated, fast, consistent, non-destructive techniques using computer vision in combination with learning algorithms. But the lack of benchmark datasets for agricultural produce has made an objective comparison of the proposed methods difficult. Hence, the proposed work aims to build a medium scale dataset for mango fruits of “Alphonso” cultivar with three classes: chilling injury, defective and non-defective. The reliability of the proposed dataset consisting of 2279 color images of mango fruits with 736 samples in chilling injury class, 632 samples in defective class and 911 samples in non-defective class, was established using a novel approach of developing a predictive model based on discriminant function analysis (DFA) which assigns group membership to each sample of the dataset. Extensive benchmarking analysis is established on the validated dataset using statistical and deep learning algorithms like support vector machine (SVM) and convolutional neural network (CNN), respectively. SVM achieved significant disease classification accuracy of 95% and 91.52% accuracy was achieved by custom CNN. The results of the proposed work indicate that the proposed color image dataset of mango fruits can be used as a benchmark dataset by other researchers for objective comparison in quality evaluation of mango fruits.