In this paper, we present computer vision based technique to detect surface defects of citrus fruits. The method begins with background removal using k-means clustering technique. Mean shift segmentation is used for fruit region segmentation. The candidate defects are detected using threshold based segmentation. In this stage, it is very difficult to differentiate stem-end from actual defects due to similarity in appearance. Therefore, we proposed a novel technique to differentiate stem-end from actual defects based on the shape features. We conducted experiments on our citrus data set captured in controlled environment. The experiment results demonstrate that our technique outperforms the existing techniques.