The paper presents an image-based paddy plant variety recognition system to recognize 15 different paddy plant varieties using 18 color-related agro-morphological characteristics. The k-means color clustering method has been used to segment the target regions in the paddy plant images. The RGB, HSI and YCbCr color models have been employed to construct color feature vectors from the segmented images and the feature vectors are reduced using Principal Component Analysis (PCA) technique. The reduced color feature vectors are used as input to back propagation neural network (BPNN) and support vector machine (SVM). The set of six combined agro-morphological characteristics recorded during maturity growth stage has given the highest average paddy plant variety recognition accuracies of 91.20% and 86.33% using the BPNN and SVM classifiers respectively. The work finds application in developing a tool for assisting botanists, Rice scientists, plant breeders, and certification agencies.