Face detection and recognition has always been one of the research interests to researchers in the field of the biometric identification of individuals. Problems such as environmental lighting, different skin color, complex background, etc affect on the detection and recognition of individuals. This paper proposes a method to enhance the performance of face detection and recognition systems. Our method, basically consists of two main parts: firstly, we detect faces and then recognize the detected faces. In the detection step, we use the skin color segmentation combined with AdaBoost algorithm, which is fast and also more accurate compared to the other known methods. Also, we use a series of morphological operators to improve the face detection performance. Recognition part consists of three steps: dimension reduction using Principal Component Analysis (PCA), feature selection using Linear Discriminant Analysis (LDA), and k-Nearest Neighbor (K-NN) or Support Vector Machine (SVM) based classification. Combination of PCA and LDA is used for improving the capability of LDA when a few samples of images are available. We test the system on the face databases. Experimental results show that the system is robust enough to detect faces in different lighting conditions, scales, poses, and skin colors from various races. Also, the system is able to recognize face with less misclassification compared to the previous methods.