Face recognition is one of the conventional problems in computer vision. Its recognition rate falls steeply when the images are captured in an unconstrained environment. One of the fundamental issues that creep into unconstrained environment capturing is that of the face pose variation. Due to face pose variation, occlusion of crucial features takes place. Occlusion may lead to information loss in the face descriptor which describes the face appearance. In this paper, we propose learning-based descriptor that combines horizontal, vertical and diagonal pattern of blocks generated from the convolution of face image with Gabor filter bank. To use only discriminative features, Probabilistic Linear Discriminant Analysis (PLDA) is used. The fusion of non-uniform texture based descriptor along with the PLDA approach aids in retaining enough of discriminative information to overcome the information loss occurring during feature occlusion. Since HVDGBP face descriptor utilizes the fundamental concept of Linear Binary Pattern (LBP) henceforth it helped in meeting low processing demands and ease of computing characteristic required for good face descriptors. Comprehensive comparative performance analysis of the robustness of the proposed face descriptor to withstand pose variations is presented. UMIST and AT&T Database is used for experimental analysis.