Fetal neurosonography is potentially used to examine the fetal brain by scanning the trans-thalamic (TT), trans-cerebellum (TC), and trans-ventricular (TV) planes. Cross-sectional analysis of these planes is useful to assess the brain anatomy, development, and abnormality for intervention and treatment plans even at the postnatal stage. To minimize the errors and processing time involved in the traditional manual subjective approach, the automatic classification of fetal brain planes is crucial. In this study, a deep learning-based method for automatically categorizing fetal brain planes from ultrasound images is proposed and evaluated. Firstly, the brain region has been segmented from the fetal brain ultrasound images using U-Net to prepare an efficient data set for the classifier model. Then, an ensemble convolutional neural network (CNN) model including well-known Inception V3, ResNet50-V2, and DenseNet-201 models with max voting is designed to classify the segmented brain planes. 2019 fetal brain ultrasound images from a widely used publicly accessible experts-annotated dataset are used to evaluate the performance of the proposed framework. The obtained results analysis shows that using the segmented images as input improves the performance of the classifier from its raw form. The gradient class activation mapping (Grad-CAM) based inspection shows noteworthy localization capability of the last convolution layer. The ensemble model has also outperformed its individual model’s performance. The suggested categorization framework is satisfactory compared to related recent works, with a testing accuracy of 97.68%. The proposed framework for fetal brain plane classification is expected to be useful for clinical applications.