Colorectal cancers are the third-largest kind of cancer in the world. However, detecting and removing precursor polyps with adenomatous cells using optical colonoscopy images helps to prevent this type of cancer. Moreover, hyperplastic polyps are benign cancers; adenomatous polyps are more likely to grow into cancerous tumors. Therefore, the detection and segmentation of polyps provide further histological evaluation. However, the main challenge is the extensive range of infected polyp features inside the colon and the lack of contrast between normal and infected areas. To solve these issues, the proposed novel Customized ResNet50 with DeepLabV3Plus Network (CRDL-PNet) model provided a scheme for segmenting polyps from colonoscopy images. The customized ResNet50 extracted features from polyp colonoscopy images. Furthermore, Atrous Spatial Pyramid Pooling (ASPP) is used to handle scale variation during training and improve feature selection maps in an upsampling layer. Additionally, the Gateaux Derivatives (GD) approach is used to segment boundary boxes of polyp regions. The proposed method has been evaluated on four datasets, namely the Kvasir-SEG, ETIS-PolypLaribDB, CVC-ClinicDB, and CVC-ColonDB datasets, for segmenting and detecting polyps. The simulation results have been examined by evaluation metrics, such as accuracy, Intersection-Over-Union (IOU), mean IOU, precision, recall, F1-score, dice, Jaccard, and Mean Process Time per Frame (MPTF) for proper validation. The proposed scheme outperforms the existing State-Of-The-Arts (SOTA) model on the same polyp datasets.