Transmission control protocol (TCP) is the most common protocol found in recent networks to maintain reliable communication. The most popular transport protocol in use today is TCP that cannot fully utilize the ability of the network because of the constraints of its conservative congestion control algorithm and favors reliability over timeliness. Despite congestion is the most frequent cause of lost packets, transmission defects can also result in packet loss. In response to packet loss, end-to-end congestion control mechanism in TCP limits the amount of remarkable, unacknowledged data segments that are permitted in the network. To overcome the drawback, Optimized Extreme Gradient Boosting Algorithm is proposed to predict the congestion. Initially, the data is collected and given to data preprocessing to improve the data quality. Min-Max normalization is used to normalize the data in the particular range and KNN-based missing value imputation is used to replace the missing values in the original data in the preprocessing section. Then the preprocessed data is fed into the Optimized Extreme Gradient Boosting Algorithm to predict the congestion. Remora optimization is used in the designed model for optimally selecting the learning rate to minimize the error for enhancing the prediction accuracy in machine learning. For validating the proposed model, the performance metrics attained by the proposed and existing model are compared. Accuracy, precision, recall and error values for the proposed methods are 96%, 97%, 96% and 3% values are obtained. Thus, the proposed optimized extreme gradient boosting with the remora algorithm for congestion prediction in the transport layer method is the best method than the existing algorithm.