This paper proposes a novel adaptive filtering scheme to remove impulse noise from images. The scheme replaces the corrupted test pixel with the median value of non-corrupted neighboring pixels selected from a window dynamically. If the number of non-corrupted pixels in the selected window is not sufficient, a window of next higher size is chosen. Thus window size is automatically adapted based on the density of noise in the image as well as the density of corruption local to a window. As a result window size may vary pixel to pixel while filtering. The scheme is simple to implement and do not require multiple iterations. The efficacy of the proposed scheme is evaluated with respect to subjective as well as objective parameters on standard images on various noise densities. Comparative analysis reveals that the proposed scheme has improved performance over other schemes, preferably in high density impulse noise cases. Further, the computational overhead is also less as compared its competent scheme.