The paper develops an efficient people surveillance system capable of tracking multiple people on different terrains. Recorded video on rough terrains is affected by jitters resulting into significant error between the desired and captured video flow. Video stabilization is achieved by calculating the motion and compensational parameters using the LSE analytical solution to minimize the error between present and desired output video captured from an autonomous robot’s camera moving on a rough terrain used for surveillance of unidentified people. This is the first paper to the best of our knowledge which makes use of this method to design mobile wireless robot for human surveillance applications. As the method used is fast then conventional methods, making the proposed system a highly efficient surveillance system as compared to previous systems. The superiority of the method used is demonstrated using different evaluation parameters like RMCD, variability and reliability. The system can be used for surveillance of people under different environmental conditions.