Machine learning is now being used for applications ranging from healthcare to network security. However, machine learning models can be easily fooled into making mistakes using adversarial machine learning attacks. In this article, we focus on the evasion attacks against Network Intrusion Detection System (NIDS) and specifically on designing novel adversarial attacks and defenses using adversarial training. We propose white box attacks against intrusion detection systems. Under these attacks, the detection accuracy of model suffered significantly. Also, we propose a defense mechanism against adversarial attacks using adversarial sample augmented training. The biggest advantage of proposed defense is that it doesn’t require any modification to deep neural network architecture or any additional hyperparameter tuning. The gain in accuracy using very small adversarial samples for training deep neural network was however found to be significant.