A career Armoured Corps officer, General Sharma was commissioned into the Indian Army in June 1950. In a career spanning forty years, he served in various prestigious posts, including as the chief of the Eastern Command, as ADC to the President of India, as the Commandant of the College of Combat (now Army War College) in Mhow, as the Director General of Military Operations at Army Headquarters, and ultimately as the Chief of the Army Staff.