The game looked to be heading for overtime before Ward slotted the puck between the legs of Rangers keeper Henrik Lundqvist with 1.3 seconds to put them 1-0 up in the best-of-seven series.
Washington took the lead through Alex Ovechkin in the Eastern Conference second-round encounter at Madison Square Gardens. And although Jesper Fast levelled the scores 15 minutes into the third period, Ward was on hand to score the winner.
The Anaheim Ducks were comfortable winners in their Western Conference semifinal opener, beating the Calgary Flames 6-1 .
Ryan Getzlaf registered a goal and three assists, and Matt Beleskey, Patrick Maroon and Emerson Etem also scored for the top-seeded Ducks, while Flames got one back through Sam Bennett in the third period.
Game two will take place on Sunday night back at Honda Center, where the Flames have not won since the 2006 play-offs.