Pittsburgh had won their first two games and led after the first period of their opening home game thanks to Evgeni Malkin's goal, but Van Riemsdyk scored twice in the second stanza and helped create Mikhail Grabovski's clincher in the third.
Elsewhere, there were first wins of the season for the New York Rangers, Vancouver Canucks and Phoenix Coyotes.
Steve Sullivan opened his Phoenix goal account in style with a hat-trick, the eighth of his career, as they routed the Columbus Blue Jackets 5-1. Oliver Ekman-Larsson scored their other two goals.
Marian Gaborik also scored a hat-trick, the last an unassisted winner in overtime, as the Rangers battled to a 4-3 win over the Boston Bruins.
Vancouver needed a shootout to win their all-Canadian match-up against the Calgary Flames, Zack Kassian making the winning shot after the game had finished 2-2 with all the goals in the second period.