The Argentinian striker headed in the only goal of the game on 61 minutes and Valencia were dealt a further blow when Antonio Barragan was sent off a quarter of an hour later for picking up a second booking.
Rayo remain 19th in the Primera Division table but have closed the gap on 17th-placed Almeria to three points, while Valencia suffered their first defeat in six matches and drop down to 9th.
The first half saw plenty of chances for both sides, although neither had much luck hitting the target. Paco Alcacer struck the bar early on for the visitors, while Larrivey struck the post but should have hit the net.
The Argentinian was in the thick of the action for Rayo, later shooting just beyond the post from outside of the area and then heading over the bar shortly before the break.
But the best chance for the hosts in the opening 45 minutes fell to Alberto Bueno, who shot over the bar after a poor clearance from Valencia goalkeeper Diego Alves.
Early in the second half Alves had to punch a free-kick from Alejandro Galvez away for a corner and Bueno skied a cut-back from Iago Falque.
Rayo had come out stronger than their opponents after the break and deserved to go in front just after the hour mark.
Saul Niguez pursued a loose ball and pulled it back for Razvan Rat to send a hanging cross towards the far post, and Larrivey rose above the Valencia defenders to head past Alves and into the net.
Valencia had two penalty appeals turned down in the space of a few minutes, the first for a supposed handball by Anaitz Arbilla and the second for the same infraction by Galvez, but referee Carlos Clos Gomez saw no problem with either incident.
But the official did take issue with a foul by Barragan on Falque and showed the full-back a second yellow card, making the visitors' quest for an equaliser even more difficult.
Their final opportunities fell to Sofiane Feghouli, who was unable to keep his header under the crossbar, and Vinicius Araujo, who sent an awkward shot over the bar in injury time.