Rayudu was ably supported by Stuart Binny (77) as the pair shared an Indian record 160-run partnership for the sixth wicket to help the visitors recover from 87-5.
Captain for the day Ajinkya Rahane opened the batting for India, scoring 34 off 49 balls, but Murali Vijay (1), Manoj Tiwary (2), Ketan Jadhav (0) and Robin Uthappa (5) all fell cheaply.
Zimbabwe were ahead of the run rate in the early part of their innings but were slowed down by some good bowling from Axar Patel, who returned figures of 2-41.
Zimbabwe captain Elton Chigumbura hit an unbeaten century (104no) to leave the hosts needing 10 from the final over to seal victory.
But Bhuvneshwar Kumar produced a brilliant final over, conceding just five runs from six balls as India clinched a nail-biting win.
The second ODI of the three-match series will be on Sunday at the same venue, with the teams also playing two Twenty20 games starting next Friday.