The 2-1 victory in Riga lifted Latvia above their Baltic rivals and into fourth in Group G of the European World Cup qualifying section.
Both goals arrived in the first half and Pahars' men held on after Deivydas Matulevicius gave Lithuania hope just before the interval.
Latvia turned to Pahars in the summer after collecting just four points from their opening six games.
After a friendly draw against Estonia last month, his competitive reign got off to a fine start as Bulvitis opened the scoring after 20 minutes.
Zjuzins doubled the lead in the 42nd minute but they were made to work after Matulevicius replied on the stroke of half-time.
It was only Latvia's second win of the campaign after a 2-0 defeat of bottom side Liechtenstein 11 months ago.