Marco Caligiuri put the hosts in front as early as the fourth minute, but Marco Reus equalised eight minutes later.
Robert Lewandowski earned Dortmund all three points as they took full advantage of a draw between Schalke and Eintracht Frankfurt to leap above both of them into second, nine points adrift of league leaders Bayern Munich.
Mainz stunned the defending champions by taking the lead in the fourth minute with Elkin Soto sending the ball into the path of Caligiuri, who blasted a volley into the net after the ball had sat up for him.
Dortmund levelled in the 12th minute, however, with Reus picking his spot in the far corner from outside the penalty area.
The visitors gained in confidence from the swift equaliser and Reus and Mario Gotze started getting their act together to create a string of chances.
But it was Andreas Ivanschitz who almost put Mainz in front again against the run of play in the 33rd minute, but his shot was too weak to cause Dortmund goalkeeper Roman Weidenfeller any problems.
And he was made to pay for squandering the chance as the visitors took the lead two minutes before the interval.
Gotze sent Lewandowski clean through on goal and the Poland international wonderfully lifted the ball over the onrushing Christian Wetklo from outside the penalty area.
Mainz did not give up and their coach Thomas Tuchel made attacking changes to his side with Marcel Risse coming on.
He had the best chance to equalise in the 54th minute, but after that miss, Dortmund shut up shop to hold on to their narrow advantage.