Dzemaili saw the first of his two hat-trick goals at the Stadio Olimpico cancelled out by Paulo Barreto and then Jonathas from the penalty spot.
Marek Hamsik had missed his first-half spot-kick and it looked like Napoli would live to regret it as Riccardo Meggiorini gave Torino the lead midway through the second half.
But the introduction of Cavani from the bench inspired Dzemaili to a third goal before the Uruguay striker notched twice in the last six minutes to keep his side nine points behind leaders Juventus.
Former Torino midfielder Dzemaili silenced the Olimpico in the 10th minute when he arrowed a 25-yard effort into the top left-hand corner of Jean-Francois Gillet's goal.
Torino had looked the brighter up to that point, and they continued to threaten, getting a deserved equaliser after 30 minutes.
Napoli failed to clear their lines from a corner and Kamil Glik managed to thread the ball through to Barreto, whose shot went in despite Paolo Cannavaro's attempts to make a clearance off the goal line.
Napoli were given the chance to regain the lead again in the 39th minute when Christian Maggio was brought down inside the penalty area, but Hamsik's spot kick was pushed away by Gillet.
Napoli did get back in front early in the second half with a similar goal to their first with Hamsik pulling the ball back for Dzemaili, who chose accuracy over power this time to thread a low shot into the bottom left-hand corner.
Cavani was sent on by Mazzarri, who clearly wanted a third goal to see off Torino, but the move backfired initially as the Uruguayan handled the ball inside his own penalty area to give away a penalty in the 75th minute
Substitute Jonathas stepped up to bring Torino level again.
The home side took the lead for the first time three minutes later when Miguel Britos was caught in possession on the edge of his own penalty area by Meggiorini, who skipped past Antonio Rosati before tucking the ball in from a tight angle.
Two minutes later, Dzemaili completed his hat-trick to bring Napoli level with another shot from outside the penalty area with the outside of his right boot.
Cavani then made up for his error in giving away the penalty by curling in an unstoppable free-kick from 25 yards and heading home Napoli's fifth in the last six minutes.