The Nerazzurri trailed 3-2 before Ricky Alvarez was shown his second yellow card for a deliberate handball 20 minutes from time.
"There was a series of terrible mistakes that changed the game," Mazzarri said to Gazzetta dello Sport.
"When we were in control and 3-2 down, Alvarez was harshly sent off.
"The first card was harsh too (a late challenge on Dries Mertens), as he tripped up.
"I saw far worse challenges that were worthy of a card.
"I think with 11 men against 11 we could have got back on level terms.
"Perhaps Tagliavento is unlucky with us, as we've had two defeats all season and both were with Tagliavento making many mistakes.
"The other was a loss against Roma with a non-existent penalty. We're unlucky with him, it keeps happening to us.
"Maybe Tagliavento is just out of shape."
Sunday's result was Inter's second defeat of the campaign and dropped the Nerazzurri to fifth in Serie A, two points behind Fiorentina, who climbed to fourth.
Despite the reverse against his former club, Mazzarri was satisfied with his team's performance in Naples.
"I complimented my players for their performance and I am proud of them," he said.
"We are growing and now must work on our style of football, which today was exceptional.
"We made life difficult for Napoli and forced them to only go on the counter-attack in their own stadium.
"I told my players that if we always play like this, we will not lose many games."
Inter end the year at home to AC Milan in Sunday's city derby.