Oliver Fink was shown the red card for the visitors in the 78th minute before Noveski struck five minutes from time to earn the hosts all three points.
Dusseldorf, who had not even conceded a goal on their travels this season, not only lost that record but also their invincibility after six games without defeat.
It was the first time these two sides had met in the German top flight and it took a bit of time for them to get to know each other.
The early stages were rather pedestrian with the first chance a wayward header by Nicolai Muller for the hosts in the seventh minute.
Ken Ilso, who was preferred in the Dusseldorf attack to Andrei Voronin, tried his luck from long range, following the example set by Dani Schahin, but neither were able to threaten an opening goal.
Mainz started to gain control of proceedings early in the second half, albeit without creating any clear-cut opportunities.
Oliver Fink was shown a second yellow card in the 78th minute, forcing the visitors to be even more defensive for the remaining 10 minutes.
Mainz chipped away and finally made the breakthrough five minutes from time when Noveski raced in at the near post to head in Andreas Ivanschitz's corner from close range and secure the points, leaving 10-man Dusseldorf no chance to recover.