It looked like Udinese coach Francesco Guidolin was going to celebrate his first ever victory against Milan after Antonio Di Natale gave the home team the lead in the 17th minute.
The Bianconeri continued to dominate the game against an off-colour Milan, but were unable to build on their lead and were punished with two late goals from Maxi Lopez and Stephan El Shaarawy.
The win lifted Milan two points above Juventus at the top of Serie A, but the Turin side have two games in hand, while Udinese are six points off the pace in fourth.
Mauricio Isla had Udinese's first chance of the game in the seventh minute when he shot over from 20 yards.
The hosts soon took a deserved lead through Di Natale, who scored his 17th goal of the season.
He exchanged passed with Gelson Fernandes and his shot took a decisive deflection off Thiago Silva to beat Marco Amelia.
Udinese continued to pose the greater threat on the counter-attack with Isla setting up Di Natale, whose volley went wide of the far post in the 35th minute.
Milan were finding Udinese difficult to deal with and the home side had numerous chances to extend their lead. Pablo Armero took too long in the 62nd minute after being put through by Di Natale and Djamel Mesbah stole the ball off him before he could shoot.
Another good break from Udinese saw Armero set up Maurizio Domizzi, who tried to toe-poke the ball beyond Amelia, who pushed it wide with his fingertips.
From the resulting corner, Danilo's shot was tipped over the bar by Amelia as Udinese laid siege to the Milan goal.
But they were made to pay for not taking their chances in the 77th minute as Milan levelled with a rare counter-attack.
El Shaarawy's shot was pushed out by Samir Handanovic, who appeared to be in two minds whether to hold the ball or push it away, and Lopez was on hand to net his first goal for the Rossoneri.
There was more confusion in the Udinese box a minute later and El Shaarawy got in as Medhi Benatia tried to shepherd the ball back to Handanovic.
The ball broke loose and Robinho's shot was saved on the line by Danilo.
Milan grabbed a shock winner in the 85th minute with Lopez this time setting up El Shaarawy, who slid the ball inside the near post from seven yards.