Real have fallen 16 points behind La Liga leaders Barcelona.
Substitute Roque Santa Cruz struck twice in three second-half minutes to help fourth-placed Malaga heap the pressure on Madrid coach Jose Mourinho, who created a surprise by omitting captain Iker Casillas from his starting line-up.
Madrid dominated the first half but were unable to turn their possession into goals and were made to pay as Isco put Malaga ahead in the 48th minute.
An own goal from Sergio Sanchez drew Madrid level midway through the second half but Santa Cruz, who had been on the pitch less than 10 minutes, stunned the visitors by putting Malaga 3-1 ahead with a double strike in the 73rd and 76th minutes.
Karim Benzema pulled another goal back with eight minutes remaining Madrid could not find an equaliser as they slumped to a fourth league defeat of the season - twice as many as they suffered in winning the title last season.