The world number three, playing his first match since March following a six-week break, survived a tough battle against rising star Milos Raonic, winning 4-6 7-5 7-6 (7/4) at the Caja Magica.
Serve dominated early on but the breakthrough came late in the set. From 3-4, Raonic won the last 12 points of the set to leave the crowd stunned.
The Canadian's big hitting continued to trouble Federer in the second set, Raonic coming up with some superb shots from the baseline to hurt Federer.
However, he was unable to take any of four break-point chances and instead cracked himself in the 12th game, Federer levelling the match with a deft drop shot.
Raonic refused to go away though and he was quickly threatening the Federer serve in the final set. Three times the Swiss staved off break points to hold and with Raonic serving well - he slammed down 21 aces in total - a tie-break ensued.
In it, a netted backhand from the Canadian gave Federer the early advantage but a forehand miss at 5-3 saw the mini-break handed back.
However, Raonic followed that with a wild forehand, bringing up two match points, and Federer clinched victory with a superb winner on the return of serve.
Earlier Rafael Nadal had also secured a place in the third round with a straight-sets win over Nikolay Davydenko.
The second-seeded Spaniard broke his opponent four times en route to a 6-2 6-2 victory in one hour and 20 minutes on the blue clay which is proving controversial in the Spanish captial.
Following Novak Djokovic's condemnation of the new surface, it was no surprise to see Nadal adding criticism.
In an interview for Spanish TV, he said: "The court is a difficult court, it's very slippery and it makes supporting movements and getting back to defend very tough.
"But the only thing we can do now is turn the page and we are not going to get the red courts back tomorrow so we have to adapt to the blue courts and the conditions as well as possible and hope for a change next year."
Nadal was joined in the last 16 by compatriot and fifth seed David Ferrer, who overcame Radek Stepanek 7-6 (7/4) 6-2.
Watch: Federer pushed to limit by Raonic...More videos
With the courts playing faster than in previous years, 10th seed Juan Martin Del Potro is fancied to go well and he began his campaign with a 6-4 7-6 (7/1) victory over Mikhail Youzhny.
Jo-Wilfried Tsonga (4), Nicolas Almagro (11) and Richard Gasquet (14) were also straight-sets winners but Janko Tipsarevic (7) and Fernando Verdasco (15) both needed three sets to progress against qualifiers.
Tipsarevic was pushed hard by Argentine Federico Del Bonis before winning 7-6 (7/5) 6-7 (6/8) 6-3, while Verdasco came from behind to beat Colombia's Alejandro Falla 6-7 (4/7) 6-4 6-4.
Gasquet is next up for Federer, while Verdasco will face Nadal on Thursday.
Gael Monfils, seeded 12th, progressed after Russian opponent Igor Andreev retired hurt when trailing 6-3 1-0.
However, eighth seed John Isner fell at the second hurdle, beaten 7-6 (7/4) 7-6 (7/3) by Croatia's Marin Cilic.
Isner would not blame the surface for his defeat though.
"I always felt that this tournament even with the red clay was the most slippery I have ever played on," he said.
"It's tough for a big guy like me but my opponent was also a big guy and like I said he was just better and deserved to win and I did not deserve to win.
"I just didn't play particularly well today which has nothing to do with the courts."
Swiss Stanislas Wawrinka needed just an hour to overcome Jurgen Melzer 6-1 6-4, and will now face top seed and defending champion Djokovic.
Watch: Ferrer and Cilic through in Madrid...More videos