Danny Gabbidon's maiden top-flight goal gave the hosts the early lead their play deserved, with half-time substitute Steven Fletcher drawing Sunderland level just after the hour.
But Palace, who lost their opening two league matches against Tottenham and Stoke, remained resolute and after Sunderland skipper John O'Shea was dismissed for tripping Jason Puncheon inside the area, Gayle stepped up to score his first goal for the Eagles.
The win was wrapped up in stoppage time, with substitute Stuart O'Keefe securing three points for Ian Holloway's side with a smart finish.
Holloway rotated his complete starting XI for the second time in a week, reverting back to the side that started in last weekend's 2-1 defeat at Stoke following a shake-up for the midweek Capital One Cup exit to Bristol City.
That meant a home debut for Croydon-born Puncheon, making his Selhurst Park bow after joining on a season-long loan from Southampton.
But it was fellow summer signing Jose Campana who created the early breakthrough for the hosts.
The former Sevilla man whipped in a ninth-minute corner that just evaded the head of Marouane Chamakh at the near post, the ball then bounced off Sunderland's Ondrej Celustka before an unknowing touch from Gabbidon steered it past Kieran Westwood.
Ji Dong-won should have levelled for the Black Cats 10 minutes later but he was in two minds whether to throw a head or a boot at Celustka's tempting cross, and in the end he did neither, missing the ball completely when free in front of Julian Speroni's goal.
Local boy Puncheon was soon involved as he picked the ball up on the halfway line before driving at the Sunderland defence and firing a low shot just wide of Westwood's left-hand post.
As Sunderland chased an equaliser, the hosts hit them on the counter-attack and Puncheon slipped in skipper Mile Jedinak, who came within inches of adding a second for the Eagles as he bent an effort just wide of the post.
Di Canio introduced Fletcher from the bench at half-time, with the ineffective Ji making way, as the Scotland international made his first appearance since suffering an ankle injury in March.
The Italian is still looking for a first win of the season after a summer recruitment drive which has seen 12 new faces enter the Stadium of Light, with a loan deal for Swansea midfielder Ki Sung-yueng announced as the latest signing just two hours before kick-off.
Palace were still getting plenty of joy out of running at the Sunderland defence and pushing them back towards their own goal as both Jedinak and Gayle created chances in the opening 15 minutes of the second half.
But it was Sunderland who scored the next goal of the game, with the returning Fletcher levelling matters on 64 minutes as he expertly headed Jack Colback's cross past Speroni.
The Scotland international was making his first appearance in five months after suffering an ankle injury towards the end of last season, but instantly gave his side more of an attacking edge in the game.
It was a defensive error that would prove to be the visitors' undoing as O'Shea lost control of a loose ball and, as Puncheon nipped in, the Black Cats' captain caught him and conceded a penalty with referee Lee Probert showing the former Manchester United man a red card.
Gayle, who was playing League Two football with Dagenham 12 months ago, coolly slotted home the resulting spot-kick to put Palace back in the driving seat.
Substitute Jonny Williams came close to adding a third for the hosts but his well-struck effort was pushed away by Westwood with six minutes remaining, with the Republic of Ireland goalkeeper again forced into action late on as he tipped Kagisho Dikhacoi's long-range piledriver behind for a corner.
But Westwood could do nothing about O'Keefe's injury-time effort as the Palace substitute curled a sweet shot into the back of the net to seal the win for the home side.