Chico Flores was sent off after bringing down substitute striker Glenn Murray late on, but the offence appeared to take place just outside the box.
Yet referee Mike Dean pointed to the spot and Murray converted in front of the travelling Eagles fans to leave a bitter taste for Garry Monk's Swans.
Jonathan de Guzman had put the home side in front midway through a dominant first half, but tactical changes from Palace boss Tony Pulis pulled the visitors back into the contest.
Swansea have now gone six games without a win and appeared to fade after a hectic league and cup schedule.
Angel Rangel, Leon Britton and Nathan Dyer all freshened up the starting side that performed admirably in their 3-1 Europa League defeat in Napoli on Thursday.
As has become the norm at home, Swansea enjoyed the majority of possession in the first half, hogging 83 per cent of the ball before the break.
Playing with a compact five across the midfield, Pulis' men seemed content to allow the home side to pass it around on the slick, rain-soaked surface. A tame glancing header from Marouane Chamakh was their only chance of note in a limp first-half performance.
Boos and whistles of derision accompanied every touch from Tom Ince, who reportedly turned down a loan move to Swansea in January and said before the match that the Welsh team would be unable to handle Palace's play. Home captain Ashley Williams made his presence felt early on when he gave Ince a two-handed shove in the back.
Monk was forced to make an earlier-than-planned change to his side after 22 minutes, bringing on Spaniard Jose Canas to replace in-form countryman Pablo Hernandez. His parting gift for the fans was a left-footed strike, dragged narrowly wide.
Pulis also had to make an unwanted substitution when Chamakh went off injured, making way for Cameron Jerome.
Some crisp interplay from Swansea carved out the opening goal after 25 minutes. Williams played the ball into Wilfried Bony, whose lay-off to midfielder Leon Britton drew the Palace defence out of position. When De Guzman latched onto Britton's inch-perfect through-ball, all he had to do was side-foot it past Julian Speroni.
De Guzman had a chance for a second goal when he chased a lofted ball over the top from right-back Rangel, but 10 yards from goal he could not get a toe to it.
Bony forced a good diving save from Speroni when he met a first-time Rangel cross, after a curling ball from former Palace junior Wayne Routledge.
A cheer rang out around the Liberty Stadium at the start of the second half when it was announced the fans' villain of the piece, Ince, was being replaced by striker Murray as Pulis switched to a more attacking 4-4-2 formation.
The rejig got the Eagles back in the game though, and they were finally able to give their fans some territory and pressure to cheer.
With only five Premier League goals away from home all season before Sunday's assignment and the lowest overall tally of any side, applying themselves away from home has become a familiar problem.
Leroy Lita made only his second appearance for Swansea, coming on for the weary-looking Bony with half an hour to go.
A hooked shot from Murray was easily gobbled up by Vorm and a header from Jerome was comfortably saved a minute later.
The game turned after 81 minutes when a mix-up led to a penalty for Palace and a red card for centre-back Flores.
Vorm and his defence failed to clear a ball over the top, allowing Murray to toe-poke it past the goalkeeper 20 yards out from goal.
With an open goal beckoning he was brought down just outside the box, but referee Dean, some 30 yards behind play, pointed to the spot and sent Flores off. Murray fired the penalty into the top-right corner.
He was on the end of the visitors' next attack too, when his goal-bound shot was blocked by replacement Jordi Amat.
In injury time, Palace's replacement midfielder Jerome Thomas tried to persuade Dean to give another spot-kick but was booked for diving.