Bernd Nehrig gave the home team the lead from the penalty spot in the 10th minute, but Thomas Kleine was sent off two minutes later.
Oscar Wendt levelled soon after, but Oscar Prib gave Furth the lead again just before half time.
Martin Stranzl and Patrick Hermann turned the game around in the 51st and 57th minutes.
Milorad Pekovic also saw red in stoppage time with Thorben Marx completing the scoring for the Foals from the penalty spot.
Lucien Favre's side picked up the points to move up to eighth, leaving Furth second from bottom of the table, just one point ahead of their Bavarian rivals Augsburg.
Buoyed by their first goal and point of the season at home, and a three-game unbeaten record, the Bavarians decided the time had come for them to win a game in front of their own fans.
They started brightly, abandoning the defensive tactics which have characterised their previous home matches, and they were rewarded with the opening goal in the 10th minute.
Gerald Asamoah was felled inside the area by Wendt and Nehrig blasted in the resulting spot-kick.
However, the smiles on Furth's faces were soon wiped away when Kleine received a red card for a foul as the last man on Patrick Herrmann on the edge of the penalty area.
Juan Arango failed to add insult to injury, though, when his free-kick was palmed away by Max Grun, but the Bavarians were up against it with 10 men for a good 80 minutes.
Gladbach broke them down in the 22nd minute to draw level with the livewire Herrmann sending in a cross for Wendt to head in.
Furth had no other choice but to revert back to their defensive tactics, yet they still managed to break and go ahead for the second time two minutes before half time.
Sercan Sararer robbed Arango of possession inside his own half and sent Asamoah free down the right wing.
The former Germany international's cross was collected by Prib on his chest, and then finished expertly past Marc-Andre ter Stegen.
Two goals in six early second-half minutes then turned the game on its head.
First, Stranzl headed in a corner which Mike Hanke had flicked on for him at the near post in the 51st minute, before Herrmann got the goal his game had deserved when he headed in a finely-weighted cross from Lukas Rupp.
Furth refused to lie down and almost levelled immediately when Ter Stegen was at full stretch to push Stefan Furstner's effort behind for a corner.
But it was always going to be difficult for the 10 men to survive and they also lost Pekovic as he gave away a penalty in stoppage time.
Marx's penalty locked up the points for Gladbach and left Furth in serious danger at the bottom, and still without a win at home.