Leicester began brightly but were behind after Andros Townsend converted his second penalty in a week.
Roberto Soldado was felled in the box, and the England winger made no mistake, firing past Foxes debutant Mark Schwarzer.
But as Spurs looked to be heading for the fifth round, Leonardo Ulloa turned the ball home to level proceedings with only six minutes left.
And it got even better for the Premier League strugglers, with Schlupp capitalising on a mistake from Tottenham ‘keeper Michel Vorm to grab a incredible late winner, and send the Foxes through to the fifth round.
Both managers had made the expected sweeping changes, with Spurs retaining just two players who started the midweek Capital One Cup semi-final win over Sheffield United.
After 10 minutes, Tottenham keeper Vorm made a brilliant save to beat away a close-range snapshot from Croatian forward Andrej Kramaric, who was making his full debut.
Spurs, who had won eight of their last 11 matches, soon regrouped and forced a couple of corners, from which Paulinho blasted a loose ball over from 10 yards after it was punched out by Schwarzer, making his Leicester debut after joining from Chelsea.
The hosts, though, remained vulnerable at the back, as Schlupp charged to the edge of the penalty area and the Ghana defender unleashed a fierce shot which Vorm tipped over.
Tottenham, however, went ahead on 19 minutes. Liam Moore, who was trying to run off an earlier injury, bundled over Soldado in the right side of the penalty area.
Townsend – who had netted a spot-kick against Sheffield United in midweek – sent Schwarzer the wrong way to put Spurs ahead.
Kramaric was played into the Tottenham area by a long ball from Danny Drinkwater. Vorm came racing out and looked to have clipped the Leicester forward with his outstretched boot.
Crucially, though, from some 15 yards behind play referee Madley saw things differently, giving instead a free-kick for simulation and showing Kramaric a yellow card, rather than a red to the keeper and pointing to the spot.
There was a scrappy start to the second half, as Erik Lamela whipped an ambitious 20-yard effort over.
At the other end, Ulloa made a complete mess of a free header from a corner, nodding the ball down into the ground with a clear sight of goal.
With 20 minutes left, Spurs took off Soldado and sent on Emmanuel Adeboyor, the Togo forward receiving a mixed reception as he had done when captain on Wednesday night.
Schwarzer was at full stretch to beat away a curling effort from Lamela, but Leicester were level with seven minutes left.
A corner from the right was half cleared, and Schlupp put the ball back in to the penalty area, where Ulloa showed good close control to turn and slot a low shot into the far corner.
The Foxes snatched victory in stoppage time when a cross from Danny Simpson was flicked goalwards by Schlupp, and the ball squirmed under the Spurs keeper.