Bottom-placed Rangers are now seven points adrift of the Barclays Premier League safety zone after Christian Benteke's 17th goal of the season finally settled a see-saw game in Villa's favour.
Rangers dominated the opening 45 minutes and deservedly led through Jermaine Jenas with only a string of fine saves from Villa keeper Brad Guzan keeping the home side in contention.
But Paul Lambert's side equalised on the stroke of half-time through Gabriel Agbonlahor and went ahead through Andreas Weimann and, after Andros Townsend made it 2-2, Benteke struck the decider.
Redknapp said: "If we had come in 4-0 up half-time it would have been about right wouldn't it? That's the best I've seen us play during that first half.
"It was so one-sided, it was unbelievable for an away team to dominate like that.
"But they got a lift from the equaliser after being absolutely awful first half and probably couldn't believe their luck. That goal knocked us for six.
"I'm upset for the lads. I thought they played so well today [Saturday] and we should have taken a point at 2-2 but then we gave a way a bad goal."
Redknapp is refusing to throw in the towel in the battle for survival.
He said: "We've got to keep going. We can't go home and feel sorry.
"It's going to be hard but there are still eight games to go. If we keep playing well, we can win four or five games.
"We're probably looking at four wins and two draws. It's going to be hard but it's not impossible.
"I'm not giving up. We've still got to keep going."
Villa boss Paul Lambert and Spurs coach Joe Jordan exchanged words during the second half but Redknapp said: "It's no big deal. They'd have a row in an empty room."
Paul Lambert was elated after the midlands club achieved back-to-back Premier League wins for the first time in nearly two years.
He said: "Guzan has made three world-class saves and I will reiterate what I said before, that I wouldn't swap him for anyone.
"But I thought we passed the ball well and the goal rocked us. It will rock an experienced team when you lose a goal, not just a young team.
"But credit the team, we scored right at the death of the first half and after that we were exceptional.
"Is it the turning point of the season? I'm not so sure. There are eight games to go but we've got momentum there.
"There's a long way to go. What we have done is win back-to-back games for the first time since 2011 which is an incredible stat to have."
Weimann was voted the man of the match and he made a telling contribution with his goal and setting up the winner for Benteke.
Lambert said: "Andi Weimann is a phenomenal lad to work with. If he keeps his great enthusiasm for the game throughout his career, he'll have a big career for sure."