Villa looked to be cruising to victory as goals from Benteke - a £7million summer signing from Genk - and Gabriel Agbonlahor earned them a 2-0 half-time lead.
But the npower League One side, who had already overcome Stoke in this season's competition, fought back through two goals from substitute Miles Storey before Benteke had the final say.
It was his third goal in two games and Villa boss Paul Lambert will hope his side can build on this success after their worst start to a league campaign in 43 years.
Stephen Ireland also impressed in the play-maker role and set up the opening two goals.
Lambert made five changes from the side which drew at home to Norwich at the weekend but there was again no place in the starting line-up for £18million club record signing Darren Bent.
Benteke was again preferred as the main striker in a 4-2-3-1 formation and the Belgian had the first effort on goal, a downward header which was comfortably saved by Wes Foderingham.
Barry Bannan curled a 25 yard drive wide but there was little to choose between the sides in the opening quarter of the game.
Villa were indebted to a goal-line clearance from full-back Matt Lowton in keeping out a shot on the turn from Paul Benson after 26 minutes.
It proved to be crucial as Villa took control with two goals in the space of 10 minutes before half-time.
Ireland created the opening goal when he got to the by-line and his cross was headed home in powerful fashion by Benteke for his second goal in two games.
Villa were forced into a change when central defender Chris Herd limped out of the action and was replaced by Enda Stevens.
But the visitors were starting to look more menacing and a superb strike from Agbonlahor doubled their lead.
Once again Ireland was the creator with his perfectly weighted pass although Agbonlahor still had much to do before cutting inside and unleashing a powerful 20 yard shot which flew past Foderingham.
The Swindon keeper produced a fine reflex save on the stroke of half-time to turn over Andreas Weimann's volley.
Swindon needed to make inroads into Villa's advantage at the start of the second period and a dangerous low cross from Simon Ferry just eluded Benson.
Gary Roberts headed over from a Collins centre and was then guilty of squandering a golden chance to reduce the arrears.
Matt Ritchie delivered a pinpoint ball across the box into the path of the unmarked Roberts at the far post but he headed against the outside of a post.
Villa started to reassert themselves, with Ireland continuing to impress, and Foderingham blocked a shot from Benteke with his feet.
But Swindon turned the tie on its head with two goals in two minutes.
With 11 minutes remaining, substitute Storey converted a centre.
Storey drew the home side level when he back-heeled a low ball into the box from Andy Williams past Shay Given.
But Benteke broke Swindon's hearts with seconds remaining when he hammered home a cross from Eric Lichaj.