Albrighton, Gabriel Agbonlahor and Ciaran Clark gave Villa a comfortable lead before some poor defending allowed Scott McGleish to score in the 90th minute.
Villa then conceded a penalty in stoppage time when Stephen Warnock handled the ball, but Brad Guzan comfortably saved McGleish's spot kick.
"We knew we had to come here and be professional and get the job done," man of the match Albrighton told ESPN.
"We didn't mess about and we did well.
"An early goal always helps and we're pleased to get that and built on that. Ciaran and Gaby both got great individual goals.
"I suppose it was difficult because it was a difficult pitch to play on but we passed the ball around nicely and I thought we dealt with it well."
Clark, whose goal was his first of the season, is hopeful to push on after a solid performance in midfield.
"It's always tough to come to places like this, we got the early goal and pushed on from there," Clark told ESPN.
"It's always hard to play teams like this, especially in the cup.
"I got [a goal] last year and it's nice to get my first goal of the season this year. I'm pleased to get my first goal and get off to a good start.
"I'm thankful to get my chance this season. Hopefully we can keep going and put on good performances."
Villa boss Alex McLeish was satisfied with the way his team managed the game.
"It was a very pleasing performance," McLeish told ESPN. "We knew it could be a potential banana skin but the players dealt with it well and handled the extra pressure and I thought they had a great mentality.
"We got a bit sloppy at the death and a wee bit complacent and we were punished. I thought 3-2 would make it look like a real close encounter."
McLeish also reserved praise for Clark following an impressive performance.
He said: "The kid has got a touch of class. He can pass the ball, he did a really good job for us as a central midfielder so it's a great boost to have someone like Ciaran playing so well."
Bristol Rovers caretaker manager Shaun North, who assumed temporary control of the Pirates following former manager Paul Buckle's departure on Tuesday, was frustrated with the goals given away by his side.
"The boys gave 100%," North said. "We knew we weren't going to have a lot of the ball but we tried to keep some shape.
"We were a little disappointed with the goals we conceded. The boys got a bit tired at the end. It would have been nice to make it 3-2 as people don't remember the game, they remember the scoreline.
"We had to make sure we were in the game after 15 to 20 minutes, and we were, and then we conceded a goal.
"They came at us and stepped it up a bit but towards the end we became a bit stronger."