Houllier left Villa at the end of last season to be replaced by Alex McLeish having missed the back end of the campaign following chest pains.
Agbonlahor endured a difficult season last term, starting only 19 games, while a number of other senior players also fell foul of Houllier's regime.
The 24-year-old, who has come through the ranks at Villa, admits it was so bad under Houllier and his assistant Gary McAllister that he would have left had they remained in charge at Villa Park this season
"At one stage I did think about it [leaving]," Agbonlahor told the Daily Telegraph.
"I thought if this is how it is going to stay and the manager is going to stay here, with this formation, then I would have to leave. It hurt more as a Villa fan and local boy. Luckily the manager left and McAllister left as well.
"It killed me a lot. It did not help me with England. The manager had his formations and he was not going to change that. Gerard was not much of a talker. It was more McAllister, but he was not very good to be honest. I did not really get on with him.
"I don't think a lot of the players were happy with the way he set up things. There was too much rotation. You look at who was left out. [Stephen] Warnock, [Stephen] Ireland and [Richard] Dunne. It was a crazy season."
Agbonlahor is enjoying life under McLeish and he believes the atmosphere around the club has changed since the arrival of the former Birmingham boss.
"It feels happier and more organised. Everyone is enjoying the training more and there is a better atmosphere around the place," added Agbonlahor.
Agbonlahor has started the season in good form and he is hoping to break back into the England set-up in time for Euro 2012 after 18 months in the international wilderness.
"2012 is still a target," continued Agbonlahor. "Anything can happen with injuries and people can lose their form. The new manager has given me a chance.
"England have really good players but if I could get that chance, then I would be excited by it.
"Hopefully I'll steer clear of injuries and play well for Villa."