The 36-year-old had presided over Spurs for 18 months only for the club to confirm the "termination of his services" on their official website on Monday morning, a day after their 5-0 home defeat to Liverpool.
The humiliation by Liverpool, which followed a 6-0 drubbing by Manchester City last month, appears to have been the final straw for the Spurs chairman Daniel Levy, who delivered the coup de grace in person at the club's training ground in Enfield.
Sugar also accused AVB of being "unprofessional" and showing "immaturity" while his questioning of his appointment was echoed by Jamie Redknapp, son of former boss Harry.
Sugar told ITV1: "The fact is he's gone.
"With the greatest respect to the chairman I think it was the wrong move to appoint him. I think the guy showed he was unprofessional in spending that amount of money."
Redknapp also had little sympathy for the Portuguese, questioning how he came to be installed at White Hart Lane after lasting just eight months at Chelsea.
He said on Sky Sports News: "After the job he did at Chelsea he was incredibly lucky to get that job in the first place.
"He had a nightmare at Chelsea and then they went and won the Champions League without him, but he finds himself as the manager of Tottenham with a squad of Modric, Bale and players like that.
"Last year he did well with the previous manager's team, but now he's had his chance to do it, it hasn't worked out for him.
"When you get beat 5-0 at home it's a bit of a tell-tale sign. It's not something Tottenham fans are accustomed to and I feel sorry to a certain extent for the fans.
"Having seen the team that was built - you look at Gareth Bale on the wing - and you've ended up with players who aren't in the same league."
Sugar had become something of a nemesis for Villas-Boas in recent weeks, who had accused the Amstrad boss of driving an "agenda" against him.
Sugar was unrepentant and instead went on to criticise Villas-Boas' approach while admitting he was surprised by the speed of Monday's announcement.
He told Sky Sports News: "I can only imagine there must have been some words which were not acceptable by the manager spoken between him and the chairman and that is why it abruptly ended.
"Young fellow AVB - he has his own ideas and I don't think he wants to listen to anyone else other than himself.
"He tried to implement those ideas and the thing is they didn't work.
"The writing was on the wall as far as I was concerned when he went out and spent like a kid in a sweet shop, I mean you just can't do that.
"It doesnt stack up, you just can't go out and buy seven brand new players or whatever it was, stick them in and hope for them to play.
"It showed immaturity."
Of who he would like as replacement, Sugar mused: "Someone asked me what would be your wild dream? I said if Alex Ferguson would decide to come out of retirement, that would be the dream.
"I think it is highly unlikely but it is a dream."