Tottenham fans vented their fury at their side on numerous occasions on Saturday as their team put in an insipid display that allowed Wigan to claim only their second win at White Hart Lane.
The home support were also furious at Villas-Boas for his decision to withdraw in-form Defoe and replace him with Emmanuel Adebayor in the second half.
Defoe himself seemed upset at the move too, sluggishly sauntering off the pitch before grabbing his jacket and walking down the tunnel.
Harry Redknapp often talked of his dislike when Rafael van der Vaart failed to take his place on the bench when he was substituted last season and Villas-Boas agrees with his predecessor's mantra on the issue.
"For sure it is not something that is very pleasing to see," the Tottenham manager said.
"The most important thing for the players to realise is that we do things for the benefit of the team.
"We understand individual frustrations but we, as a team, are more important.
"The game wasn't going well for Defoe in our thought process. Maybe he could think the opposite but we don't take him off to punish him. We took him off to improve the team."
Adebayor was expected to be Villas-Boas' first-choice striker this season following his permanent move from Manchester City.
The Togolese, who was Tottenham's top scorer while on loan last season, is yet to start a Barclays Premier League game for the club this term, but it appears Villas-Boas may recall him for Sunday's clash against former team Manchester City.
There have been reports that Adebayor is unhappy at his lack of game time this season, but Villas-Boas insists there is no problem with the player's attitude, citing recent injuries as the reason why he has not started games.
"We're quite happy with his attitude," Villas-Boas said.
"He had an injury which took him out for six games and then another injury took him out for two games. We have the injury records if you want to look at them.
"From the 11 that we've played, I think he's been out for eight of them and he wasn't here at the start of the season.
"In the games when he was available he showed the quality striker that he is.
"We have no doubts whatsoever. We have him fit now and hopefully ready for the next upcoming games. We know his ability."
Wigan boss Roberto Martinez has now racked up three wins from his opening 10 matches.
The Spaniard usually has to wait until Spring before his team put in the kind of performance they displayed at Spurs yesterday.
He has urged his players to maintain the momentum they have now built so they do not have to go through the customary rigmarole of having to wait until the very last moment to avoid relegation.
"We have to make it easier for ourselves (this season)," Martinez said.
"It is very dangerous to do what we normally do. You don't want to do that again.
"And the other aspect is that as a football club we want to progress, we want to get a little higher up the table.
"We need to get points and we need to get in a better position, a healthier position."
Matchwinner Ben Watson, a boyhood Arsenal fan, could not resist a dig at the club's north London rivals afterwards.
"I am an Arsenal fan so for me it was a special day. It's always special scoring here," the former Crystal Palace midfielder said with a smile.
"Who do I think will finish higher between Arsenal and Spurs? Arsenal."