Monkey chants were heard at the San Siro during the second leg of a Europa League last-16 tie during which an inflatable banana was seen in the crowd.
Emmanuel Adebayor seemed to be the particular target of the racist abuse but had the last laugh, prodding home in extra-time with the game finishing 4-1 on the night and putting Tottenham through on away goals after the tie finished 4-4 on aggregate.
However, progress to the quarter-finals was overshadowed by appalling abuse from the terraces, with attention now turning to what action governing body UEFA will take.
"It's a very sensitive situation," Spurs manager Villas-Boas said of the chanting. "UEFA set out to act on that situation.
"It was very, very easy to hear the chanting so I am sure that UEFA will act on it.
"It's difficult for Inter Milan because it is something that in some way has happened before.
"It doesn't mar the game but it is something that should have been avoided."
Inter have been reprimanded for their supporters' behaviour only recently.
Last month they were fined £43,000 by Italian league chiefs after their fans abused AC Milan duo Mario Balotelli and Sulley Muntari, with reports of inflatable bananas accompanying monkey chants.
Inter were also fined £12,900 for racist chants about former Manchester City striker Balotelli earlier that month in a 3-1 win against Chievo.
Tottenham have yet to confirm whether an official complaint will be made, but UEFA, who declined to comment on the matter, might be expected to act having done so against another Italian club earlier in the Europa League.
Lazio were fined for racist chanting in their Group J tie at White Hart Lane in September and were reprimanded again for racist behaviour in November's return game.
Subsequent trouble during their last-32 match against Borussia Monchengladbach, including more racist chanting, led to a two-game order for Lazio to play behind closed doors.
The fact Tottenham had to contend with more racist abuse understandably overshadowed what was a difficult night on the field.
Having romped to a 3-0 win the first leg, Spurs were outclassed at the San Siro as Antonio Cassano, Rodrigo Palacio and a William Gallas own goal took the game to extra-time.
Inter scored again through Ricardo Alvarez but not until after Adebayor scored the vital away goal.
Villas-Boas said: "We set out today [Thursday] to try and get a goal away that would put us in an extremely confident position.
"I think today [Thursday] what happened to us was a bit like what happened to Inter in London.
"They are not as bad as their performance in London and we're not as a bad as the performance we showed today [Thursday].
"There are still lots of positives in an evening like this because the desire to reach the last eight of the competition was always there from the players, even in extra-time, which is an excellent sign for us.
"I think the organisation wasn't there because their mind wasn't there. Every goal for Inter Milan was a boost of motivation for them and a knife in the back for us.
"We had a lot of problems in our organisation, which they did extremely well [to exploit] and is why we had a very, very hard game."
While Tottenham can look forward to Friday's quarter-final draw, Inter were left to consider what could have been.
"Both my players and I believe the best way to respond is not with words but through your performances," Inter manager Andrea Stramaccioni said.
"We have taken a huge step forward in terms of our play. We believed right from the first minute, we're still there, we've got ability, we're all together and we want to finish the season on a high.
"Tonight's [Thursday] performance was our best display in terms of the football we produced.
"The lads approached the match brilliantly, shrugging off any fear. It's just a shame about the result and the fact we conceded by switching off."
FOX Sports Asia will be covering the Champions League quarter-final draw LIVE at 7pm HKT/4:30pm IST