The Tottenham defender, currently on loan at Sunderland, said he had been subjected to monkey taunts long before the violent scenes which marred the end of the Euro 2013 play-off.
Sports minister Hugh Robertson sent a letter to UEFA president Michel Platini on Wednesday urging tough sanctions from the governing body following the "disgraceful scenes" of racism in Serbia.
Rose was sent off after the final whistle for kicking a ball in anger in response to provocation as scuffles broke out involving Serbian supporters, players and other officials.
Rose told Sky Sports News: "I remember getting slapped twice and then I got ushered away.
"That's when I kicked the ball - and the referee sent me off. I don't understand, the game had finished by then but he still sent me off for kicking the ball.
"As I went off again there was monkey chanting, but the monkey chanting started long before I got sent off.
"After 60 minutes my head wasn't really on the game. They have to be banned. I don't understand how else they can learn from it, they have to be banned."
Robertson has given his full support to the Football Association, who reported a number of incidents of racism to UEFA.
The minister said: "The scenes at the end of the game last night [Tuesday] were disgraceful.
"I have written to UEFA president Michel Platini, in support of the FA, urging them to investigate immediately.
"Racism in any form is unacceptable and must be stamped out. We would expect tough sanctions from UEFA on anyone found guilty of racist abuse."
England assistant coach Steve Wigley was caught up in the trouble and was manhandled as he made his way to the tunnel after the game.
Rose said the issues were a culmination of problems which had been brewing throughout the evening at the Mladost Stadium.
The 22-year-old added: "It started when we went out for the warm-up.
"They started the monkey chanting straight away. I asked the lads if they could hear it and they said they could hear it.
"Halfway through the warm-up I went to 'Wigs', the assistant manager, and told him what was happening.
"He said I had to try my best to get through it and they would deal with it straight away after the game.
"In the first half I went down to get the ball for a throw-in and the fans started again with the monkey chants, but the first half was nowhere near as bad as the second half.
"In the second half I had two stones hit me on the head when I went to get the ball for a throw-in. Every time I touched the ball there was monkey chanting again.
"After 60 minutes my mind wasn't really on the game after that. I was just so angry and it was just so hard to concentrate.
"I could have cost the lads the game because I made a few mistakes through not concentrating.
"Then obviously we scored. After 90 minutes' worth of abuse I just expressed my emotions as soon as we scored.
"Then the next thing I know all the Serbia players have run over and were all surrounding me, pushing me and a brawl broke out."
Connor Wickham's injury-time strike clinched a 2-0 aggregate success.
Sunderland backed their loan player, saying they were "shocked" by what happened in Serbia.
Sunderland's chief executive Margaret Byrne said: "As a club we strongly believe that the power of football should be used to promote inclusion and celebrate diversity and there is no place for any form of racism within the game of football and society as a whole.
"The scenes in Serbia last night [Tuesday] shocked everyone and Sunderland Football Club wholeheartedly backs the Football Association's stance on the matter."
Professional Footballers' Association chairman Clarke Carlisle also believes a "significant" ban is the most adequate punishment for the "utterly deplorable" scenes.
The 33-year-old watched the chaos and violence unfold in the ESPN studio alongside former England captain Paul Ince, who called for Serbia to be excluded from the next five international tournaments.
Carlisle told BBC Radio Five Live: "You should not have to experience those scenes in this day and age. It was utterly deplorable behaviour.
"It's quite farcical. A guy is racially abused and then he gets punished for containing his reaction to that.
"We saw how the whole debacle progressed into punching, kicking... We saw photos of headbutts going on in the staff.
"But Danny Rose, for kicking a ball away, ends up getting himself sent off. This is just wrong.
"The message has to come from the top down that this behaviour will not be tolerated.
"It's immaterial where it is, who the perpetrators are. What's material is that the governing bodies lay down extremely harsh sanctions so that they deter countries and people from acting in this way.
"I believe Serbia should be banned because it's a repeat offence.
"Banning them for a start, from any tournament, would be progress but I think if it's significant - if it's a couple of tournaments - then that would cause that nation to address the issue that has deprived them of international competition."
A match between the two countries at the 2007 Under-21 European Championship in Holland was marred when England defender Nedum Onuoha was racially abused. Serbia were fined after that incident.
Reading striker Jason Roberts accused UEFA of not dealing adequately with past racism issues and backed players to walk off the pitch if subjected to racist abuse.
The 34-year-old told BBC Radio Five: "I was absolutely fuming at what I saw.
"To think this was an England Under-21 game and another generation of players is seeing this sort of behaviour makes me very sad and makes me aware of the fact we haven't moved far enough.
"Everyone says UEFA needs to be stronger, do this and do that.
"They've shown they have no stomach for this battle. They haven't taken it on in the past.
"I doubt they will now with paltry fines and slapping on the wrists.
"So that's why it's time for players to take action. I'll go one further and say players should walk off the pitch because - guaranteed - if that happened, things would change."
In his letter to Platini, Robertson said several England players had been subject to "extreme provocation and racism". He also offered to help UEFA's investigation in any way.
Robertson's letter says: "I am writing to you about the incidents at the England v Serbia UEFA European U21 championship play-off second leg last night [Tuesday].
"As you are aware, a number of England players were subject to extreme provocation and racism.
"The British Government absolutely supports the FA's strong condemnation of the incident and joins it in urging you to take the strongest possible sanctions against anyone guilty of the offence.
"As I am sure that you will agree, these scenes, and attitudes, have no place in sport, the modern game of football, or society as a whole.
"Please let me know if there is anything we can do to help your investigation."