UEFA have been widely criticised for the leniency they have shown toward Serbia following racist chanting and violence toward England's under-21 players during a match in October.
The Serbians were fined 80,000 euros (£65,000) and ordered to play a match behind closed doors.
Ouseley, who on Thursday described the sanction as "paltry", has demanded more drastic action, telling Radio Four: "If necessary games should be stopped and sections of the crowd who are behaving badly should be shown the door."
The Football Association expressed their disappointment at the punishment for Serbia, but Ouseley attacked the FA for undermining his organisation's efforts with their treatment of the John Terry and Luis Suarez cases.
"It (football) clearly is (slipping back to the bad old days)," he added. "If you have your position undermined by people who are in powerful positions and I am talking about the FA."
Peter Herbert, chairman of the Society of Black Lawyers, described UEFA's stance on racism as "abundantly weak".
"As far as Serbia is concerned, they don't even include the word racism or discrimination in their final report," he said on Sky Sports News.
"What did they think was going on then? Did they think people were having a tea party?
"It just beggars belief as far as we're concerned that they should take such an incident, which is not the first time the Serbian FA have been guilty of this in terms of the conduct of their fans, and actually after the second or third occasion give what is effectively a green light to this happening again."