Partizan Belgrade midfielder Ninkovic was prominent in the unsavoury scenes at the final whistle in the Euro 2013 play-off on Tuesday which saw players and staff from both sides scuffling on the pitch with Rose then sent-off for kicking the ball away in anger at what he perceived was racist abuse from the crowd.
UEFA on Wednesday night announced charges against both the Serbian FA and the English for improper conduct of their players, and against the Serbs for "alleged racist chanting'' by fans.
Ninkovic claims Rose was the instigator in the trouble, telling Serbian newspaper Novosti: "The guy three or four times made inappropriate gestures towards our fans.
"I was angered by his reaction so I rushed over to explain to him that he should not do that.
"I did not mean to hit him or insult him. I walked over to him and pushed him.
"A lot of his behaviour annoyed me but I know I was wrong and I know I will get a fine. I just hope it will not be too big.
"He received a red card and I got a yellow. That is probably an indication that his share of the story was significant."
The English FA has sent UEFA a full dossier of evidence, including video footage and a catalogue of incidents which they reported to match officials during and after the match.
And FA general secretary Alex Horne has questioned whether they will play matches in Serbia again.
Horne said in a statement: "We were shocked and appalled by the disgraceful events that occurred in Serbia.
"Our players and staff were subjected to racial abuse (and) violence as well as missiles being thrown at them throughout the match. What occurred is inexcusable and not acceptable.
"It is also clear that we must defend Danny Rose, who was sent off due to the frustration of being a target of racial abuse.
"FA officials made UEFA officials aware of racist abuse from sections of the crowd aimed at a number of England's black players at half-time.
"No football team should be asked to play in any environment where racial abuse, violence and threatening behaviour is prevalent. We must question the validity of sending a team to Serbia in the future.''
However, the Serbian FA statement, which they translated into English, denied that their fans were guilty of "any occurrences of racism''.
"The FA of Serbia absolutely refuses [sic] and denies that there were any occurrences of racism before and during the match at the stadium in Krusevac,'' the statement read.
"Making connection between the seen incident - a fight between members of the two teams - and racism has absolutely no ground and we consider it to be a total malevolence.
"And while most of the English team players celebrated the score, their player number three, Danny Rose, behaved in [an] inappropriate, unsportsmanlike and vulgar manner towards the supporters on the stands at the stadium in Krusevac, and for that he was shown a red card.''
The cases will be dealt with at a meeting of UEFA's control and disciplinary body on November 22.