Show Racism the Red Card says it understands why players such as Rio and Anton Ferdinand and Jason Roberts refused to wear the shirts publicising the Kick It Out week of action.
The players were apparently unhappy at the organisation's lack of voice over the high-profile racial abuse cases in English football over the last year involving John Terry and Luis Suarez.
A statement from Show Racism the Red Card said: "We fully understand the anger of the Ferdinand family and Jason Roberts in relation to the FA handling of the John Terry case. Both Rio and Jason are long-standing patrons of our campaign and are recognised in our Hall of Fame for their work.
"The issue of not wearing the Kick It Out (KIO) shirts at the weekend highlights the displeasure of certain players in relation to the footballing authorities' handling of the incidents of racism in the game.
"We call on the players involved to now sit down with us, KIO and the PFA to draw up a plan of action to present to the footballing authorities and Government."
Kick It Out's chairman Lord Herman Ouseley said the players needed to speak openly about their grievances.
He told BBC Radio Five: "The issue is that the T-shirts have become the story whereas the actual grievances of black players, both current and former, have not come out in the open.
"The black players who have expressed themselves by saying they are not going to wear the T-shirt are doing so because they genuinely believe there are grievances that have not been addressed.
"Those grievances can only be addressed if we confront them, not by gestures of not wearing a T-shirt, but I understand why people don't do that."
Meanwhile, former Liverpool and England star John Barnes has called on Rio Ferdinand to explain his decision to boycott a Kick It Out T-shirt campaign.
Barnes told talkSPORT: "If you are brave enough to go against Sir Alex Ferguson as Rio did, hen he's brave enough to say who he is unhappy with, because I can't see that they are unhappy with Kick It Out.
"They should target the right people. Kick it Out have no power to ban or fine people.
"I understand the reasons for them highlighting it, but if they are not happy with the ban that John Terry got then it's the FA and they shouldn't target Kick It Out."
Meanwhile, Wales and Aberdeen keeper Jason Brown claimed black players could be driven towards setting up a breakaway anti-racism group.
He told Sky Sports News: "People come out and say there can't be a breakaway, I'm all for that. We don`t want to be rebels and breakaway, but if they`re not doing enough, they're driving us to go down that road.
"I know four to five players who have met with Kick It Out and the PFA and gave them suggestions, but it seems they must have got lost because they have done nothing.
"We're not here to become rebels, we love the game but at the same time, if people we trust, the Union, the PFA, Kick It Out, are not doing what we ask of them, what are they here for?" he said.