Friend opted to dismiss the former Manchester United man at the Britannia Stadium for a challenge on Stoke's Charlie Adam, despite Brown winning the ball cleanly and making minimal contact with his opponent.
The decision has been widely condemned by pundits ever since - relegation-threatened Sunderland went on to lose the Barclays Premier League game 2-0 - and Gallagher conceded the official had blundered.
He told Sky Sports News: "I can keep this very short. We have to take it smack on the chin. The referee has lost concentration or not seen it.
"It is not a red card, that's all you can say.
"Brown's gone in hard but he's entitled to do that. As a refereeing body you have to take that smack on the chin."
After a weekend full of controversial decisions, Gallagher also felt Phil Dowd had made a mistake in not sending-off Everton's Kevin Mirallas for a wild lunge on Liverpool's Luis Suarez in the Merseyside derby.
"He should have been sent-off," said Gallagher. "He's come at speed and it's high.
"The referee has got a body between him and Suarez and he hasn't seen it."