City were looking for a win to put them three points clear after Manchester United lost 3-2 to Blackburn on Saturday, but they did not take their chances and Ji Dong-won won it in stoppage time with a goal that had a hint of offside about it.
And Mancini admitted after the game that he was disappointed with the manner in which his side conceded late on.
"I'm really disappointed," he told Sky Sports 1. "Second half we had a lot of chances to score. We didn't score, but we can't concede a counter-attack like this.
"We conceded two, three counter-attacks in the last 10 minutes which were really dangerous. A strong team can't concede this game. Sometimes when you can't win - because it was difficult - you can't lose.
"You need to defend those. Sometimes when you attack with seven players it's enough, because if not you can concede a counter-attack like we conceded.
"In the first half we didn't play very well, second half much better. We had to win this game. This moment is a strange situation."
John O'Shea paid tribute to manager Martin O'Neill for inspiring Sunderland to victory, telling Sky Sports 1: "To be fair to City they're a great team and they had a lot of chances, but as a unit we were great today [Sunday].
"Credit has to go to the manager for giving us that bit of belief to say that we weren't going to lose this game, and he did that at half-time.
"He said, 'We don't want to lose this game 1-0 and people say we did okay, we kept them quiet'."
O'Shea, a former United player, was pleased to do his old side a favour.
He said: "They had a disappointing result yesterday [Saturday] and this will have cheered that up a bit, but more importantly for us it's a fantastic result. The way the lads played showed the character we have in the team and we have to take that forward now."
Midfielder Lee Cattermole said Sunderland's change in fortunes since O'Neill's arrival was to do with belief.
He said: "We've got a spirit where we're going into games believing we're going to win.
"We had some bad luck at the start of the season, I think we were playing the same as we're playing now but we weren't having that luck in the final third."
He also paid tribute to returning goalkeeper Simon Mignolet.
"Fair play to Simon," he said. "He had a real serious facial injury and I thought he was magnificent today [Sunday]."
O'Neill also hailed the spirit in his team. "It's a remarkable start [to the year]," he told Sky Sports 1. "It was a remarkable performance by the team. We had a lot of problems beforehand, problems during the course of the game and obviously problems with Manchester City.
"The spirit is fantastic. Two things [summed it up]: Seb Larsson yesterday [Saturday] was in bed all day, couldn't stand up really, but he was determined to play the game. He saw the game through
"The goalkeeper's done four, five days' training since the injury. He got fitted with the mask a few days ago. He played because Kieran Westwood was ill. His performance was fantastic.
"[It was] epitomised by the captain [Cattermole] today [Sunday], fantastic performance. But the whole team were remarkable, absolutely remarkable. I'm ecstatic."