Fernandinho had fired City into an early lead, only for Wilfried Bony to head Swansea level in first-half stoppage time.
But Toure put the visitors back in front with his 12th goal of the season and Kolarov got their third with an excellent solo effort.
Bony got his second in time added on to give Swansea hope, but City held on.
City's previous visits to Swansea had ended in a defeat and a draw but, after a difficult period either side of the break, they recovered to claim victory on this occasion.
City have now dropped points in just one of their last nine games and this three points moved them top of the table, at least until Arsenal face Cardiff later this afternoon.
It did not take long for Manuel Pellegrini's side to threaten the home goal as Edin Dzeko, Alvaro Negredo and Jesus Navas combined, with the Bosnian firing wide from the edge of box.
Bony smashed a fierce effort just wide at the other end during a lively opening before Swansea suffered the misfortune of losing Pablo Hernandez to injury in just the eighth minute.
The winger, so often hindered by injuries since joining from Valencia, went down clutching his left hamstring and was in tears as he left the field to be replaced by Roland Lamah.
Matters got worse for Swansea when they fell behind in the 14th minute as Fernandinho scored the first Premier League goal of 2014.
Samir Nasri's corner struck Jonathan de Guzman, and the Brazilian midfielder made no mistake in hammering the loose ball beyond Gerhard Tremmel.
Swansea responded well to going behind, with Jonjo Shelvey firing wide and De Guzman failing to beat Hart after a lovely flowing counter involving Lamah and Bony.
At the other end Jesus Navas beat Ben Davies with an outrageous dummy which led to Dzeko seeing his volley deflected wide off Chico Flores.
Swansea continued to press and referee Phil Dowd waved away penalty claims against Vincent Kompany when an Ashley Williams shot struck the Belgian's arm.
The hosts were given two scares when Negredo and Dzeko failed to capitalise on sloppy play in the Swansea defence, and the failure to take those chances as Bony, enjoying one of his best days in a Swansea shirt, gave Joe Hart no chance with a header from De Guzman's pinpoint cross.
Flores and Tremmel almost undid all Swansea's good work six minutes into the second half. The pair hesitated dealing with Nasri's lofted pass and Negredo nipped in, only to shoot straight at the keeper.
There was no such reprieve for the hosts in the 58th minute when they failed to clear a Nasri cross and Toure pounced to fire past Tremmel with the aid of a deflection off Williams.
Eight minutes later the clinching goal arrived. Kolarov robbed Wayne Routledge just inside the City half and drove forward unchallenged before cutting inside on to his right foot to beat Tremmel, who ought to have done better.
The excellent Bony worked Hart at the other end before the keeper got a stroke of luck after he came for, and got nowhere near, a De Guzman corner which Williams could only prod wide of an empty net.
Bony pulled one back in stoppage time with a superb low drive, but it was not enough for Swansea.