This was always likely to be a tricky tie for the Hammers with their opponents, leading the Sky Bet Championship, in such buoyant mood after recent results and bidding to win an 11th straight League Cup tie at Turf Moor.
Yet, in a much-improved second-half showing, Sam Allardyce's substitutions helped turn the tide in the visitors' favour with Kevin Nolan, one of those replacements, winning the all-important first spot-kick and Collison earning and converting the second.
The hosts had shaded the first half but found stubborn resistance in the form of Hammers goalkeeper Adrian, who twice repelled efforts from Junior Stanislas, while a stop from Danny Ings sandwiched in-between was the pick of the bunch.
Allardyce's men were struggling to ask questions at the other end despite fans' favourite Carlton Cole and Modibo Maiga both deployed as strikers.
In the second half the contest began to open up but it was still the Clarets who looked more threatening, full-back Kieran Trippier hitting the crossbar with a wicked free-kick after the hour mark.
However, with Collison, Stewart Downing and Nolan all introduced, it was West Ham who drew first blood.
Nolan was brought down by Jason Shackell in the area and, although the centre-back looked to have won the ball, referee Robert Madley pointed to the spot with Taylor making no mistake.
As the visitors tried to force extra-time, Collison sprung on the break and was hauled down by Keith Treacy, with the winger being sent off, before the Welsh midfielder himself stroked home the penalty to hand Sean Dyche's men their first defeat since August.
Allardyce decided to retain just James Tomkins and Mohamed Diame from the XI which began the game against Swansea 48 hours earlier, with Carlton Cole starting for the first time since resigning and Pelly Ruddock making his debut.
It was a chance for the next line of products from the Hammers' highly-regard academy to shine but it was a former trainee who created the first decent chance.
Stanislas, one of Burnley's four changes, drifted in from the flank and forced Adrian to tip a right-footed effort around the post.
From the resulting corner, Ings swivelled to find some space with Adrian again well-placed to push the ball above his crossbar.
Even with two recognised strikers in their line-up, the visitors were struggling to fashion openings, a rusty Carlton Cole also needing treatment half an hour in for a knee injury.
At the other end Dyche's men almost got their reward before the interval.
Trippier's cross was met by Sam Vokes' head and although he planted his effort against the post, the assistant referee's flag brought a halt to proceedings.
Sensing the need for extra invention, Allardyce replaced Diame with Collison for the restart but he must have felt deflated when debutant Ruddock's wild effort was dragged out for a throw.
They were beginning to finally some attacking intent, though, and had the ball in the net, only to be denied by the offside flag, as Joe Cole stopped Maiga's pot-shot before dispatching a neat overhead kick.
A tense cup tie suddenly sparked into life, Treacy's well-struck volley just failing to stay low enough to trouble Adrian.
The stopper was all at sea after the hour, though, as Burnley again hit the woodwork.
Tomkins' needless foul on Ings set up Trippier's set piece and the full-back's viciously swerving effort beat Adrian before cannoning off the crossbar.
Allardyce continued to turn to the bench in a bid to wrestle the game's momentum and it was Nolan whose introduction did lead to the opening goal 76 minutes in.
The former Bolton midfielder collected the ball in the area and went to ground under a challenge from Clarets captain Shackell, who looked to have nicked the ball away fairly.
Nevertheless referee Madley pointed to the spot, booking Shackell, and allowing Taylor to convincingly slam home the spot-kick in front of the away fans.
The hosts pushed for a leveller but were caught on the break in stoppage time when goalkeeper Tom Heaton mishit a clearance.
Treacy tripped Collison from behind, with little doubt about either the penalty's award or the red card and the midfielder followed suit by finding the same corner as Taylor had 14 minutes earlier.