Neil Harris had given the visitors the lead but Junior Stanislas' equaliser that sent the tie into extra-time sparked ugly scenes as supporters poured over the advertising hoardings.
Stanislas added another from the penalty spot, leading to Millwall players heading for the dressing room, before play got under way again and Zavon Hines added a third to seal victory at the end of a poisonous evening of hostility.
Kenny Jackett's League One side, two divisions below the hosts, pushed their Barclays Premier League opponents all the way despite only being able to name four substitutes due to an injury crisis - but the focus will inevitably be on the trouble in and around the ground.
One man was stabbed in the chest near the stadium among the chaos, with a number of arrests made for disorder and breaches of banning orders.
Instead of talk being of Jack Collison bravely declaring himself ready to play despite his father dying in a motorcycle accident on Sunday, there were concerns at kick-off of the trouble that led to a tube delay and empty seats.
Fans taunted each other, with many of them concentrating on their rival supporters rather than the action on the pitch. That was until Harris opened the scoring in the 26th minute for the visitors.
Scott Barron had launched a long throw into the area which Danny Gabbidon, making his first appearance in 18 months, failed to clear. Harris tucked away the finish when the ball fell invitingly.
It could have got better for Millwall when Alan Dunne raided forward down the right and exchanged passes with Gary Alexander - but Robert Green was down sharply to smother the full-back when he was through on goal.
At the other end, Julien Faubert was furious with the assistant referee for not flagging for a penalty after David Martin hauled him down on the edge of the area, with play allowed to continue.
It was the closest West Ham had come to Millwall's goal in the first half, until Scott Parker went on a surging run and fired over the crossbar from the edge of the area.
The Hammers struck the woodwork just after the break. A cross was partially cleared and fell to Collison, who took a touch before clattering the post with his finish, with Millwall goalkeeper David Forde not offering a dive.
Carlton Cole forged a promising effort for himself after muscling his way past Andy Frampton before cutting inside - but his finish was wayward.
It was a case of Millwall holding on as West Ham looked to attack. Gianfranco Zola's response was to bring on Frank Nouble for Radoslav Kovac to give Cole more support, with Hines also introduced to add pace.
The League One visitors provided a threat on the counter attack and Marc Laird just missed the angle of post and crossbar with a 25-yard effort.
Chris Hackett also went close when he tried his luck from long range, with Green required to tip over the crossbar.
There was trouble between fans and police in West Ham areas of the stadium towards the latter stages of the tie as fans tried to encroach on the pitch.
Stanislas' leveller, clipping in Nouble's cross with three minutes remaining, sparked a pitch invasion and more violent scenes, before stewards and police cleared the playing surface.
In extra-time, Hines' cross led to Frampton handling and referee Paul Taylor pointing to the spot, with his assistant flagging for the offence.
Stanislas tucked away the penalty, leading to more bedlam among fans.
Hines raced through, using his pace to beat Frampton, and finished into the top corner for the third.