Mackay made 11 changes to his starting line-up and watched in horror as they found themselves 2-0 down with seven minutes gone thanks to goals from Ravel Morrison - after 20 seconds - and Matt Jarvis.
They continued to play like 11 strangers with West Ham threatening to run riot until, out of nothing, Craig Noone curled in a stunning goal just before half-time.
And Cardiff looked like pulling off an unlikely comeback when Peter Odemwingie, on his full debut, tapped in an equaliser.
But instead Ricardo Vaz Te struck with a header two minutes from the end to send the hosts through with a 3-2 win.
However, Mackay said: "In the second half we showed great belief in ourselves and looked like the team that was going to win it.
"I'm disappointed for them after the way they played in the second half.
"I'm asking people to give me headaches in terms of who I am going to pick for Premier League games, and certainly in the second half four or five did."
It meant Sam Allardyce was able to celebrate a victory in his 100th game as Hammers manager - against the team who had inflicted a home defeat in his first match at the helm two years ago.
"It's always good to win and Sam is pleased we are through, although he's slightly disappointed with some aspects," said assistant manager Neil McDonald.
"We probably thought it was too easy and took our foot of the pedal. And Cardiff scored a great goal just before half-time.
"When you are on the back foot after conceding just before half-time it changes everything. People come in expecting a pat on the back but instead we gave them a talking-to.
"To be fair to Cardiff they got the equaliser but we stuck together, we worked hard for each other and we'll take a late winner and a 3-2 win."