Allardyce felt mistakes at the back by his men in the second half of the Barclays Premier League contest - which was 1-1 at half-time after Peter Odemwingie had cancelled out Andy Carroll's early opener - certainly cost them dear as the Potters wrapped up victory.
Marko Arnautovic scored after squeezing through a crowd of West Ham players and Odemwingie then finished off a counter-attack move by lashing the ball in.
But the Hammers boss also pointed the finger at Pawson, who did not give a penalty for what Allardyce viewed as a "blatant" handball by Stoke defender Marc Muniesa in the home box shortly after Arnautovic had made it 2-1.
"I think in the cold light of day, we have to blame ourselves," Allardyce said.
"But there is a big part to blame the referee for as well.
"There is absolutely no doubt about that in my mind - I have just reviewed the situation on the laptop and watched a blatant handball that stops Andy Carroll chesting the ball down and maybe having a chance to shoot and score a goal.
"It is arm to ball, not ball to arm, and the referee is in a fantastic position to see it.
"So I am totally and utterly baffled by that decision.
"But the other thing is that it is really our fault for the poor defending we showed in the second half - on two occasions, where we just had to defend correctly, and when we were in the ascendancy, particularly with the first goal (Arnautovic's).
"We were in control, and it was looking like if anyone was going to score at that stage, it would be us.
"But then we fall asleep, and I just don't know where our defending went then.
"So those two bits - one from the referee and one from our defence - is why we are not here with some points on the board."
Stoke boss Mark Hughes felt differently about the Muniesa incident.
"I think there was contact, but in fairness to Marc Muniesa I think he wasn't even looking at the ball when it actually hit him," Hughes said.
"He made a genuine attempt to challenge for the ball and was up against Andy Carroll, who is about a foot and a half bigger than him, so you have to use your arms for leverage to try to get anywhere near a challenge that will put him off.
"It probably did hit his elbow, but it wasn't deliberate and it would have been harsh if it had been given - and even it had been, I still think we had enough to take the game away from them."
Aside from the lapse in the opening stages, which saw Carroll head West Ham in front in the fifth minute, Hughes was "really pleased" with his side's overall display.
Stoke, on 34 points from 30 matches and nine points clear of the relegation zone, have now lost only once in their last seven league fixtures.
Hughes has tried to implement a more flowing style of play at the club since becoming manager last summer.
And he feels people are now really seeing that the Potters are moving in the right direction.
"I think people are recognising that now," he said.
"I always said it was a slow process, and that remaining difficult to beat was key to it.
"We have done, certainly here, but some of our play has been excellent too and I am really encouraged by that."
The major plus for West Ham, who have 31 points from 29 games, was Carroll's performance and what was his first goal of the season.
The striker, the Hammers' record signing, has had a frustrating 2013/14 due to injury and suspension, and this was only his seventh appearance of the campaign.
He is looking to shine between now and the end of the term, still holding hopes to make the England squad for this summer's World Cup.
And Allardyce said of Carroll: "We can keep him fit, we can play off him, he can hold the ball up, he can score goals and that has to be a big part of what he continues to do from now on.
"He is going to be tired - his match fitness is not quite going to be there, but then there is only games that can bring it.
"Andy playing and resting now will be the best thing for him, and the more matches he plays, the sharper he gets."