The Reds did not seem to come alive until Andy Carroll's introduction in the 55th minute but by then they were 2-0 down to goals from Ramires, in the first half, and Didier Drogba.
Carroll pulled one back just past the hour and almost headed the equaliser late on only to be denied by a great save from Petr Cech, with television replays inconclusive over whether the ball crossed the line before the Czech clawed it away.
But Agger admitted Liverpool paid for their poor start.
"We can't use that (fightback) for much. It simply was not good enough for the first hour and we can only blame ourselves," said the Denmark defender.
"We definitely were not good enough.
"Andy did really well when he came on and got a lot of balls to work with but sadly it was not enough."
Debate continued long after the final whistle over whether Carroll's header had actually crossed the line but Agger said he had no idea from his position on the pitch.
"I definitely couldn't see anything because I was at the other end so people watching television will have to decide that."
Midfielder Jordan Henderson admitted he went up claiming the equalising goal more in hope than expectation.
"I appealed because that is what you have got to do," said the England Under-21 midfielder, who was at a loss to explain Liverpool's sluggish start.
"I am not sure why we started quite slowly in the first half. In the second half we came out and played some good stuff and we should have got back into the game.
"I think we will be disappointed when we look back.
"We wanted to win this game and we didn't and we are very disappointed."