The Hammers' striker was injured for their 4-1 defeat at Anfield in December, but after two goals in his last four matches he is finding a bit of form ahead of the visit of the Reds.
It will be the first time Carroll has faced his former side since being off-loaded by Brendan Rodgers for £15million - a massive £20million loss on his record fee of January 2011.
Carroll will no doubt have a point to prove to Rodgers in particular, but Agger said the focus has to remain on themselves and not whom they are facing.
"West Ham is a tough place to go. They have a few more than Andy, it is a good strong team," he told Press Association Sport.
"You cannot underestimate anyone, we see that week in, week out.
"I think their whole team will give us a hard time, but we have to focus on our game and the way we play.
"The most important thing is to win the next one."
Liverpool went top of the table with a 4-0 victory over Tottenham last Sunday to make it eight successive league wins, but they could be passed by Chelsea if they beat Stoke on Saturday.
The Reds found themselves in a similar position at Christmas when they were top on December 25 yet two games later were outside the top four.
Agger said for that reason no-one was getting carried away with reaching the summit again.
"The most important thing is to be top of the league at the end of the season because the league is so tight," added the Denmark international.
"I think we showed confidence against Tottenham, but the most important thing is to stay focused and know your job.
"We wanted to win our last game and that was what was in our head and now we have prepared for this week."
Manager Brendan Rodgers is confident his players will not be distracted by the title run-in, especially being chased by Chelsea and Manchester City, who both have more experienced sides in terms of winning it recently.
He, for one, will not be losing too much sleep over what the chasing pack are doing.
"We can't afford to worry about other teams," he said.
"Manchester City are a squad full of outstanding players with an excellent manager and are a club who have been building for a number of years.
"Chelsea have some top, experienced players and a world-class manager and if I spent my time worrying about our rivals and what they have got I'd be wasting my energy.
"We are very calm and focused on what our job is and for me it is about the next training session and instilling the confidence and belief in the squad that we can get results.
"We will continue to work that way until the end of the season.
"A lot of teams would swap where we are in the league. For me it is just about the concentration on performance, that is what our focus has been on and we have been performing to a real high level.
"What has been brilliant about this team is the way we are playing and scoring goals in big games and that gives us great confidence going into these remaining games.
"West Ham have won six out of the last nine and we respect them as an opponent."