The England winger lines up against his old club Aston Villa on Saturday in an Upton Park encounter between two sides who are struggling to bring their top form to the Premier League consistently.
Downing has been criticised in some quarters for underachieving after winning 34 caps for his country, although his most recent appearance came last year in a 1-0 victory in Norway, which was Roy Hodgson's first game in charge.
He joined West Ham in August in a £5million move from Liverpool and Allardyce said: "Other people's opinion is of no interest to me. My opinion is the only opinion that matters between me and Stewart Downing.
"Stewart Downing had a difficult pre-season where he wasn't sure where he was coming or going. Finally he made a decision on West Ham United which we were delighted with and there's very good progression and much more contribution as the more games he plays.
"The more and more quality we see on a much more regular basis. There's a settling-in period like anybody else. People divide their opinion far too quickly in this game. Real Madrid have done it with Gareth Bale, haven't they?
"He's no good and he's a waste of money and all that bull**** is coming out of there now and then he scores two and now he's a hero.
"We all know as professionals we put in the ear plugs when that sort of stuff starts happening and get on with what we should professionally and that is concentrating on ourselves and our game and not getting unfocused by praise or criticism.
"You get praise through the media brilliantly as well but that can affect you just as much as the criticism so just don't listen to it, try to keep it at a minimum and keep focused and moving forward."
The same could apply to Joe Cole, another former England star who has struggled with injuries in recent times but played his first 90 minutes of the season for the Hammers in the Capital One Cup win over Burnley in midweek after recovering from a hamstring problem.
"It was a big boost for us as we never expected Joe to be out as long as he was," Allardyce added.
"It was a real blow after the way he started against Cardiff and then picked up what was considered only to be a slight hamstring problem before the game against Stoke.
"It's taken a lot longer than we thought. As long as we keep Joe fit he is a massive player for us and one we look forward to using and staying fit."
Asked if he needed to prove his fitness to earn a new deal at Upton Park Allardyce said: "It's not about proving his fitness, it's about staying fit. There is a big difference. Joe is fit. Even in his thirties his fitness stats are very good.
"Joe's concern is not picking up injuries. A proven period of time when the injuries do not reoccur is more what we are trying to look for.
"At the end of the season we evaluate what the contribution of the player is, not just what he has contributed in the games he has played but also how many games has he played.
"If he has only played five and you divide that over his wages, it's pretty expensive per game. That is a really good equation to work by when you talk about new contracts."