The winner remains undefeated but Manfredo has retired following the defeat -his final record one of 37 wins, 20 by knockout, and seven losses.
Chavez dominated the early exchanges of the bout and took the first three rounds but in the fourth Manfredo engaged his opponent in a brawl.
'The Pride of Providence' came out swinging in the fourth and he performed with credit but Chavez gave as good as he got and it was all over in the next.
Chavez hurt Manfredo in the fifth and referee Laurence Cole stepped in at 1:52 to end the fight.
"I want to fight the best," Chavez said after the fight, via his translator. "Sergio Martinez is a great champ. 'Canelo' would be a great fight for the Mexican public. But it all depends on what my promoter says. I fear no one."
On Twitter, his promoter Lou DiBella said, "Pete Manfredo Jr. just retired. Great kid. Really poignant speech. Wish you could have heard it. He did himself and Providence proud."
Top Rank's Twitter quoted Manfredo as saying, "I'm like a throwback fighter and they don't let that happen nowadays. You get hurt and they stop it. ...
"There are a lot of weak minded people in this world and this game makes you a strong person through training. ... I don't want to promote or be part of boxing. I just wanna (eat) a sandwich."