It emerged over the weekend that Frazier was fighting liver cancer and he was reported to be in hospice care.
And a family statement reported by Philadelphia's WPVI-TV read: "We, the family of the 1964 Olympic boxing heavyweight gold-medallist, former heavyweight boxing champion and International Boxing Hall of Fame member Smokin' Joe Frazier, regret to inform you of his passing.
"He transitioned from this life as "One of God's Men," on the eve of November 7, 2011 at his home in Philadelphia, Pennsylvania. We thank you for your prayers for our father and vast outpouring of love and support.
"Respectfully, we request time to grieve privately as a family. Our father's home going celebration will be announced as soon as possible. Thank you for your understanding."
Frazier won Olympic gold in 1964 after going to the Games as a replacement for the injured Buster Mathis, who had beaten him in the trials.
He won the professional heavyweight title in 1970 by stopping Jimmy Ellis in the fifth round at Madison Square Garden and defended it successfully four times before losing it to George Foreman in 1973.
Frazier is best remembered for his three bouts with Muhammad Ali, the first of which saw him unify the world title when he inflicted Ali's first professional defeat in the so-called 'Fight of the Century' in 1971.
But in two further fights, including the epic 'Thrilla in Manila' in 1975, it was Ali who was victorious.
Ali, 69, said on Sunday upon hearing of Frazier's illness: "The news about Joe is hard to believe and even harder to accept.
"My family and I are keeping Joe and his family in our daily prayers. Joe has a lot of friends pulling for him - and I'm one of them."
British heavyweight Dereck Chisora, also talking before Frazier's death, told Press Association Sport: "It's a sad time. I watched all the fighters as a kid and he was a real inspiration.
"The guy's lived his life and you know what, if you have to die after you've achieved all you've achieved, you go in a good way.
"What can you say about 'Smoking Joe'? A great fighter."
Frazier retired in 1976 after losing a rematch against Foreman, and aborted an attempted comeback five years later after a draw with the unheralded Jumbo Cummings.
Reaction to "Smokin' Joe" Frazier's passing
Some of the sporting world's biggest names have been quick to pay tribute to Joe Frazier on Twitter. Here's a taste of what has been said:
@FloydMayweather: RIP Smokin Joe. My thoughts and prayers go out to to the Frazier family. We lost an all time great tonight.
My Condolences go out to the family of the late great Joe Frazier. #TheMoneyTeam will pay for his Funeral services.
@SHAQ: Another legend lost R I P to the great joe Frazier, peace brother you will be missed.
@OscarDeLaHoya I will miss you my friend. R.I.P. Joe frazier.
@KingJames: R.I.P Joe Frazier! Heavyweight Great!! One of the best! #SmokingJoe.