England had appeared well set to chase down 217, on a slow wicket, only to suffer a ninth defeat in 10 games Down Under this winter.
Last man James Tredwell edged Shane Watson behind with two balls left after he and Chris Jordan had been left needing eight from the final over.
Stuart Broad and Ben Stokes claimed three wickets apiece in a suffocating bowling performance that restricted Australia to their lowest score of the series.
England were on course at 154 for three following a half-century from Joe Root, but lost their last seven wickets for 59 runs to be bowled out for 212.
There was some drama at the end after Ravi Bopara was given out stumped by television umpire Kumar Dharmasena when Australia wicketkeeper Matthew Wade spilled a stumping chance onto the bails.
Replays showed Bopara's foot was in the air when the ball made contact with the stumps, although it appeared he may have recovered his ground when the bail fell off.
Cook will now fly back to England on Monday with just one win over Australia - in Perth on Friday - to show for three months of touring.
Victory for the hosts meant they won the series 4-1 and were able to celebrate victory on Australia Day.