Alberto Zaccheroni's men knew a draw would be enough to secure a place in Brazil next summer, but that looked unlikely when Tommy Oar fired the Socceroos ahead eight minutes from time.
However, Keisuke Honda held his nerve to convert from the spot in the dying seconds to give Japan safe passage through from Asian qualifying Group B.
In an entertaining first half, Yasuhito Endo directed an early free-kick inches wide of the left post before blazing over the bar with his left foot 10 minutes later.
Japan continued to press and Shinji Kagawa forced a one-handed save from Mark Schwarzer at close range before the visiting defence managed to clear the danger.
Australia then found some momentum and produced the best chance of the half soon after when a superb through-ball from Brett Holman teed up Robbie Kruse one-on-one with Japan's goalkeeper.
However, Eiji Kawashima came out on top of that duel before Tim Cahill blazed the rebound over the bar.
Japan threatened again eight minutes after the interval as Kagawa played across a ball across the box which looped over the head of Schwarzer and to the feet of Honda, who poked a shot wide of the right post from two yards out.
Kagawa came even closer moments later, curling an effort from the left side of the area which clipped the crossbar on its way out.
Japan continued to create all the openings but with little reward and they were punished eight minutes from time by Oar's somewhat fortuitous goal, with the midfielder's angled cross-shot dipping over the head of Kawashima and dropping into the net at the far post.
However, luck was on their side in the final minute of the match when Matt McKay handled the ball in the area. The pressure of a watching nation was on Honda, but he slammed home to spark celebrations.
Oman now lie second in Group B behind Japan thanks to a 1-0 win over Iraq.
Abdul Salam Al Mukhaini scored the only goal for the home side before the half-time whistle.
Oman are five points shy of the leaders but, more crucially, two ahead of Jordan and Australia in the battle for a top-two finish and automatic qualification for next summer's World Cup.
The Socceroos face Jordan in Melbourne next week in what is a must-win match for both nations.
Iraq remain at the foot of the table, two points adrift of the pack.
South Korea moved to the top of Group A after Kim Chi-woo grabbed a 1-1 draw against Lebanon in the seventh minute of injury-time.
Lebanon looked set to grab a win to keep their campaign alive when they led deep into injury-time thanks to Hassam Maatouk's 12th-minute opener.
But Kim's deflected free-kick broke home hearts with a red card to Walid Ismail moments later further souring the night.
South Korea moved ahead of Uzbekistan on goal difference with Iran a point further back after they beat Qatar 1-0 in Doha.
Reza Ghoochannejhad scored the decisive goal on 66 minutes.