Ajax had won the first leg 2-0 but Iasmin Latovlevici's excellent first-half goal put Steaua ahead on the night and Vlad Chiriches struck from long range to force extra time.
It was no less than the Romanian side deserved and following a goalless extra-time period, Latovlevici netted the winning penalty after Lasse Schone and Niklas Moisander had missed for Ajax.
Ajax's Christian Eriksen thrashed an early free-kick over the bar and Kolbeinn Sigthorsson's close-range effort was crucially blocked by Florin Gardos.
Steaua did not threaten in the opening half-hour but were then handed a first sight of goal when Ajax keeper Kenneth Vermeer handled a back-pass. The Eredivisie side put all 11 players on the goal line for the free-kick seven yards out and it was striker Sigthorsson who blocked Raul Rusescu's shot.
But the breakthrough came five minutes later when Latovlevici controlled Chiriches' long diagonal pass on his chest and neatly lifted the ball over the onrushing Vermeer, who stayed down in pain after a collision which also involved full-back Ricardo van Rhijn.
There were no further attempts on goal until 10 minutes after half-time, when Steaua's Cristian Tanase fired wide from long range.
Viktor Fischer responded in kind but Steaua went close to an equaliser in the tie when Adrian Popa volleyed against the post from 20 yards.
Montenegro forward Stefan Nikolic was sent on in place of Tanase with 25 minutes remaining in which for the hosts to level matters.
Chiriches volleyed wide soon after but was not to be denied, arrowing a 20-yard shot into the top corner to make it 2-0 on the night and 2-2 on aggregate.
That finally awoke Ajax from their slumber and Moisander headed one Eriksen corner against a post before the next fell to substitute Derk Boerrigter, who was denied by keeper Ciprian Tatarusanu.
They could not find a way through, though, and with nothing to match the late drama provided by Chelsea's Eden Hazard against Sparta Prague the tie went to extra time
Vermeer held Latovlevici's drive in the first extra period and there was no addition to the score as the game headed for spot-kicks.
Eriksen scored Ajax's first but Tatarusanu saved from substitute Schone and Nikolic added to Rusescu's successful effort to put Steaua 2-1 up.
Siem de Jong and Lucian Filip were both successful but defender Moisander blazed over before Latovlevici settled the tie.