Farid El Allagui grabbed a double for the Irn-Bru First Division side as they moved into a 2-0 lead but Dorin Goian and Nikica Jelavic looked to have forced extra-time when they levelled things up at 2-2.
But Mark Millar had the final say with an injury-time winner as Falkirk booked their place in the fourth round.
Ayr claimed their second top-flight scalp in this year's competition after defeating Hearts 4-1 on penalties following a 1-1 stalemate at Somerset Park.
Danny Grainger and Rudi Skacel were both denied by Ayr goalkeeper Kevin Cuthbert in the shoot-out while Mark Roberts coolly slotted home the winning spot-kick for the Irn-Bru First Division side, who eliminated Inverness in the previous round.
In normal play Gareth Wardlaw equalised to cancel out Scott Robinson's 48th-minute opener for Hearts and send the tie to extra-time.
Hearts were left angry that Eggert Jonsson's second-half 'goal' was disallowed for handball.
Gary Hooper's early goal sent Celtic on their way to a comfortable 2-0 win at Dingwall as they avoided another shock against Ross County.
Hooper netted from close range after a 14th-minute corner before Scott Boyd's own goal early in the second half removed any doubt about the outcome of the tie.
Celtic survived one tense moment in between the goals when Michael Gardyne volleyed against the post but County otherwise did not look like emulating their 2010 Scottish Cup semi-final win over Neil Lennon's men.