Following on from the efforts of MK Dons and Luton on Saturday, the npower League One side overcame top-flight opposition thanks to two goals from Matt Smith at Boundary Park.
Smith scored the opener as early as the third minute, and although Luiz Suarez equalised 14 minutes later, the Latics striker grabbed his second on the stroke of half-time.
Reece Wabara's header made it 3-1 after 48 minutes before Joe Allen again reduced arrears 10 minutes from time, but Oldham held on for a famous win.
Tottenham also bowed out as they were beaten 2-1 at npower Championship side Leeds.
Luke Varney fired the home side in front after 15 minutes at Elland Road, with Ross McCormack doubling the lead five minutes after the interval.
Clint Dempsey pulled a goal back for Spurs eight minutes later but they could not add to their tally as Neil Warnock's men held firm.
Brentford earned a fourth-round replay against Chelsea after holding the European champions to a 2-2 draw at Griffin Park.
The League One side twice took the lead through Marcello Trotta (42) and a Harry Forrester penalty (74), but Oscar grabbed the first equaliser after 55 minutes before Fernando Torres rescued Chelsea seven minutes from time with his first goal of 2013.
Meanwhile on Saturday, Arsenal, Manchester City and United edged into the fifth round of the FA Cup while Premier League sides Norwich and QPR were dumped out by Luton and MK Dons.
Luton produced the surprise result of the FA Cup fourth round with a 1-0 win at Premier League Norwich.
Scott Rendell's 80th minute goal proved decisive in a clash between two sides separated by 85 places in the English football ladder.
The Blue Square Premier side's victory at Carrow Road means they are only the seventh non-league team to reach the fifth round of the competition since World War Two.
Norwich were not the only top flight team embarrassed, as MK Dons won 4-2 at QPR.
Harry Redknapp, who made sweeping changes ahead of Tuesday's visit of Manchester City in the league, saw his team thrashed by their League One opponents.
QPR knocked MK Dons out in the third round of this competition last year, but Armand Traore's fourth-minute own goal and Ryan Lowe's 40th-minute effort gave the visitors control at half-time.
Ryan Harley (50) and Darren Potter (56) made it 4-0 before Jay Bothroyd and Fabio pulled two goals back inside the last seven minutes for the Premier League side.
Olivier Giroud, who netted twice in the midweek defeat of West Ham in the Premier League, repeated the feat as Arsenal won 3-2 at Brighton.
The Frenchman gave the Gunners a 16th-minute lead at Brighton, but Ashley Barnes' header levelled the scores 17 minutes later.
Giroud restored the visitors' advantage 11 minutes into the second half, only for Argentinian striker Jose Ulloa to level on his debut six minutes later.
However, Theo Walcott struck to win it for Arsene Wenger's side five minutes from time.
Manchester City booked their place in Sunday's fifth round draw with a 1-0 victory in the early kick-off at Stoke.
Pablo Zabaleta struck the only goal five minutes from time as City repeated the scoreline which won the 2011 final against the same opponents.
Yet victory may still have come at a cost for Roberto Mancini's men after captain Vincent Kompany limped off in the first half.
Neighbours Manchester United were convincing 4-1 conquerors of Fulham in the evening match.
Ryan Giggs converted a third-minute penalty before Wayne Rooney (50) and a double from Javier Hernandez (52 and 66) added to the lead. Aaron Hughes, whose handball gave away the early spot-kick, replied for the Cottagers 13 minutes from time.
Johnny Heitinga's 90th-minute goal gave Everton a 2-1 success at Bolton.
Steven Pienaar's scrappy 18th-minute goal gave David Moyes' men the lead before Marvin Sordell - whose two goals saw off Sunderland in their third round replay - squared things nine minutes later.
Reading eased into the last 16 with a 4-0 win over Sheffield United.
Noel Hunt opened the scoring in the sixth minute, Mikele Leigertwood's long-range goal against one of his former teams doubled the advantage five minutes before the break, before Hunt and Garath McCleary made sure of the win within nine minutes of the second half starting.
Blue Square Premier Macclesfield were unable to emulate Luton's exploits, Jordi Gomez's seventh-minute penalty giving Premier League Wigan a 1-0 win at Moss Rose.
There was late heartbreak for League Two Aldershot as they lost 2-1 at Championship Middlesbrough.
Danny Hylton's 89th-minute goal appeared to have earned the Hampshire side a replay but Lukas Jutkiewicz added to his 83rd-minute effort with a last-minute winner.
Blackburn were convincing 3-0 winners at Derby thanks to goals from Colin Kazim-Richards (45), Scott Dann (66) and Jordan Rhodes (71), while Chris Dagnall's goal 20 minutes from time decided another all-Championship clash as Barnsley won 1-0 at Hull.
Lee Novak's 74th-minute penalty gave Huddersfield, under the leadership of caretaker manager Mark Lillis following the sacking of Simon Grayson, the lead against Championship rivalsLeicester.
However, Chris Wood earned a replay for the visitors with a goal eight minutes from time as the game ended 1-1.
Sunday's FA Cup Results
Brentford 2-2 Chelsea
Leeds 2-1 Tottenham
Oldham 3-2 Liverpool
Saturday's FA Cup Results
Stoke 0-1 Manchester City
Brighton 2-3 Arsenal
Manchester United 4-1 Fulham
QPR 2-4 MK Dons
Norwich 0-1 Luton
Bolton 1-2 Everton
Macclesfield 0-1 Wigan
Reading 4-0 Sheffield United
Huddersfield 1-1 Leicester
Hull 0-1 Barnsley
Middlesbrough 2-1 Aldershot
Friday's FA Cup Results
Millwall 2- 1 Aston Villa