The Evo-Stik Northern Premier League side trailed at half-time to a Jonathan Franks goal, but Stephen Turnbull’s superb free-kick got them back on level terms before Rivers burst into the area and shot into the far corner to seal a famous upset.
The game had been billed a potential shock in the pre-match build-up, but Hartlepool dominated the first half at Victoria Park and took a deserved lead in the 31st minute when striker Franks headed home at the far post.
The League Two strugglers spurned several clear-cut chances to double their lead before the interval and they were made to pay for their profligacy when Spartans midfielder Turnbull curled home a delightful free-kick 11 minutes after the interval to level the scores.
Marlon Harewood thought he had bundled home Hartlepool’s second in the 66th minute as Blyth goalkeeper Peter Jeffries fumbled his shot on the turn, but, while replays suggested all of the ball had crossed the line, referee Andrew Madley was unmoved.
Blyth, who reached the fifth round in 1977-78 and third round in 2008-09, grew in confidence as the second half wore on and in the final minute they were rewarded for their spirited endeavour.
Striker Rivers skipped past a challenge from left-back Dan Jones, ran into the Hartlepool area and fired a low shot across Pools keeper Scott Flinders and into the net.
Hartlepool had six minutes of added time in which to find an equaliser and they very nearly did when centre-back Matthew Bates took aim from the edge of the area, but his shot cannoned off the crossbar before being hacked to safety.