The Norwegians, reeling from Friday night's opening defeat to Iceland, fell behind to Marko Suler's 17th-minute opener but Markus Henriksen equalised 10 minutes later before Riise kept his nerve from the spot.
The visitors, themselves desperate for points after an opening loss to Switzerland, started well and forced the first save in the 15th minute when Jasmin Kurtic's 25-yard shot was parried by Rune Almenning Jarstein.
Two minutes later Suler grabbed the opener with a right-foot shot low past Jarstein after picking up Kurtic's left-wing cross on the edge of the six-yard box.
Slovenia keeper Jan Oblak parried a Riise effort from long range in the 22nd minute and the Fulham man remained the biggest danger, although Tim Matavz came close to extending Slovenia's lead when he fired over on 24 minutes.
Henriksen settled home nerves in the 27th minute when he headed home following a lofted ball into the box from Tarik Elyounoussi.
Kurtic headed just over for Slovenia before the break, and the second half started slowly before Daniel Braaten tested Oblak with a shot from the edge of the box.
With both sides anxious to avoid a second consecutive defeat, chances were relatively few and far between until the last minute, when Henriksen's shot from six yards was well saved by Oblak.
Deep into injury time, Dejan Kelhar fouled Alexander Soderlund in the Slovenian box and Riise stepped up to nervelessly fired home the penalty and give Norway their first points.