It was the Barcelona forward who led the way for Dunga's side in an error-strewn fixture that bared witness to yet more calamitous defending from Brazil.
However, moments of brilliance from Neymar ensured that eight-time Copa America champions emerged with the win.
It was Peru who opened the scoring in just the third minute, but while Christian Cueva found the back of the net, the goal was all but handed to the Alianza Lima forward.
In scenes all too reminiscent to those in the 7-1 defeat to Germany in last year's World Cup, a dangerous David Luiz backpass put goalkeeper Jefferson under pressure, with the Botafogo keeper's weak clearance going straight to Cueva who made no mistake from just 10 yards out.
Peru would not lead for long though, with an unmarked Neymar getting on the end of a Dani Alves cross in the fifth minute to head home.
Neymar nearly handed Brazil the advantage on 13 minutes, but he was denied by a goal-line clearance from Carlos Zambrano.
The match tightened up and neither side could create any clear-cut chances in the remainder of the first half as the teams headed to the break level at 1-1.
After the interval it was Neymar who came closest to putting Brazil ahead when he rattled the crossbar with a superb curling effort from 25 yards out.
Costa should have made it two in the 75th minute, but he poked the ball wide after being put through on goal by a great pass from Neymar.
The Shakhtar Donetsk forward made amends in the 92nd minute when he made no mistake after getting on the end of a defence-splitting ball from Neymar on the left.
The win is Brazil's 11th in a row, with Dunga's men undefeated since the 2014 World Cup.
In the day's other Group C encounter, a second-half goal from Jose Salomon Rondon was enough to hand Venezuela a surprise 1-0 victory over Colombia.
Colombia have been tipped by many as a potential dark horse at the tournament, but instead it was they who were on the receiving end of an upset.
In a first half in which neither side offered much, it was Ronald Vargas and Alejandro Guerra who came the closest to opening the scoring for Venezuela, but the score remained goalless at the break.
With Radamel Falcao and Carlos Bacca expected to pose a major threat, Venezuela sat back and were content to base their attacks on the occasional counter. And the tactic worked well, with Colombia failing to make inroads.
Colombia looked better after the break, with James Rodriguez coming close with an effort from the edge of the box.
But on the hour mark it was Venezuela who grabbed a deserved opener, with Zenit St Petersburg striker Rondon on hand to header past David Ospina after Alejandro Guerra had headed a cross into the danger area.
Colombia pushed for an equaliser, but with Falcao unable to make his mark on the match and James Rodriguez finding himself isolated, Jose Pekerman's side struggled to carve out real opportunities.
The best chance for an equaliser fell to replacement Teofilo Gutierrez. Just a minute after coming on to replace the ineffective Bacca, the River Plate man hesitated when through on goal and was denied by Venezuela keeper Carmilo Vargas.