The Durban side punished mistakes from the hosts by scoring two tries in each half and then holding on for victory.
Riaan Viljoen touched down twice, while Willem Alberts and Lwazi Mvovo also crossed, with Frederic Michalak kicking nine points.
Christian Lealiifano kicking kicked 16 points for the Brumbies, whose two tries came from Jesse Mogg and Nic White.
The Australian side headed into the match in high spirits following their victory over the Highlanders last week, but there were mistakes aplenty from both teams in the opening 10 minutes as dropped possession and wayward kicking led to a scrappy start with turnover ball prevalent.
Michalak kicked a penalty from just outside the 22 awarded for an infringement at the maul to put the Sharks in front.
The Brumbies response was frenetic, a series of drives taking them to within inches of the Sharks line and resulting in a penalty under the posts which Lealiifano had no problem dispatching to level the scores.
It did not take long for the Sharks to go back in front, though, with the first try of the match.
Following a succession of attacks from close range, scrum half Charl McLeod took a quick tap penalty from in the shadow of the Brumbies posts, shifting the ball sharply right to Michalak.
The Frenchman fired a cut-out pass wide to full-back Viljoen who crashed over virtually unopposed in the corner to put the Sharks 8-3 ahead.
A second penalty from Lealiifano reduced the deficit to two before a fantastic try in the 28th minute had the home support on their feet.
Taking a risk inside their own 22, Lealiifano sensed an overlap and spread the ball wide to full-back Mogg.
Mogg accelerated clear and chipped the ball over the covering defence and, as the bounce deceived wing Mvovo, regathered to canter round behind the posts.
The conversion was a formality as the Brumbies led 13-8.
Having tasted success with a stunning counter once, the Brumbies tried again six minutes from the interval, but with very different consequences.
A lofted pass wide was easily intercepted by Odwa Ndungane and the winger had the presence of mind to cut infield and find the supporting runner in Viljoen, who raced away to cross for his second try.
Michalak's successful conversion had the Sharks in front again at 15-13.
But it was the Brumbies who led at the break as Lealiifano kicked his third penalty to edge the home team into a one-point lead.
The Sharks' willingness to keep the ball alive at every opportunity paid dividends early in the second period.
After moving the ball through several pairs of hands, a thrusting run by centre Tim Whitehead resulted in a Brumbies hand patting the ball back towards their own goal-line.
An unfortunate bounce meant the Lealiifano could not collect and flanker Alberts snaffled the loose ball to cross by the base of the post.
Michalak added the extras as the Sharks led 22-16.
A fourth penalty from Lealiifano reduced the arrears again before the Sharks claimed the bonus point with a fourth try.
Another turnover deep inside Sharks territory saw lock Steven Sykes break down the left touchline and pass infield, where Mvovo showed a clean pair of heels to the Brumbies defence to streak away into the corner with Michalak converting once more.
The Brumbies piled on the pressure in a bid to find a way back into the match, but, despite conceding a stoppage-time try to replacement White, which was converted by Lealiifano, the Sharks held on for victory.