With the game tied 2-2 with five minutes remaining, Fabrizio Miccoli put the hosts in front with a superb free-kick before substitute Mauricio Pinilla made it 4-2 two minutes from time.
New Inter striker Diego Forlan reduced the deficit deep in stoppage time but Palermo held on for victory.
Diego Milito had put the visitors ahead in the 33rd minute before Miccoli equalised shortly after the re-start.
Milito added his second in the 51st minute from the penalty spot after Matias Silvestre had brought down Walter Samuel inside the area.
Inter's joy was shortlived as Abel Hernandez pulled his team level shortly after.
Inter coach Giampiero Gasperini fielded debutants Jonathan and Forlan in his line-up.
The hosts almost got off to the perfect start as Hernandez came close to giving them the lead but his powerful header was saved by Inter goalkeeper Julio Cesar.
Shortly after, Julio Cesar kept out Miccoli's 30-yard strike towards the far post.
Inter almost went ahead in the 26th minute, when Forlan fed Esteban Cambiasso inside the area and he struck wide.
But Inter's efforts were soon rewarded.
Palermo failed to clear a corner kick and the ball fell to Milito who made no mistake from close range and fired past Palermo goalkeeper Alexandros Tzorvas.
Inter continued to press forward and could have doubled their lead on the stroke of half-time had Forlan's eight-yard shot not gone wide.
Instead, the hosts pulled level two minutes into the second half.
Edgar Barreto spotted Miccoli inside the area and he fired past Julio Cesar to bring La Favorita to life.
Inter soon moved back in front as the referee pointed to the spot after Samuel fell inside the area.
Milito sent the goalkeeper the other way, firing the ball in right down the middle.
But Palermo reacted and made it 2-2 two minutes later.
Miccoli set up Hernandez and his right-footed shot from the heart of the area beat Julio Cesar.
Both teams wasted chances and with the game looking to be heading for a draw, Miccoli came to the rescue.
The veteran striker's curled free-kick stunned Julio Cesar four minutes from time.
Two minutes later, Pinilla's bullet-strike from outside the area doubled the advantage.
Last season's runners-up did not give up and cut the gap to 4-3.
Substitute Wesley Sneijder crossed the ball to Forlan, who fired from the six-yard box past Tzorvas for his first goal in Serie A.
But they suffered defeat in a thrilling opener.