Madrid looked like they would have to settle for a draw at the Bernabeu as they remained locked at 1-1 with Betis heading into the final stages.
However, summer signing Isco Alarcon popped up in the 86th minute to power home a header and earn Madrid the points.
New coach Ancelotti, taking charge of his first competitive match since replacing Jose Mourinho, believes Madrid were full value for the victory but admitted they were far from firing on all cylinders - especially at the back.
"I told the players that the first game is never easy. We had a good pre-season and we are excited. We didn't play well, but that's normal," he told reporters.
"The most important thing is to win the match, and I think we deserved to win because we created many opportunities to score."
He added: "We can improve. We've had some good combinations, especially out wide. The problem above all was defensive.
"It was a problem of balance. There was too much space between the lines. Betis played a good game. Our players have been playing with their national teams this week and in the physical aspect Betis were better."
Ancelotti caused a surprise before kick-off when he opted to name Diego Lopez in his starting XI in place of long-serving club captain Iker Casillas.
Mourinho's decision to replace Casillas with Lopez for the latter half of last season caused a massive stir but Ancelotti opted to continue with the latter.
Ancelotti said he made his choice on the day, explaining: "As I said, they are decisions based on minor details, but that's the decision I made.
"Diego Lopez was the goalkeeper for this match and we'll see for the next game. I spoke with Iker, who is very professional. He wants to play and that's good. Diego played and he was the goalkeeper today [Sunday]."
Meanwhile, the 21-year-old Isco was delighted to make a dream debut for the club, although he too admitted they can do better.
The 21-year-old, who joined from Malaga for a reported 27million euros this summer, said on his club's website: "We started with a win at home, which is important. It's the first goal I've scored from a header in La Liga and I am very happy to win three points.
"We were a little surprised by Betis, but it's the first match of La Liga, we need to improve but it was important to win".
"We need to work a bit harder, but we have a good feeling about Madrid for the rest of the season."