Three days after thrashing Sevilla 7-3, Madrid looked to be on course for another goal-laden victory as they built a 3-0 lead by the 48th minute, Cristiano Ronaldo netting either side of a Karim Benzema header.
However, in the space of three second-half minutes Rayo scored twice from the penalty spot through Jonathan Viera to get back into the derby at 3-2 and, in the final half-hour, the home side came close to an equaliser on a number of occasions, including hitting the post through Viera.
Ancelotti said: "In the first half we scored two fantastic goals on the counter-attack and we enjoyed having control, but the second half was a disaster.
"Rayo have made it difficult for us because they've pressed well. In the first half we didn't really suffer, but in the second we did."
It was a similar story against Sevilla on Wednesday where Madrid were cruising at 3-0 after 32 minutes only to see their opponents pull two goals back in quick succession before the break.
Ancelotti warned his star-studded side they must not allow that to happen, saying: "We cannot think that when we are winning 3-0 the game is over, because any little detail can change things. It's happened to us twice and I'm sure that it won't happen again.
"Against Sevilla it could have been a coincidence, but today is the second time and it is not.
"We have to change the attitude that we have at times on the pitch, because a team of this quality cannot allow a game they are winning 3-0 to fall back into the balance.