The result left Rangers four points clear at the top of the Clydesdale Bank Premier League.
When the dust had settled after one of the more enthralling yet trouble-free encounters between the Glasgow giants, Lennon revealed his side's second-half performance, where they managed to turn a 2-1 advantage into a 4-2 deficit, had left him "angry".
Steven Naismith helped himself to a double, the first coming with a wonderful drive midway through the first half after Celtic's Old Firm debutant Kelvin Wilson had failed to clear his penalty area properly.
The visitors responded positively and goals from Gary Hooper and Badr El-Kaddouri, whose long-distance shot Rangers goalkeeper Allan McGregor somehow managed to let in, gave them the interval lead.
However, goals from Nikica Jelavic, Kyle Lafferty and Naismith after the break widened the gap between the two teams with Celtic's troubles compounded by the dismissal of Charlie Mulgrew, who was sent off in the 75th minute after picking up the second of his two yellow cards for a foul on Steven Davis.
With a shock home defeat to St Johnstone in August still nagging away at him, Lennon must get his players back on track for the Scottish Communities League Cup third-round tie at Ross County on Wednesday night.
"It is a concern because we have lost two games already this season," he said.
"It's not good enough. We got our tails in front yesterday [Sunday] and looked okay and composed and we didn't show any of that in the second half.
"Obviously we lost our discipline by getting a player sent off and we lost our positional discipline at times which was very disappointing from my point of view."
Lennon, who revealed skipper Scott Brown would be rested until a solution to the ongoing ankle problem which led to him coming off in the second half was found, was at a loss to explain why his side started the second half so timidly.
The former Celtic skipper said: "It is frustrating. We were pleased to be in front at half-time, that was a bonus, but I would have been pleased even if it had been 1-1 because I felt comfortable.
"We made the players very wary of what to expect at the start of the second half but we didn't deal with it very well.
"I felt if we could have stayed in front for 10 or 15 minutes of the second half we could probably have caused them a few more problems at the other end but we never got out of the blocks in the second half."