The reigning Serie A champions took the lead through an Alberto Aquilani header in the 12th minute but went on to concede an Edinson Cavani hat-trick.
Speaking after the match, Allegri expressed his disappointment.
"This was a very odd game," he told Sky Sport Italia. "We had a lot of possession and we were in the lead, but then they were a bit lucky to equalise almost immediately after.
"We should not have allowed them to attack on the counter, we knew they would play this way. I am happy with my players, we just lacked a bit of determination in some moments."
Milan are currently at the wrong end of the table, with just one point from two games, but Allegri confessed he is not concerned about his side's league situation.
"I am not worried, we know we are a good squad," the coach added. "We come off two good displays against Lazio and Napoli.
"It is true we have to improve because we have conceded five goals in two games."
Milan welcome Udinese to the San Siro next on 21 September.