Despite a man-of-the-match performance by Johan Elmander, who scored a brace on his full debut, the Canaries still leaked three goals on their way to reaching the third round of the competition.
Martin Olsson, Anthony Pilkington, Leroy Fer and Steven Whittaker were also on target for Norwich but it was the goals shipped to Shakers trio Anton Forrester, William Edjenguele and Jessy Reindorf that had Hughton rattled.
"I thought at 3-0 we were very comfortable and we had another very good chance to make it four," he said.
"With that comes some disappointment but any team from any division are not going to come here and lay down. Particularly on the occasion they are going to try and enjoy it as much as possible and give it a good go.
"If they give it a good go they have a chance to get goals and they did just that."
Having drawn at home to Everton and lost at Hull last weekend, Hughton reckons the Carrow Road faithful would have enjoyed the end-to-end nature of the clash with Sky Bet League Two outfit Bury.
"I don't think there would have been many supporters who would have been unhappy with that, they would have enjoyed it," he said.
"It was certainly an enjoyable game for the supporters and I have to say the quality of the goals was excellent. There were some really good goals, especially the last one for them."
Bury boss Kevin Blackwell was pleased with the effort his players showed on the night and felt Norwich stopper Mark Bunn deserved credit for limiting the visitors to three goals.
''The Norwich crowd gave everyone of our players a standing ovation at the end. It was a game and half and their goalkeeper has got man of the match,'' he said.
''He's pulled off save after save and we've hit the bar and the post and had two strong penalty shouts. I'm proud of the players. It was a good performance.
''How we played was key, some of the Norwich players said when they came off the pitch that we play some good stuff so I'm pleased with the players overall.''