The 52-year-old, who hung up his whistle at the end of last season, reckons only eight of the current crop are good enough and believes referees' chief Mike Riley and his assistant Keren Barratt are part of the problem.
He told the Sun: "Don't get me wrong, there are some really top referees out there.
"I reckon currently there is a 'great eight', in no particular order: Mark Clattenburg, Howard Webb, Mike Dean, Andre Marriner, Lee Probert, Martin Atkinson, Phil Dowd and Chris Foy.
"But as for the other 10 on the elite list, well...some are just not up to it while others need more experience, better nurturing and coaching."
That puts referees including Mike Jones, Kevin Friend and Lee Mason on Halsey's list of those who are not good enough.
Halsey believes much of the blame lies with the body that oversees referees - Professional Game Match Officials Ltd (PGMOL) - as well as their general manager Riley and select group manager Barratt.
"[PGMOL] are running the whole show on an apparent power trip while creating an atmosphere of fear and paranoia," Halsey said.
"That is not helping the weaker officials improve and the system often confuses and frequently frustrates the best referees who get mixed messages about what is expected of them."
Halsey believes the wrong officials are often picked for big games and blames that on Riley and Barratt.
He added: "Riley had a reputation for being an over-fussy and pedantic ref. And his manner as general manager of elite referees - whereby he liaises with the Premier League and League Managers' Association - appears not to have changed."
Halsey described Barratt, and the dual role he plays as both their boss who draws up the match lists and also offer advice as a "mentor", as a "big problem".
He said: "How can he do that (be their mentor) when he wears another hat as head of the evaluation committee, which adjudicates and has the power to mark down referees, axe officials from matches or ultimately kick them off the list?
"There is a clear potential conflict of interest in the two roles."