The 48-year-old has paid for a run of two points from four games which means the Scots - bottom of Group A - have little chance of reaching their first finals in 16 years.
A seven-man Scottish Football Association panel has been considering Levein's future since late last week, and the former Hearts and Dundee United manager will not now be in charge for this month's game against Luxembourg.
That friendly, against opponents ranked 142nd by FIFA, would represent a gentle start if a seventh permanent manager since the turn of the century is found in time.
Gordon Strachan is a clear early favourite with bookmakers Sky Bet, while former bosses Walter Smith and Alex McLeish are also prominent in the betting.
Levein, who succeeded George Burley in December 2009, leaves with a record of 10 wins, nine defeats and five draws from his 24 games.
Several senior players had spoken out in support of the manager before his departure, which was met with disbelief by midfielder Charlie Adam.
He tweeted: "On my way back from training to hear Craig Levein has gone. Unbelievable. Can't believe the Scottish papers got what they wanted."
The SFA stressed they would "honour Craig's contract" after Levein pleaded with chief executive Stewart Regan and president Campbell Ogilvie to be allowed to stay on until the end of the World Cup campaign.
In a statement, Regan said: "We are taking this decision with real sadness.
"Craig has worked hard to bring success, and has been thoroughly professional in his approach to the job.
"He has achieved a great deal with the team and in other areas, such as the performance strategy. However, he would be the first to agree that football is a results-driven business.
"For that reason we have relieved Craig of his duties with immediate effect.
"Billy Stark will take interim charge of the team for the friendly against Luxembourg. There will be a short delay in announcing that squad, which Billy will select, and we have informed the Luxembourg FA of this development.
"After the Luxembourg match, the board will meet to begin the process of identifying and recruiting the next national coach to take us forward. We will give ourselves time, given our fixture schedule, to find the right person for the job.
"The challenge for the squad now is to improve our position in Group A and, subsequently, improve our position in the FIFA rankings.
"We also must begin preparations for the UEFA European Championships in France in 2016, which for the first time will have 24 qualification places available."