A meeting of the IOC's ruling committee looked at a number of sports with modern pentathlon viewed as being most at risk but the 15-strong board chose to drop wrestling.
It will now have to vie with other sports including baseball/softball, squash and karate to be added back into the programme when the full IOC session meets in September.
IOC communications director Mark Adams told a news conference in Lausanne: "The executive board recommended that wrestling not be included on the list of 25 core sports and it now joins shortlisted sports vying for inclusion on the 2020 Olympics programme as an additional sport."
Adams said the vote by the executive board had been done by secret ballot, and that IOC president Jacques Rogge had not taken part.
He added: "This is not what's wrong with wrestling, it's what's right about the other 25 core sports.
"This process is about looking to renew the Olympic Games and the executive board made their decision based on their experience.
"They have a collective intelligence there with representatives from many sports, national Olympic committees and international federations and they came to that decision."
The executive board studied a report from the IOC's programme commission which assessed each of the sports at last summer's London Games.
Taekwondo was also viewed as being possibly at risk with the report looking into more than 30 separate areas, including TV ratings, ticket sales, anti-doping and global popularity.
Wrestling will be one of eight sports now who will have to campaign for inclusion when the IOC session meets in Buenos Aires in September. Baseball/softball - the last sport dropped from the programme back in 2008 - is viewed as a front runner while squash has also been running a high-profile campaign.
The other sports pushing for inclusion are roller sports, sport climbing, wakeboarding and wushu. Golf and rugby sevens will be part of the programme for the 2016 Games in Rio de Janeiro after winning inclusion in 2010.
British Wrestling chief executive Colin Nicholson said he was "surprised and disappointed" by the IOC's decision.
Nicholson told Press Association Sport: "Wrestling proved very popular in London and it has a very wide appeal right across the world.
"In Britain it was beginning to grow from a very small base and has the support of Sport England, who have increased their grass-roots grant."
Nicholson said it was imperative the Great Britain programme pushed forward towards Rio 2016 regardless - and said the fact wrestling remains a Commonwealth sport gives it some consolation.
Nicholson added: "We will be looking to deliver success in 2016 where we have two athletes who we believe are genuine medal contenders.
"We are fortunate that we are a Commonwealth sport so our athletes will continue aspiring towards 2014 and 2018. In the meantime we will remain hopeful that the IOC may give wrestling another chance to remain part of the Games."