Scolari takes on England in the first game of his second spell as Brazil manager on Wednesday.
The 64-year-old could have been sitting in the home dugout had history taken a different course. Six-and-a-half years ago the Football Association (FA) approached Scolari asking him if he was interested in taking over from Sven Goran-Eriksson after the 2006 World Cup.
The Brazilian, then contracted to Portugal, seemed interested initially, but he then pulled out of the running, citing media intrusion as the reason for his withdrawal.
Scolari gave the FA an immediate reminder of his talents when he knocked the Three Lions out of the World Cup in the quarter-finals in Germany, and he admits the decision to turn down the chance to succeed Eriksson still "hurts" to this day.
"Of course it hurts," said Scolari, who went on to manage Chelsea two years later.
"It hurts a lot because I would have loved to have been the manager of the England team. Who wouldn't? It's a wonderful national team.
"When I was invited to be national coach of England, when we sat down I still had a contract with the Portuguese national team and I was not willing to break that contract.
"It was my duty to fulfil it. Imagine what it would have been like when we played each other in 2006 and I had signed that contract with England."
Although England have found stability under Roy Hodgson, Scolari knows all about the unpredictable nature of international football.
Should the Three Lions be on the lookout for a new coach in the future, it appears Scolari would be interested in taking over the reigns.
"Who knows what will happen. One day, maybe," Scolari added with a smile.
"I wish all the best for the English players and their manager on their path to innovation."
Roman Abramovich appointed Scolari as Chelsea manager in 2008 with the hope that he could marry success with Samba football.
The former Portugal manager's reign started well, but his team suffered a mid-season collapse and he was fired by Chelsea's owner after just over six months in charge.
Abramovich's ruthless approach to dismissing managers has continued since, but Scolari on Tuesday refused to criticise the Russian.
"I had a marvellous time in London. I loved it," Scolari said.
"I hope I will have the opportunity tomorrow [Wednesday] to talk to Frank Lampard and Ashley Cole.
"What is happening right now (at Chelsea), I don't have any opinion on."
Talk of a "golden generation" was rife around the England team in 2006. Lampard, Steven Gerrard, John Terry and Rio Ferdinand were all in their prime when they lined up against Scolari's Portugal in Gelsenkirchen.
Yet those players, and their successors, have failed to live up to expectations ever since.
Scolari was therefore loathe to agree with the idea that England's new crop of players are capable of succeeding where their predecessors have failed.
"All the great national teams will always remain great nations teams, but they all have past times of innovation like England is currently going through," he said.
"That is why they might be in a difficult situation currently.
"But I do admire the pedigree amongst their players. And the Premier League gives them all the opportunity to have a great national team."
Scolari is under immense pressure to succeed following his re-appointment as head coach.
He won the 2002 World Cup with Brazil, knocking England out in the process, but the most successful international country of all time has struggled ever since.
Scolari hopes to bring the glory years back, however, and insists his men can win a sixth World Cup when the competition comes to Brazil next summer.
He said: "In 2013 we will have 12 or 13 games. It's important we get settled now. We want fans to get fascinated about our big goal, which is to be champions in 2014."
Chelsea defender David Luiz will have a test on his calf injury before Scolari decides whether he plays Wednesday's match.