The Dragons are finally starting to put together a string of positive results under Chris Coleman's guidance, and Friday's 2-1 victory over Scotland at Hampden Park was their third win in their last four games.
The sole defeat in that quartet of fixtures came against Croatia in October, as Luke Modric ran the show in a comfortable 2-0 win for Igor Stimac's side.
The win over the Scots moved Wales up to third place in Group A at the halfway stage of the campaign, although they trail pacesetters Croatia and Belgium by seven points.
A play-off place may yet be achievable for Coleman and company, but Williams believes that possibility will evaporate if they cannot claim all three points at the Liberty Stadium.
He said: "We want to win the game first and foremost and see where we are after that.
"But we are seven points off them so if we want to do anything in the group we need to win this game, it's that important.
"We think we can win it, obviously we are trying to catch them and we want the points.
"They are doing well in the group, they are battling it out with Belgium at the top and they will want the points as much as we do.
"But we will have to see how it goes and see where we are after tomorrow night."
Wales will have Gareth Bale available after the Tottenham star trained yesterday and today, and he appears to be over the worst of the illness which saw him withdrawn at half-time in Glasgow, although Aaron Ramsey will miss out after his red card.
Wales made light of Bale's absence as they came from behind against Scotland, but Williams welcomes the 23-year-old's availability.
"He seems fine, he trained today and yesterday and he has been working closely with the doctors, getting his energy levels up and the right food down him," he said.
"He seems fine and I am sure he will be ready for the game.
"You always want him in the team, you could see he was struggling with illness in Scotland and he will hopefully be back to full fitness for Tuesday."
Williams also hopes their recent improvement has seen Wales turn a corner, and put them back on the right track they were following under Gary Speed.
He said: "The performance on Friday was one of the best I have been involved in with Wales, it was similar to the run we had under Gary.
"We were all proud of the achievement of winning at Hampden and the way we dug in after going a goal down right on half-time."