Hazard was sent off by referee Chris Foy in the second half after he appeared to try and kick the ball from underneath a ball boy who was laid prone on the ground.
Chelsea players reacted angrily when Hazard saw red, believing he had tried to get the ball loose to re-start play as the Blues failed to make any inroads on Swansea's 2-0 advantage from the first leg.
"Demba Ba told me the ball boy held on to it," Williams told Sky Sports 1.
"I saw him kick him in the rib and you can't do that to a young boy."
Williams preferred to focus on Wednesday night's result which secured Swansea a final date with League Two Bradford at Wembley next month.
"At times tonight [Wednesday] it was like a bar fight especially when they were pushing on," he said.
"The team defended brilliantly. That's what we worked on. We wanted to bring them back here and score a goal and keep a clean sheet. We didn't score but we kept a clean sheet."
Swansea will play in their first major cup final in their centenary year.
"It's unbelievable tonight," midfielder Leon Britton told Sky Sports 1.
"To do it over two legs against a team like Chelsea - tonight we made history.
"The club has come on leaps and bounds over the past 10 years."
Watch: Hazard kicks ball-boy in bizarre incident... More Videos
Swansea manager Michael Laudrup said he expected Hazard would regret the incident when he views it on television.
"I saw it from a very big distance from the bench," he said.
"I've since seen it after the game. Hazard is a great player. When he sees the images I think he will regret it."
Laudrup denied the club had given the ball boys instructions to hold onto the ball when it went out of play.
"No, no. Definitely not," he said.
"I think he was pushed and on top of the ball. I can understand all the frustrations when you are behind."
The Dane described the upcoming final against Bradford - whose giantkilling run has seen them beat Premier League opposition in Wigan, Arsenal and Aston Villa - as a fairytale for both clubs.
"Nobody had thoughts about finals. Now we are there," he said.
"This final is between the small fairytale and the huge fairytale. If we are the small fairytale what can you say about Bradford?"
Laudrup, in his first season at the Liberty Stadium, was full of praise for his players as they heaped more pressure on Chelsea's interim boss Rafael Benitez, who has now gone out of three competitions since taking charge.
"What can I say. Over two legs we just beat the European champions from last season without conceding a goal," Laudrup said.
"It was a game I expected. We had possession but they had moments where we had to defend deep.
"We came well into the game. We had the first two good chances with Wayne Routledge and the best one for Michu which Cech saved.
"That gave me the confidence that we were on the right track."
Benitez said that Hazard and the ball boy had exchanged apologies in the dressing room after the match.
"They apologised to each other," he said.
"They both talked. The boy knows he was wasting time. Hazard was frustrated and trying to get the ball back.
"We can not change the situation. We are disappointed because we lost a player.
"The best thing for us is to move forward."
Speaking about the match, Benitez added: "Again I will say we have to manage the game.
"They managed the game better and they are in the final."
South Wales Police confirmed it intended to investigate the incident.
In a statement Chief Superintendent Josh Jones, of South Wales police, said: "South Wales is aware of the incident involving a ball boy at the Liberty Stadium in Swansea tonight [Wednesday].
"We are making arrangements for him to be interviewed. South Wales Police takes such matters very seriously."