Williams, who was sacked by the ex-world number one during the summer after 13 years together and is now caddy for Australian Adam Scott, disparaged Woods at a caddies awards dinner in Shanghai.
He had told the audience that the aim of his celebration when Scott won a world championship in August was "to shove it right up that black a*******'".
Williams has since issued an apology on his website while admitting the comments "could be construed as racist", but Sabby Dhalu, Secretary of One Society Many Cultures, believes the golf authorities need to take action against the 47-year-old New Zealander.
Dhalu told Press Association Sport: "Obviously we welcome the apology but I would say that more sterner action is needed. I think he should face some disciplinary action.
"I think there should be an investigation into his behaviour and I don't think an apology is enough. The only way to root it out and prevent other people from behaving in the same way is to take sterner action, in the form of disciplinary action and I think an investigation should be conducted."
The WGC-HSBC Champions tournament organisers in Shanghai decided not to involve themselves in what Williams said.
Press officer Michele Mair said: "This was a private function. There will not be any further comment on this matter.''
There has been no reaction either from European Tour officials, although it is their policy not to discuss any disciplinary action taken.
The incident involving Williams comes at a time when England soccer captain John Terry is under investigation for an alleged racial slur after clashing with QPR defender Anton Ferdinand, and less than a month since Manchester United's Patrice Evra accused Liverpool striker Luis Suarez of racially abusing him.
Dhalu added: "This recent event (Williams) and the situation with Anton Ferdinand and John Terry shows that we clearly can't be complacent in terms of the need to combat racism in sport.
"Obviously we've made huge progress in the last 30-40 years or so in combating racism, certainly racism in sport is nowhere near as bad as it was a few decades ago but nevertheless it still exists and we cannot be complacent and we must do more to root it out in sport and the whole of society, because to a certain extent racism that is expressed in sport is a reflection of racism that still exists in the whole of society."
Williams' apology appears good enough - for the time being at least - to keep him in his current job with Scott, who described it as "not as issue for me" and also said he did not think Williams is a racist.
Scott said: "Steve issued a statement and apologised and he did the right thing. That's all there is to say about that from my side of things.
"It's not an issue for me. I think everything in that room last night was all in good spirits and a bit of fun, and I think it probably got taken out of that room in the wrong context.
"Anything with Tiger involved is a story. I value Steve's contribution to my game and while he's caddying I hope he can caddy for me.
"There was a lot of language used last night and it's just this was reported.
"I don't really think that stuff has ever left the room before and it's probably good reasons why. I think it's probably all very unnecessary."
Asked if he thought Williams was a racist, he replied: "I don't think so. I think we all know that's not the case.''
Woods is already in Australia and was told about the remark.
His agent Mark Steinberg was reported as saying: "If multiple reports, which all seem to be accurate, are true, then it's sad it's come down to this.
"It's a regrettable comment and there's really nothing that Tiger can do or say. He's just going to move on.''
Woods had Williams on his bag for the last 13 of his 14 major victories between 1999 and 2008, but dispensed with his services before his return from injury three months ago.
Williams took great delight when Scott won the week of Woods' comeback, calling it the "best win of my career''.
Although comments at the caddy awards dinner are normally regarded as off the record, what Williams said shocked many of those present and the story came out.
Northern Ireland's Graeme McDowell was present when Williams spoke and said: "The comments were surprising, yes.
"I'm aware that he's released a statement this morning apologising. I would doubt severely if he meant it racially like it came over and, you know, hopefully his apology will in some shape or form settle the matter.
"We don't want something made out of nothing. It was a fun night and we really hope that it doesn't become too big of an issue.
"These are racially sensitive times, especially in sport. It's unfortunate because it was a very sticky situation.
"I don't think Stevie Williams was trying to be racial. I don't think it was a racial comment. I think he was trying to be funny and make a joke of it.
"It was an embarrassing situation that he was put in. He was up in front of his peers and colleagues and it came out wrong.''
It is not the first time Williams has overstepped the mark in a public address.
Three years ago at a dinner in his home country he made a disparaging remark about Phil Mickelson and Woods got him to apologise.