The Swans had been seen as slight favourites as the two sides met for the first time in the top flight, but Steven Caulker's 62nd-minute header gave Cardiff the bragging rights and moved the Bluebirds above their rivals in the Barclays Premier League table.
Swansea started strongly, but faded as the game went on with Michu and Alejandro Pozuelo failing to convert their best chances.
Williams admitted his side had been some way below their best, nd was frustrated to see them concede from a set-piece after Caulker got the better of Chico Flores to head into the corner, with Jonjo Shelvey having wandered off the post he was guarding.
"We did not play well, we did not get into our rhythm, we did not pass it well enough. We weren't solid enough off the ball, we didn't show enough heart, in my eyes," said the Wales international
He added: "It was bitterly disappointing. I don't think we played well, I don't think we showed enough desire.
"How many times have I spoken about defending set-pieces? It's getting ridiculous now.
"If we can't defend set-pieces we don't deserve to win the game and we didn't again today.
"We have spoken about it so much, it looked like we had tightened up.
"I don't think they would have scored today if it hadn't come from a set-piece, I don't remember Michel (Vorm) having a save to make.
"(At set-pieces) you've got your man, I know my man, it's me against him, he doesn't score against me. If he does it's my fault, it's that simple."