The European champions were all over the place at the back in Wednesday night's dramatic Capital One Cup win over Manchester United and were hardly any better in Sunday's first league defeat of the season against the same opponents.
They have now conceded 11 goals in just four games since Terry began his domestic racism ban, although the Chelsea captain did play in one of them when he started their Champions League defeat at Shakhtar Donetsk.
Terry serves the final instalment of his ban, while manager Di Matteo revealed fellow defender David Luiz would almost certainly miss the Swansea game though injury.
That would mean Branislav Ivanovic being pressed into centre-back service in a new-look defence that would almost certainly see right-back Cesar Azpilicueta handed his full league debut, with Ashley Cole returning from injury on the opposite flank.
Some would argue any defence without the erratic Luiz would be an improvement but it is the absence of Terry that has been more cause for concern.
"He's obviously been an influential player for us, for sure, and a fantastic defender," Di Matteo said.
"When he plays, he's always played very well for us.
"The other players have done a fantastic job by coming in and playing when they've been in the team but, certainly, JT is a player that we miss."
Di Matteo added: "The last few games, we've conceded a few more goals than at the start of the season.
"Generally, it's been down probably to a few individual mistakes.
"But I think our track record is very good and, surely, will continue to be."
If not, Chelsea can surely forget about staying top of the Premier League and reaching the Champions League knockout stages.
"We have to improve but I don't think it's just as simple as that," Di Matteo said.
"You look at other teams and they're conceding goals, so we certainly have to improve, from the last few games, our defensive organisation."
It is not just at the back where Chelsea have problems at present.
Fernando Torres and Daniel Sturridge both look well short of being the kind of ruthless finisher that would ease the pressure on the defence, leaving them to rely more and more on their much-vaunted trio of Juan Mata, Oscar and Eden Hazard.
Mata's goals and assists, in particular, have been absolutely vital, so the news he would almost certainly join Luiz on the sidelines was another blow.
"A few others have a few niggling injuries as well, so we'll have to see for tomorrow [Saturday]," Di Matteo added.
Throw in the Mark Clattenburg controversy, and a match Chelsea might have been expected to win reasonably comfortably suddenly looks a much more difficult proposition.
But Di Matteo said: "We are okay.
"The team is playing well, despite one defeat that we had in the league."
Acknowledging the importance of Wednesday night's thrilling comeback, he added: "It was a great win for us, for the spirit of the group as well.
"Winning is always a good factor for the team and for the group.
"I think we showed how strong we are and that we can compete against any other team."
Under new boss Michael Laudrup, Swansea made an even better start to the season than Chelsea but have been inconsistent since.
Di Matteo said: "I think they're a bit more clinical now maybe compared to the last couple of seasons when they had a lot of possession but didn't really create a lot of chances.
"They go forward a little bit quicker and score a few number of goals more than maybe in the past.
"I think that's changed in the Swansea team."