The Blues have spent the past six days embroiled in the controversy over inappropriate comments referee Clattenburg is alleged to have made to Blues midfielder John Obi Mikel during Sunday's 3-2 defeat to Manchester United.
Chelsea looked set to end a difficult week with a win thanks to Victor Moses' 61st minute header, but were denied by Pablo Hernandez's late equaliser, which also ensured Di Matteo's side surrendered top spot in the Premier League table to Manchester United.
But the Italian does not feel off-field events were a factor in his side's failure to win.
He said: "I wouldn't read anything into what has happened in the last week and the result today [Saturday]. It is very difficult to win away from home in the Premier League.
"This is our second away draw of the season - we have won the rest of the games. It is disappointing not to have hung on to our lead but even when you are not brilliant, you need to get a result like that.
"It's a long season and it is going to stay very tight at the top. There are a few teams fighting for that.
"We are in touch and we are going to show how strong we are in the future."
Di Matteo was pleased to return to talking about football, particularly with the Clattenburg allegations following so closely on the back of the John Terry racism scandal; the England defender served the last of his four-game FA suspension on Saturday.
And he praised the performance of referee Kevin Friend.
"Talking about the football is how it's supposed to be week in, week out. Everyone would like it to be this way, including me," he said.
"I've got to say the officials were very good. Kevin Friend did very well in the game to keep both teams calm and let the game flow.
"He didn't have any influence on the game. I congratulated him after the game because they were very good.
"I didn't speak to the players before the game about the referee. Generally our team is very good (with officials).
"There has to be something really out of the ordinary for them to react. In general, we are a fair team."
But Di Matteo did acknowledge the two thrilling clashes with Manchester United - the league fixture was followed by the nine-goal Capital One Cup tie in midweek - may have hurt his side.
He said: "We have certainly felt (tired) this week after the two games with Manchester United, including extra-time.
"It would have been great to finish off the week with a 1-0 victory. The disappointment is we have conceded a late equaliser.
"I thought we had more of the game and clear chances than Swansea.
"When the opposition equalises so late on then you tend to think it's a bit lucky.
"To be honest, they didn't have any real clear chances, but they scored a quality goal."
Swansea merited a point after having the better of the opening half and playing with a swagger reminiscent of their best form of last season, although they struggled to break down a Chelsea rearguard well-marshalled by Branislav Ivanovic and Gary Cahill.
Hernandez's low strike capped a good week for the Swans, who were narrowly defeated by Manchester City before dumping Liverpool out of the League Cup at Anfield.
He said: "I think it was a deserved point. It was, in many ways, a copy of the game we played last week against Manchester City.
"That time we deserved a point but didn't get anything, we did this time.
"They had a few goal chances, we had a couple, two or three half chances. There were not that many chances. A point for each was a fair result.
"Their goal was a bit lucky but we came back, got some fresh people on the pitch and really put them under pressure. I think we deserved it. It was a great goal by Pablo.
"It has been an important week for the club. we have had three different results the last week but three very good performances.
"It's nice to see that we can play like that against the big teams in our way of playing. That's important for us."