But the Sweden international concedes that Albion and the Blues have both enjoyed success in the 21 months since Di Matteo left the Hawthorns.
Di Matteo has led Chelsea to Champions League and FA Cup glory while the Baggies have become established in the Barclays Premier League under Roy Hodgson and now former Chelsea coach Steve Clarke.
Olsson, part of the Swedish side which defeated England in mid-week, told Press Association Sport: "I rate Roberto Di Matteo very highly.
"I was very clear when the club sacked him that I wasn't (in favour of it). I really enjoyed working with him.
"He is a good coach and I really connected with him as a person as well.
"But everything happens for a reason I guess and, for him, it turned out very good and, for us, it turned out well.
"We got Roy Hodgson as manager and really pushed on from there and Roberto went to Chelsea and had a fantastic career there.
"It will be good to see him but, when it is game time, we will be looking to keep going and keep doing what we've been doing this season."
Regarding Di Matteo's success at Stamford Bridge, Olsson said: "I'm not surprised he has done well. I think everyone is surprised how well he has done.
"He has won the FA Cup, the Champions League, and is doing well this year. I didn't think anyone, not himself either, thought he was going to do that well.
"But I'm not surprised he has come in and done well because I know how big a name he is there and how well respected he is at the club. My hat goes off to him. It will be special Saturday.
"I remember last year we played them and we beat them and, after the game, Andre Villas-Boas was sacked.
"For Roberto, it will be special to come to the Hawthorns again. Hopefully he will get a good reception."
Di Matteo insisted he bore no grudges over his controversial dismissal in February 2011, which has ended up working out spectacularly well both for the Baggies and him personally.
The Italian lost his job after a lengthy run of bad results, although he had just led them back into the Barclays Premier League at the first attempt and they were not even in the relegation zone.
However, replacement Roy Hodgson did turn their campaign around - ultimately earning him the England job - while Di Matteo's subsequent adventures are now the stuff of legend.
His astonishing Champions League and FA Cup triumphs with Chelsea last season led to him being appointed manager full time and gave him licence to be philosophical about his Hawthorns demise.
"Look at where I'm sitting now, so I'm not so sure it was a bad thing," said Di Matteo ahead of tomorrow's [Saturday] Barclays Premier League game between the sides.
The 42-year-old added: "That's football. That's life.
"In a manager's career, you're going to get the sack at some point if you have a long career.
"It's happened to everybody in the industry, and it makes you stronger. It doesn't mean you're not a good coach or a good manager.
"There are many examples of good managers where it didn't work out with a club. They were successful before and were again after."
Asked if he held any grudges over his dismissal, he said: "No grudges, no. Not at all."
"I had a great time. I worked with a lot of good people there.
"The supporters were very kind to me as well. I had almost two years there.
"Despite the fact that last season's game was not a good memory, because we lost, I can reflect on good memories there."
The game Di Matteo mentioned saw West Brom seal his fate for a second time, but it worked in his favour that time as he was promoted from assistant manager at Chelsea to caretaker boss, with Andre Villas-Boas the fall guy.
Di Matteo said of the March upheaval: "That defeat led to a reaction. It was a very difficult time we found ourselves in.
"I was asked to fill in until the end of the season, and I was very pleased to help the club."
Di Matteo masterminded the greatest climax to a season in Chelsea's history but he could find himself under real pressure if they fail to win tomorrow [Saturday] after three league games without a victory.
He said: "For us, every week and every game is a match that defines our path, really. Last week the game against Shakhtar [Donetsk] was a defining game.
"This one now, in the league against West Brom, then the Champions League and Manchester City.
"Every game we play can define our season, and the games after too."
Clarke worked under Jose Mourinho at Stamford Bridge but has no regrets about branching out and eventually becoming number one at Albion this season.
He said: "It would be wrong to say it is just another game because I had a great association with Chelsea.
"Nobody can take that away from me. It was good times. But I've moved on and I have to do the job now for Albion.
"I could probably still have been at Chelsea now if I'd chosen to take that path.
"I decided to take away the safety net to challenge myself and push myself. It's a decision I made a number of years ago and I'm comfortable with that decision."
Clarke is mindful of not getting carried away with West Brom's current success.
He said: "The supporters have to enjoy this moment because they know, the same as I know, there will be days over the coming months that won't be so good, and weeks that won't be so good.
"This game is a good benchmark for us. It's not dissimilar to when Manchester City came a couple of weeks ago and we wanted to challenge ourselves against them.
"We lost the game but we came out of the game with a higher self-esteem, if you like, because we knew we'd given them a really tough game.
"We'll aim to do that again against Chelsea on Saturdays - but this time we're looking for a positive result."