After the Blues had overturned a two-goal deficit, they were reduced to nine men when first Branislav Ivanovic was dismissed before Torres received a second yellow card for diving following Evans' challenge.
The decision prompted angry exchanges between managers Sir Alex Ferguson and Roberto Di Matteo before Javier Hernandez scored the winner for the visitors.
Asked whether there was any contact between himself and Torres, Evans told Sky Sports 1: "At the time I didn't feel like I'd made contact but I've seen it again and there was a bit of contact.
"I didn't feel like it was enough to bring him down.
"He probably realised he wasn't getting the ball and I think the referee's made the right decision in sending him off."
A David Luiz own goal and Robin van Persie's ninth strike of the season had put United firmly on top but Chelsea showed their courage to equalise through Juan Mata and Ramires.
However, the dismissals changed the face of the game and Hernandez scored the decisive goal in the 75th minute to end United's 10-year wait for a win at Stamford Bridge.
"I think that's probably the hardest game we've had this season, especially towards the end of the first half," Evans added.
"Chelsea really came at us and put the pressure on us and it was tough. We were probably lucky to see it out 2-1 at half-time.
"You've got to show character in games like this.
"We came out for the second half and you've got to stand up coming here to Stamford Bridge.
"It's a tough game, we knew we were going to be under pressure and they get the equaliser and the sendings off in-between that but we managed to finish it off in the end."
If Chelsea were feeling hard done by with the two dismissals, Chelsea were dealt another blow as replays showed Hernandez was in an offside position before scoring.
"We don't care. We win. We have the three points, that's the only thing on my mind," the Mexican said.
"For some people it will be be very controversial and for other people it won't.
"We're very happy because we get the three points."
Ferguson believes Torres had only himself to blame for being sent off.
"Did he intend to dive? I thought he went down," said the Scot.
"I think Jonny may have just caught him a little bit but he can either carry on running - which he could have done - but he chose to go down.
"He could have carried on and scored so I cannot understand it. It's his own fault."
Ivanovic's dismissal was less controversial, the Serbia international seeing red for bringing down Ashley Young, who had been played clean through by Van Persie.
And Ferguson admitted that was the turning point.
"You have to give credit to Chelsea, they never gave in," he added.
"They showed a great desire to get back in the game.
"I think the sending off from the right-back was the turning point. I think we wouldn't have won at that point.
"But we put Chicharito (Hernandez) on and he gets the winning goal and I think the momentum was with us then."
Di Matteo blasted the officiating as Chelsea suffered their first league defeat of the season and their second in a week following their midweek Champions League loss at Shakhtar Donetsk.
The Italian is adamant there was contact between Torres and Evans and believes the Manchester United defender should have been sent off as he denied the Spaniard a goalscoring opportunity.
"The key decisions from the referee today probably influenced the result," said Di Matteo.
"He sent off Fernando Torres and the third goal is offside and it clearly has decided the game.
"He's (Torres) put the ball between his legs and is through and gets a kick on his shin and he goes down.
"It's a free-kick for us and I think it should have been a sending off for Evans because he's through to the goal.
"Everything went for United and that really makes us unhappy."