The Cottagers are expected to name a strong line-up to face the Polish champions as they look to avoid a repeat of their 1-0 defeat in Poland.
The result at Stadion Miejski leaves the Cottagers second in Group K, three points behind Dutch outfit FC Twente and just a point ahead of both Odense BK and Thursday's opponents Wisla.
"You never want to lose and because the table is so tight we need to win," Dempsey said.
"We need to get points as you don't want to be waiting until the last game to get our of the group stage.
"We're going to do everything we can to win the game.
"While the league is most important, so too is this competition as you want to be a part of as many competitions as you can.
"To play in these games gives you great memories.
"I mean getting to the Europa League final the last time we were in it is something I will never forget and hopefully we can have a nice little run.
"It is nice to be able to play in this competition as it allows you to play different teams from all over [Europe]."
Dempsey was rested for the first match with Wisla, which was marred by Moussa Dembele's controversial sending off.
The incident occurred when an innocuous passage of play flared up in Poland, with Dembele pushing over Wisla's Gervasio Nunez after he caught him on the back of the ankle.
The Argentinian midfielder went down clutching his face despite being pushed at shoulder height, leading referee Martin Hansson to hand the Belgium international a straight red card.
Dempsey watched the match on television but, while annoyed by the decision, remained philosophical about it.
"It is frustrating to see but what can you do about it? The ref made a decision that he felt was right.
"Things, though, don't always happen the way you want and now we can only concentrate on Thursday's game.
"We need to take advantage of having the same amount of players as they do and play the football the way we know we can. Then, hopefully, we can get the right result."
Wisla's preparation for Thursday's match did not go smoothly, with visa problems delaying the arrival at the team hotel of Serbian trio Milan Jovanic, Marko Jovanovic and Ivica Iliev.
Coach Robert Maaskant told his club's official website ,www.wisla.krakow.pl: "We hope the problems with visas will be solved, and this trio will be able to play. We came here for a favorable outcome but it won't be easy."