Last year the Blues became the first defending champions in Champions League history to crash out at the group stage.
But even though they've been consigned to the second tier of European competition, Azpilicueta, who was impressive in Chelsea's 4-1 win over Wigan on Saturday believes the Stamford Bridge outfit will go all out to win it.
He told TalkSPORT: "We want to play every few days because it means we are still in competitions.
"It is very important for the team to keep going in the Europa League and the FA Cup.
"It's not the same [as the Champions League], it's true, because the Champions League is the top but the Europa League is a big competition and we have the chance to win it.
"Never in the history of Chelsea have we won this cup and we are going to try and win it."
European Cup-winning manager Roberto Di Matteo was replaced by interim boss Rafael Benitez when Chelsea failed to advance from the Champions League group stages.
Chelsea have struggled for positive results in recent weeks, including drawing at npower League One Brentford, resulting in this week's replay.
Azpilicueta, though, is confident Benitez's methods are working.
"The first months I was here we had Roberto [Di Matteo] with us and now Rafa came here and we are working very hard with him," added Azpilicueta, who won his first cap for Spain last week.
"We are trying to do the things that he asks and I think that we are playing well and doing the right things.
"When we have lost we have done a good game but when you make mistakes you pay.
"When you don't win it is difficult but hopefully we now recover the smile that is important for the team and for the future."
Meanwhile, boss Rafael Benitez was pleased to see some of his players return from injury and suspensions for the upcoming clash against Sparta Prague.
"Before we didn't have these options in some positions. Now it's much better because you can manage the game in a different way.
"In the past we didn't have these kind of options and it was more difficult to change the game."
John Terry, who missed the Wigan match with a knee injury, is uncertain for Thursday's trip to the Czech Republic.
Demba Ba is ineligible for the continental competition, while Nigeria duo Victor Moses and John Obi Mikel could come into contention following the completion of the African Nations Cup.