Bayer have won just one of their last 13 games and Friday night's 2-1 loss away at struggling Hamburg proved to be the final straw for the former Liverpool defender.
Youth coach Sascha Lewandowski, who worked alongside Hyypia as co-manager last season, will take over at the BayArena until the end of the season.
A statement on www.bayer04.de read: "Bayer 04 has parted with team boss Sami Hyypia with immediate effect.
"After yesterday's 2-1 defeat in the Bundesliga away game at Hamburg we came to the conclusion that we can no longer guarantee qualification for an international competition under the direction of the Finn.
"Sascha Lewandowski takes over the office of head coach until the end of the season.
"The youth coach had led the work in the previous season together with Hyypia as we qualified for the Champions League."
Leverkusen currently occupy the final Champions League place in the Bundesliga but results over the weekend could see them drop to sixth and put them in danger of failing to qualify for the Europa League.
Hyypia was appointed alongside Lewandowski as caretaker manager in April 2012 after the exit of Robin Dutt and the duo were handed the job permanently six weeks later.
They guided the club to a third-placed league finish and qualification for the Champions League in their first season in charge while exiting the Europa League in the last-32 against Benfica.
Lewandowski returned to his old post as youth coach the following summer and while in sole charge Hyypia guided Bayer to just two losses in their opening 15 games league games this season.
They also made it to the last-16 of the Champions League but an extra-time loss in the DFB Pokal to second-tier Kaiserslautern was the start of a poor run of form that saw their title hopes ended.
A 6-1 aggregate loss to Paris St Germain also saw their European hopes quashed and the club have now decided that change is needed to arrest the slump.
"This decision was more than difficult because Sami has done a great job over the past two years," said sporting director Rudi Voller .
"What he, his coaching team and the first-team achieved was well above expectations and earned him praise from all sides.
"Therefore it was all the more painful for us in the current slump. With the dramatic developments of recent weeks we finally had no other choice than to bring about the separation now.
"We want and need to do everything to try to save the season."
He added: "Sascha Lewandowski knows the team inside out, he knows the structures of the club and therefore does not need a long time to settle in.
"We think this is a great advantage in our current situation and that appropriate measures can be taken quickly enough to steady the ship.
"Sascha has proven that he is a top trainer and helped us before in a critical situation after the failure of Robin Dutt together with Sami Hyypia.
"We firmly believe that he will succeed again."
Lewandowski admitted that taking over would be difficult considering his relationship with Hyypia but also stressed it was a call he could not refuse.
"It's not an easy situation for me, now taking Sami's place," he said.
"We jointly carried the responsibility for this team for more than a year so I regret that Sami's hopes could not be realised.
"But as an employee of the club, I also have a responsibility to meet in my current job .
"We will do everything to still bring this season to a successful end."