Spurs manager Andre Villas-Boas has found himself at the centre of speculation over his future in recent weeks despite the north London club sitting in sixth place in the Barclays Premier League table and just three points adrift of the top four following their midweek victory at struggling Fulham.
Bale's £85.3milllion switch to Real Madrid during the summer left Villas-Boas with a huge gap to plug and the club has invested the proceeds in an attempt to do that and more.
However, former Spurs midfielder Poyet has urged the manager's critics to give him time as he attempts to mould a new team.
Asked if Bale's departure had made all the difference, the 46-year-old Black Cats boss, who will welcome his old club to the Stadium of Light on Saturday evening, said: "In one game or two, yes, it can make a difference. In 38, no, I don't think it's possible to say that.
"In the last year and a half, Tottenham in some games did not play very well and they won because Gareth came up with something unique.
"Now it's a totally different situation. Spurs have got plenty of options, plenty of new players who need to adapt to football here and to the way they play.
"Trying to keep everybody happy and playing too many games and a few injuries, it's not easy. It looks easy from outside, but it's very difficult.
"It's really strange that you change so many players and the players gel together very quickly. Most of the time, you need six or eight months to make it work."
Such was the pressure under which Villa-Boas found himself that reports last weekend suggested a list of potential successors had been drawn up.
One claimed Poyet, who made 98 appearances and scored 23 goals for the club before returning as a coach, was among the prospective candidates, although that is something he took with a pinch of salt.
He said: "I am not surprised by anything. Every day, there is going to be something in the Press about someone, especially when you have played there.
"But I am not bothered, I am just concentrating on Sunderland."
The Black Cats' focus is simply on dragging themselves out of relegation trouble with Wednesday night's 4-3 home defeat by Chelsea leaving them rooted to the foot of the table and five points adrift of safety.
Poyet had admitted the time is fast approaching where they will have to take risks, and he has not ruled out employing a twin strike-force of Jozy Altidore and Steven Fletcher, something he tried briefly during a disastrous 1-0 defeat at Hull in which Lee Cattermole and Andrea Dossena were both sent off before half-time.
He said: "It's an option, yes. We have done it once - we tried the same away from home and it didn't work in that crazy game where we had the two players sent off.
"At the right time against the right rivals, maybe it will be possible."