Ashley is constantly the brunt of abuse from a big section of disgruntled Toon fans, but their failure to sign a high-profile striker on deadline day has only heightened some of the anger.
After selling Andy Carroll for a club-record £35million fee to Liverpool in January, the club promised to re-invest and Pardew insisted he would be given the cash to spend.
And while Newcastle have made inroads in the transfer market, their spending has been way below what they have recouped in the last 12 months.
"It's difficult at Newcastle and I don't think the owner's relationship with the fans helps. That's obvious," said Pardew.
"I don't think there's anything I can say or do that will improve that. Hopefully over the next three-four years, if I bring some success to the club, maybe that can be looked at in a different light. But certainly not in the short term."
Only this weekend, Newcastle managing director Derek Llambias - another target for Toon fans' annoyance - insisted that Ashley would not be selling.
Pardew admits that he himself has clashed with Llambias, but he has also defended his immediate superior.
"I think Derek is the type to call a spade a spade and so am I. We've exchanged those words and we have to put it right and make sure we work together to move forward. That's where we are," he said.
"I think it's important that Derek hears from me the message from the fans. I made that perfectly clear to him, how important it was."
Pardew himself was a controversial appointment by the Ashley regime following Chris Hughton's sacking, but he insists he is happy with how things are going.
"In terms of where I am with the club, I've really enjoyed working here. I love the people, they've been great to me. Hopefully I can repay them with the squad I've got," he said.
"I would have loved to have the striker and filled the number nine jersey.
"We'd have had the complete picture then and it would have almost been too easy for Newcastle. That's how I've tried to look at it.
"I've not been helped by some of the key players that we've lost this summer," he admits.
"Some because of past histrionics with the board, some just had the desire to leave - like Jose Enrique, who was just determined to leave, there was nothing we could do. Andy Carroll was the same, to an extent.
"I've had difficult periods in that but I'm hoping it will smooth out now and that the players that really come to the fore to us, we manage to keep. We'll see."