Magilton took control of Victory in January, but failed to turn the side's poor A-League form around as he guided them to just two wins from 12 matches with the two-time champions limping into eighth position.
The club said the 42-year-old would be one of the candidates to take the post for next season, but Magilton declined to be a part of the process and is returning home.
"On behalf of (assistant coach) Chris (O'Loughlin) and myself, I'd like to thank everyone at Melbourne Victory for the opportunity that was presented to us," Magilton said.
"We have thoroughly enjoyed the experience and wish the club every success in the future."
Managing director Richard Wilson was full of praise for Magilton and what he was able to achieve in a short period of time.
"Jim arrived at the shortest of notice, on a leap of faith, stepping into a role that he had limited background on," Wilson said.
"We realise that we didn't ultimately reach the finals this season, which was our goal, but we also believe Jim was unlucky not to have finished with a better record, given we conceded goals in the dying stages of games against Sydney and Gold Coast.
"He was a total professional throughout the three months he was here. His preparation week to week was exceptional. His team selection saw the integration of several youngsters and was always in the best interests of the club.
"In summary, he left no stone unturned, tackled every issue head on and he will only be remembered in a positive way from everyone involved in Melbourne Victory. We certainly wish Jim and Chris well in their future coaching endeavours."
The club have a shortlist of candidates and the process for appointment is under way, but an announcement is not expected for a number of weeks.