The Take That member, who has been a judge on the ITV singing competition for the past three years, chose to announce his news on Saturday night's episode.
Gary revealed his departure close to the end of the show after one of his groups, Rough Copy, gave a memorable performance of In The Air Tonight.
"I'm so glad for my last year on the show, I get to mentor you," Gary said, praising the three-piece band.
The 42-year-old star has previously denied rumours that he would quit the show.
Speaking at the launch of the series, Gary dismissed the speculation. "It's far too early to think of next year," he told HELLO! Online. "It's all about this year."
It looks like the Take That star is leaving his mentoring role to focus on his music career. Gary delighted fans last month with the news that he is set to release his first solo album in 14 years, Since I Saw You Last.
"After 14 long years, I'm proud to announce my new solo album on November 25th! I'm excited for u to hear it. More info to follow #gbsolo," tweeted Gary.
The release date of his new album coincides with the current series of X Factor, so fans will no doubt be hoping to hear the first single performed during the live rounds, just as previous judges Cheryl Cole and Kelly Rowland have done to promote their singles.