In a statement released by ITV, Caroline wrote: "I have had a brilliant time working on The X Factor over the last few years, and hosting the main show was just fantastic, I made some amazing friends."
"I'm looking forward to carrying on working with the team over at ITV," she added.
"Hosting the main show was just fantastic," said Caroline Flack
Although it is not yet clear what kind of projects Caroline will be working on, Peter Davey, commissioning editor ITV Entertainment, also said: "We thank Caroline and Olly for all their hard work on The X Factor and the contribution that they've made over the past few years.
"We look forward to working with Caroline again soon and wish Olly the very best as he returns to concentrate on his music career."
As well as hosting The X Factor with Olly, Caroline previously hosted its spin-off show The Xtra Factor for three years from 2011 until 2013.
Olly Murs announced his departure just hours before
The Troublemaker singer told The Sun: "This was an incredibly hard decision to make and one I didn't take lightly as I've really enjoyed co-hosting The X Factor. It's been one big adventure and I loved every minute of it. But my heart belongs to music and I want to concentrate 100 per cent on making my next album and getting back to performing."
He added: "Last November I was doing the show, promoting a new single and I felt increasingly like the music side of things had to be shoehorned into the TV schedule."
Nick Grimshaw has also stepped down as a judge
It has been speculated that Olly was given the chance to "walk before he was pushed". A source told the Sunday Mirror: "Simon likes Olly and doesn't want him to go under a cloud so he's being given the chance to walk before he's pushed."
Last week Nick, who sat on The X Factor panel for one year, confirmed his departure. A spokesperson for the 31-year-old said: "He loved working on last year's series alongside Rita, Cheryl and Simon but it was very much a once-in-a-lifetime experience. He remains a massive fan of the show and is looking forward to watching it this year."