"I'm missed and Dermot's missed – let's be honest!" he told ITV's Lorraine on Friday morning.
Louis Walsh says he's missed on The X Factor
"When you've got a new panel, it takes time for them to settle. I think Rita is good, she's funny," Louis explained. "Simon is good, he needs a few new lines. Cheryl is Cheryl."
Referring to his replacement Nick, he asked: "What is that other guy? What's his name?"
After being reminded of the BBC Radio One presenter's name Louis added: "Simon hired them, this is Simon's show. He hires and he fires so he has to make it right."
Louis was a judge on the show until 2014
As for whether Louis would ever return to the judging panel? He would want to be back alongside his friend and former judge Sharon Osbourne.
He explained: "It if was going to be fun like with Sharon, Kelly… they were the best, they were fun… you have to have fun."
The X Factor returned to ITV over the August bank holiday weekend, but reportedly had a disappointing launch with an average audience of 7.1 million viewers – the lowest audience for a launch show in a decade.
However Simon is confident that the show will continue to be a success and still beat Strictly Come Dancing in the ratings. Speaking at the new series launch night, he explained: "This show does feel better, the talent's great, they're interesting, so if it us versus Peter Andre, I'm going to put my money on us."