The Gavin & Stacey star made the rare comments about the couple after being asked if he had been to their home.
WATCH: Inside Prince Harry and Meghan Markle's stunning Montecito home
"I have. It was more a sort of play date scenario [with the kids]. Then we went out for dinner after, it was lovely," he told the Sun.
He also confessed he had "a huge amount of respect and admiration for Harry" before adding that he will "always be in their corner".
James and Julia attended the couple's royal wedding in 2018
The presenter and Julia are parents to three children: Max, 11, Carey, seven, and four-year-old Charlotte, while Prince Harry and Meghan share Archie, three, and Lilibet, who turns 1 next month.
Sadly for the Duke and Duchess, their playdates could come to an end next year as James recently revealed that he will be leaving the The Late Late Show, which he has hosted since 2015, next year. It is thought the actor wants to move back to the UK with his family.
James and Harry have a good relationship – so much so, that the royal appeared on his show last year.
Harry and Meghan recently celebrated Archie's third birthday
In the interview, which was aired a month before Harry and Meghan's interview with Oprah, Harry got candid about family life, revealing Archie's first word – crocodile – and the unique gift the Queen sent his nearly two-year-old for Christmas – a waffle maker.
"My son is now over a year and a half, he is hysterical, he's got the most amazing personality, he's already putting two, three words together, he's already singing songs. His first word was 'crocodile', three syllables," he told James at the time, when asked about fatherhood.
He later added: "Interestingly, my grandmother asked us what Archie wanted for Christmas and Meg said a waffle maker, she sent us a waffle maker for Archie so breakfast now, Meg makes up a beautiful organic mix, in the waffle maker, flip it, out it comes, he loves it. Archie literally wakes up in the morning and says 'waffle'."
Make sure you never miss a ROYAL story! Sign up to our newsletter to get all of our celebrity and royal news delivered directly to your inbox.