What's more, the famous dad even revealed that he and Holly ended up stealing from celebrity hotspot Nobu, although the story was left on a cliffhanger!
WATCH: Holly Willoughby and Phillip Schofield reduce This Morning viewers to tears
Firstly, Phillip explained that he considers Holly a "dangerous friend to have" at times.
Phillip recalled their lunch date on Instagram
To give a little context, the presenter told his Instagram followers that when he was little, his school teachers would write in his reports that little Phillip was "easily led".
The 58-year-old then said that Holly is at the top of the list of friends that he would "trust to lead me into mischief".
Phillip then recalled one particular lunch date with doting mum Holly, when the pair had dined at Nobu, and had so much to drink that they "staggered" home.
The pair are very close friends
The next day, the father-of-two discovered that he had "a full container of hand soap" in his pocket.
Shocked, he phoned Holly to tell her, and after checking her own coat pockets, Holly found that she, too, had returned home with hand soap!
Phillip then left the story on a cliffhanger, telling his followers: "Oh my god, we robbed Nobu."
The TV star's full story goes as follows: "Holly can be a dangerous friend to have. In my old school reports, some of my teachers would write that I was 'easily led'. In my adult life there are very few friends that I trust to lead me into mischief.
"But Holly is at the top of that group. We've enjoyed wonderful and outrageously drunken lunches where just occasionally we had the time to let our hair down and reflect on life.
"Once after a particularly 'long' lunch in Nobu, we parted and staggered off to our respective homes. The next day I phoned Holly because I couldn’t remember paying the restaurant bill. Holly said: 'Check your coat for the receipt.'
"When I put my hand in my pocket I felt a full container of hand soap. I was surprised, had I stolen it from the loos? I called Holly and she checked her bag, 'oh Christ I've got one too!' she yelped.
"Oh my god, we robbed Nobu."
For anyone as desperate to know how that story ends as we are, you'll have to get your hands on Phillip's autobiography, Life's What You Make It, on 15 October.
Like this story? Sign up to our newsletter to get other stories like this delivered straight to your inbox.