Even when he was at school James, who has dyslexia and says he isn't academic, found himself being compared to his sporty and smart sisters.
"It's always been, whether at school or now, that I am Pippa and Catherine's little brother," James, 29, told Mail Online. "It's probably going to be the story of my life. [But] I am James Middleton. I am very proud to be 'the little brother of…' but, equally, they are proud that I am who I am."
"It's always been, whether at school or now, that I am Pippa and Catherine's little brother," said James
"My sisters were very successful and confident," he added. "It was, 'Oh, you are Pippa's brother! Are you Catherine's brother?' And so I was automatically put into the top sports teams and academic sets. But I was a bit of a squirt when I first started, so then it would be, 'Are you sure you're a Middleton?'"
The businessman went to boarding school like his older siblings, but found it hard to settle in at first, saying he spent the first year "feeling not particularly happy or confident".
Pippa, knowing this, would visit him at his boarding house with her friends. "It was great in hindsight, but [at the time] you think, 'How embarrassing,'" said James.
Pippa would comfort her younger brother when he found it difficult to settle in at boarding school
He was 23 when he became the brother-in-law to the future king, Prince William. And when William and Kate started dating, James was still at school and very conscious of the intense interest that surrounded the now royal couple.
"I used to be first up in the morning to make sure I could get to the papers and rip out any stories about my sister, so nobody would see them and there would be no conversations about them," said James.
Having royal connections can have its downfalls too. "When somebody is interested in me for other reasons, I can spot it a mile off," he said. "All I want to do in my life is be known as me and for what I do."
"Marriage is absolutely not something I'm scared of," he admitted
The entrepreneur, who has his own marshmallow company Boomf, has had his faire share of time in the spotlight too. He has been dating Donna Air for almost four years and is faced with the question of when will he marry her?
"We are thinking about where we are going next," admitted James. "Marriage is absolutely not something I'm scared of [but it] isn't necessarily the be all and end all. She makes me very happy [and] I think I make her very happy. I want children. I've said that before. I will have children, but you'll have to wait."