It is the first time that the five-month-old baby boy will be in the same room as his extended family.
Prince William drove his wife Kate Middleton and son George to the gathering which is hosted by Her Majesty at her official residence.
The Duchess of Cambridge was festive and fashionable in a red coat with her hair loosely styled into waves.
William wore a navy blue suit with a blue shirt and a patterned tie.
Heavily pregnant Zara Phillips, who is due to give birth in early 2014, arrived in a black cab looking cosy and casual in a wool bobble hat.
The royal couple were seen arriving at the palace along with senior members of the royal family including Camilla, Duchess of Cornwall, Sophie Wessex, Prince Andrew and his daughter Princess Beatrice and Peter and Autumn Phillips and their two daughters Isla and Savannah.
Prince Harry was unable to attend as he has not yet got back to the UK from his South Pole charity expedition. Princess Eugenie, who now lives in New York, was also notably absent from the occasion.
Also in attendance were the Duchess of Kent, the Duke of Gloucester, Lady Helen Taylor, Lady Louise Windsor, Princess Michael of Kent and Viscount David Linley.
There are only two events each year that see the royal family gather on such a scale. The Trooping The Colour ceremony, which honours the Queen's official birthday, is followed by a family lunch attended both senior and minor members of the family.
The pre-Christmas lunch allows the extended family together, including those that aren't invited to spend Christmas with the Queen at Sandringham.