Mark Shand was taken to hospital on Tuesday evening after he slipped and fell backwards while lighting a cigarette and hit his head, the New York Police Department (NYPD) said.
In a statement, Clarence House said: "It is with deep sadness that we have to confirm that the Duchess of Cornwall's brother, Mark Shand, has today passed away in New York."
Mark Shand and the Duchess of Cornwall in 2009
In the statement Clarence House described Mark as a man of "extraordinary vitality, a tireless campaigner and conservationist."
Mark was Camilla's younger brother and a travel writer and conservationist. He had authored several books including Queen of the Elephants which became a BBC documentary and won the 1996 Thomas Cook Travel Book Award and the Prix Litteraire d'Amis.
He was the Chairman of a wildlife foundation called Elephant Family, which paid tribute to Mark in a statement.
"Today we've lost the head of our family. Mark Shand was a true force for conservation a legend and inspiration," it read.
Mark Shand with his daughter in 2012
Sotheby's are reported to have said that they have been honoured to have helped Mark raise money for his favoured cause at numerous events in London and New York.
Numerous artists and designers, including Ralph Lauren and fashion house Marchesa, had decorated Easter eggs for the event which were selling for as much as $7000.
It was the finale of a month-long egg hunt that happened all over the city.
Prince Charles's niece Princess Eugenie was also at the event, as well as fashion designer Dianne von Furstenberg and socialite Atlanta de Cadenet.
Mark had gone on to the after-party which was being held at the Diamond Horseshoe, a luxury nightclub near Times Square.
Mark Shand and Prince Charles in India in 2013
Camilla is due to fly to Canada on May 18 with Prince Charles for a three-day tour which will mark the centenary of World War One.