The family have a long history of spending summers here as the castle was popular with Margrethe's mother Queen Ingrid