The couple, who divorced in 1996, have remained on good terms since their split and may even formalise their reunion with a second marriage.
"Mark my words, they will remarry," a close friend of the couple told The Daily Telegraph. "It is only a matter of time."
"It wouldn't surprise me at all," said another, when asked about the possibility. "They are a wonderful couple together and, better still, pretty amazing parents."
CLICK ON PHOTO FOR GALLERY
"The Duchess is being treated as if she and the Duke were still married," an insider told the Sunday Express, and added that the visit was "just like old times".
It is believed to be the first time since the reigning monarch has invited her former daughter-in-law back to the castle since 1992, after an incident involving Sarah caused embarrassment for the royal family.
The Duchess and Prince Andrew, both 53, are still good friends and raised their daughters together. The couple also live in the same grounds of the Royal Lodge, the Queen Mother's former country home in Berkshire.
"There was no reason for Andrew and I to get divorced," Sarah has said in the past. "There wasn't another person in our lives to go to.
We got divorced because I had to go out to work, and Andrew and I believed that it wasn't right for me to be commercial while I was still in the royal family."