The former couple met on the set of their 1984 film Racing with the Moon, and soon after they fell in love and got engaged when Elizabeth was 23. However, the relationship was short-lived and continued on-and-off for two years.
WATCH: Are you looking forward to Downton Abbey: A New Era?
During her recent chat with Daily Mail, the actress explained: "There's a road I could have taken, but I decided to call it off. It just felt like we were on different tracks.
"We were only in our early 20s. It would have been nothing short of a miracle had that become a lifelong partnership because we were both really young and figuring out who we were."
Sean went on to marry Madonna, while Elizabeth moved to England in 1992 and met now-husband, British film-maker Simon Curtis. They are now doting parents to two daughters - Matilda, 28, and Grace, 23.
The pair met on the set of 1984 film Racing with the Moon
Elizabeth, who plays Lady Cora in the beloved period drama, previously touched upon her failed engagement with the publication, and described their relationship as "exhausting".
"It was the frenzy of Sean's feelings that overwhelmed me," she said. "It was also the first real experience of boys I'd had, which was kind of nice. "In the beginning it was fun. We were a similar age and in the same place professionally, so we were experiencing everything together."
She added: "It was so exhausting, I couldn't sustain it. We also wanted to do different things. He likes to pretend he's not really a Hollywood person, but he is."
Like this story? Sign up to our newsletter to get other stories like this delivered straight to your inbox.