The new series of the show, which premiered earlier this month, saw Princess Diana, played by Emma Corrin, become part of the cast.
But when the team behind the acclaimed series asked the Earl whether they could film at the family's ancestral home, Althorp House, where he and Diana grew up, he refused.
Charles spoke about his feelings towards the programme in a candid interview on Sunday.
Appearing on the presenter's ITV show Love Your Weekend with Alan Titchmarsh, the host asked Charles whether there is an unease on his part watching The Crown.
WATCH: Princess Diana told Phillip Schofield her secrets
The Earl replied: "There is a bit. Actually, The Crown asked if they could film at Althorp and I said obviously not."
He went on to explain his reasoning, saying: "The worry for me is that people see a programme like that and they forget that it is fiction.
"They assume, especially foreigners, I find Americans tell me they have watched The Crown as if they have taken a history lesson. Well, they haven’t."
Image: Love Your Weekend, Sunday 22nd November, 10am on ITV
When Alan asked him how truthful the show is, Charles responded: "It is very hard, there is a lot of conjecture and a lot of invention, isn’t there? You can hang it on fact but the bits in between are not fact."
The Earl went on to say that he felt a duty to protect his late sister's legacy. "I feel it is my duty to stand up for her when I can," he revealed.
"She left me for instance as guardian of her sons, so I feel there was a trust passed on. And we grew up together, you know if you grow up with somebody they are still that person, it doesn’t matter what happens to them later.
"So yeah, I feel very passionately that I have a role to honour her memory."
Like this story? Sign up to our newsletter to get other stories like this delivered straight to your inbox.