During a chat with Alan Titchmarsh on Love Your Weekend, the 72-year-old was asked about the difficulties he faces when a character leaves the show - especially if there's no way back.
WATCH: Downton Abbey releases official trailer for sequel
"Killing them off is hard," he explained. "Yes, I mean, the audience always thinks you've done it wilfully. You know, I come in on some tyrant, 'Off with his head', but it isn't, of course, it's the actor saying it's the end of my three-year contract and I've got a play on Broadway."
He added: "The trouble is, a servant can get another job – there is no problem with that. They go off, and they work for Lady so-and-so instead, and that's fine. But a member of the family who was never prepared to be seen again, for them it’s the grim reaper. There is no way around it."
During the course of the franchise, fans were left devasted when they forced to say goodbye to the likes of Matthew Crawley, Lady Sybil Crawley, William Mason and Michael Gregson.
Lady Sybil (far right) left in season three
The show's creator then teased details about the upcoming film, which is set for release in March. He said: "I'm pleased with it actually, I think it works pretty well. It's a feel-good evening, and I don't know about you, but I could use a bit of feel-good right now, and you know there's a kind of fun in getting the club together again.
"Because it's been a long time, particularly the young ones, their careers have changed entirely over this period."
Like this story? Sign up to our newsletter to get other stories like this delivered straight to your inbox.