On Tuesday the Frasier actor spoke via a satellite link to Freddie Glenn, who murdered 18-year-old Karen Elisa Grammer in 1975 when she was leaving her job at seafood restaurant Red Lobster in Colorado Springs.
"I accept that you live with remorse, but I live with tragedy every day," Kelsey told those at the man's parole hearing. "I accept your apology. I forgive you. However, I cannot give your release my endorsement.
Kelsey Grammer says he forgives the man who killed his sister
"I want to believe you have actually changed your life… things you say, I accept a lot of it," he added. "If someone did to your sister what you did to mine, what would you think should be done to him?"
The 59-year-old star also said that the idea of the man's parole "would be a betrayal of my sister's life".
Following the hearing a prison spokeswoman said that Glenn's request for release had been denied, meaning he will next be eligible for parole in 2017.
When he was first convicted in 1975, Freddie Glenn was given the death penalty, although that was later overturned and he was allowed to seek parole after 30 years in prison.
Kelsey honoured his sister by naming his eldest daughter Spencer Karen Grammer
Speaking in 2009, when Freddie Gunn last sought parole, Kelsey paid tribute to his sister.
"She was my best friend and the best person I knew," he said. "I loved my sister, Karen. I miss her. I miss her in my bones. I was her big brother. I was supposed to protect her. I could not. I have never gotten over it... It very nearly destroyed me."
In honour of his sibling, Kelsey named his eldest daughter Spencer Karen Grammer.