"Our son will be called Gabriel as there is a tradition in our family of going by our middle names," Kelsey revealed.
The young boy was born early on Tuesday in Los Angeles.
"We are blessed and excited to have this lovely young man join our family - he is magnificent" Kelsey told Entertainment Tonight!
Kelsey Grammer and Kayte Walsh in 2014
The 59-year-old actor has four other children from previous marriages: Spencer, 30, Greer, 22, Mason, 12 and nine-year-old Jude, and a two-year-old daughter named Faith with Kayte.
Kayte and Kelsey wed in 2011 and were left devastated in 2012 when they announced that they had lost Faith's twin, a little boy.
'It was unspeakably painful and we know that people will understand our desire to keep the news private then, as we know they will respect our privacy in this matter now,' they said in a statement at the time.
"I don't take it for granted," Kayte told People in 2012. "It's such a miracle and a gift to bring a baby into the world."
Kelsey Grammer and Kayte Walsh renew their vows in 2012
But the actor and his wife are hopeful for further children, revealing in 2012: "I think we'll have at least two more, if not three or four. I think it's a great way to live a life."
The happy couple met when Kayte was an airhostess for Virgin Airlines. In 2012 they renewed their vows, opting for an intimate wedding in Las Vegas' Little Church of the West.
Speaking of the renewal, Kelsey said: "Kayte’s the most wonderful person in the world. I’m just so happy! We’re really excited."
"We wanted to celebrate and re-dedicate (ourselves) to how much we love each other and to our growing family around the bend... I love that she had the courage to marry me twice!"
He added: "I've always wanted shot-gun wedding pictures so we can show the kids later on!"