The 32-year-old, who has claimed he had a five-month affair with Kym, also said that he ended his romance with the actress because he thought she would never leave Jamie.
"I don't know if our relationship was the cause of Kym's divorce or just the catalyst - they had problems in their relationship already," Oliver told the Mirror.
"But what happened between us wasn't just a careless fling. It was real. Sadly, at the time, I thought Kym would never leave Jamie so I decided it was time to get out. It couldn't go on the way it was.
"But at the end of the affair the stress and roller coaster of emotions was too much and made me unhappy.
"There's no chance of going back," he continued. "I moved on and met someone new. I realise what true happiness is with Rhian (Sugden). But I'm glad Kym split from Jamie. He didn't make her happy."
Oliver says he and Kym had an affair for five months
At the weekend, Oliver spoke about how he fell for mum-of-three Kym, 37, while they were working on the set of Corrie last year.
He said they were together for five months, and even spoke about sharing a home together, before splitting.
"I thought she meant it when she said we were going to be together… and perhaps she thought she did, But looking back, she was never going to leave him for me.
Kym and Jamie divorced early this month
"At the time I thought it was love - we both did… She was torn between me and her family and I think she wanted both. There was only so long I could carry on as number two. Eventually my patience wore thin and I had to put a stop to it. When Kym realised I was giving up, she was upset."
When the story broke of Kym and Olver's 'affair' in October of last year, neither commented on it. EastEnders actor Jamie did tweet Oliver directly, writing, 'you need to come and see me', before deleting it ten minutes later.
Oliver has since moved on with new girlfriend Rhian, while Kym remains single following her 10 January divorce from 33-year-old Jamie, who was recently spotted in LA with model Chloe Peers.