If you don't love him or even care for him, get rid of him. Even if you two have a kid, get rid of him. It sounds pretty clear you two weren't meant to be together.