You do what makes you happy and you explain to the children who are no longer children, let's face it.
Do you want to live a non-existant life for the rest of your being?
We have one life to live.
Also, obviously this um, thing you did which was wrong, played a part in why your marriage didn't work because someone other than your husband gave you feelings from the "soul" how could you learn to find that in your husband after that?
And, so perhaps your husband has the right also to find total happiness in his life, which he has not got over the years ....
You are really already thinking about this other man again...