I think you should cut your losses sweetie... this guy is bad news. Any man that has the ability to go on vacation from his marriage, decieve an unsuspecting single woman for sex then go on about his merry way back to family life is displaying sociopathic tendencies. Do you really want to persue a relationship with a man that is capable of that level of deception with zero guilt?
He will use you in the same way he'd use anyone because he is self-serving and only concerned with whatever suits his needs the moment he needs them. Its not wrong of you to desire contact with a guy that made you feel the way he did... but it would be foolish, in my opinion.. to attempt to re-connect to a situation that will lead no where..
One day you may find yourself married with a couple of girls of your own, you may have a husband that pushes his bounderies with some other woman.. and if that other woman KNEW he had a wife and kids... you would probably hope she'd do the right thing and send him away, send him home.
I think you don't know his wife.. and you owe her nothing... but I think guys would be less douchebaggy if girls didn't allow them to be that way.. know what i mean?