I agree that he sounds like he is being VERY selfish, and pretty hypocritical. Sure, you cheated on him as well with his friend, and I can see why he would be angry about that. But he doesn't have a right to tell you that you can't be upset by what he did. They don't cancel each other out. He's trying to justify cheating and continuing to talk to her. He's guilting you into not bothering him about it and letting him go on with it... AND it sounds like he's blackmailing you by saying he'll leave if you confront him about it.

It's hard, I know. But I really think it's time to cut your losses and move on.