Other than the affair (and I know this is a huge "other") how was the marriage.

Whatever you do, your daughter should be your first concern. I'm not saying what way this pushes the decision, but please consider her best interests.