i had been in a similar situation. i was married and he was my first and only. when we split and i was thrown back into the dating world...it was scary, still is. i used to look down on people that had more than one or 2 different sexual experiences under their belt. that is until i myself had a few more partners over the years, even a one night stand. i learned to get over other people's past because i learned how tough it is to be single these days.
Just try to remember that if you decide to date him, he is choosing to date you and only you. obviously his past partners meant nothing to him cuz they are in his past.
also try to remember that those past experiences made him the person he is today. is he a bad person? a player? if he is then stay away, but if he has a good heart and you know that, then go for it.
Are you saving yourself for marriage, or just when you fall in love?
Oh yeah, 4 or 5 partners is not much at all for a man. i once dated a guy that had like over 100 partners ( yes i used protection)!! is he hiding any more partners from you?
keep us posted on what you decide. also talk about it with him, find out why he slept with those women, that can help too.