First and foremost, welcome to the Forum.
Second, when he is wanting sex and you are not, are you giving him other types of pleasuring, IE ~ Hand Jobs, BJs, etc, or are you just saying...no, I'm not in the mood?
As a man, I can say, going without sex from my wife can be a real drag...not just because I am horny and need something, but because I am missing out on the incredible bonding time between us both. But, when she is not in the mood and instead gives me something just for me...it is ALMOST as good as the real thing!
Or, maybe that will not work at all for him as he want sex with you and not your hand/mouth.
Another thing you can do is just let him pleasure him self with you via doggy style or something similar until you are yourself in the mood.
Best of Luck.