I don't think this sounds normal. Middle school children shouldn't want to play with Elementary school children, unless perhaps your son is only separated from him one or two grades. Even so, it doesn't sit well with me.
How does your son feel about this? He may feel just as uncomfortable as you. Since you say he is shy, maybe he's waiting for you to set boundaries for this boy?
How well do you know the boy's mother/parents? I don't know what grade middle school starts where you live, but it doesn't seem reasonable to leave a 6th-7th grader home alone with no supervision whatsoever. The boy's relationship with his mother/parents may be strained for some reason or another that you have no way of being aware of.
He could even be autistic; he can't distinguish facial patterns that denote mood or nuance of meaning. Autistic people usually blend in very well with "normal" people; it's very difficult to tell that they have a personality problem, but you realize that their personality is different.
I suggest that you try to talk to his parents. Maybe you could find out something more that will help you decide what kind of boundaries to put on this friendship.
As it is, I think you're doing the right thing by creating limits. Of course, I'm not a mother, but I'm saying it as a former shy child who often had bullying "friends." Let us know how it works out.