This paragraph really hit home to me.
I play in a rec softball league through work in the summer and I get our team organized each week (send out invites, make sure we have enough players, etc). The first couple years I would generally use the term girls, mainly because I would also use the term guys. At some point I thought that girls just seemed to weird to say when referring to adult women. Can't remember what made me think about it but that's irrelevant. So I started to make sure I would just go with the terms male and female rather than guy and girl.
I never really thought about why saying girl when referring to adult women sounded weird but saying guy when referring to adult men didn't. But what you said in this paragraph, in terms of boy-girl and guy-girl, nailed it. Terms like lady or gal sound just as weird to say, if not weirder, than girl. So men and women it is then.